- cctbxbb - phenix-online.org

Post-PHENIX changes
by Graeme.Winter＠Diamond.ac.uk 21 Sep '18

21 Sep '18

Morning All, I see from http://phenix-online.org/ That PHENIX 1.14 has been released Now, I see at https://github.com/cctbx/cctbx_project/pulls we have a few fairly hefty pull requests parked as “hold for phenix release” - I would certainly not suggest merging all these in one go however probably we should give some thought to stepping through some of them, deciding which ones we should look at first and starting to merge things? I for one would like Nick’s allocator code to come back into play (so we can make use of it for DIALS) Thoughts / opinions? For info: PHENIX folks did you tag the version which was released in PHENIX 1.14 to allow a release branch for .1 etc. releases? Many thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 1

repr for flex arrays
by Graeme.Winter＠Diamond.ac.uk 14 Sep '18

14 Sep '18

Hi Folks, Is there a reason for not having a helpful __repr__ for flex arrays i.e. Grey-Area:002-decide-when graeme$ dials.python Python 2.7.14 (default, Apr 17 2018, 15:02:10) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from dials.array_family import flex >>> a = flex.double([1, 2, 3]) >>> print a <scitbx_array_family_flex_ext.double object at 0x1095673c0> (I have a feeling I asked this before but cannot find reference) For example >>> print a.as_numpy_array() [ 1. 2. 3.] is more useful… but annoying to type I appreciate a could be “big” but numpy seems to handle this OK even - could __repr__ be punned to as_numpy_array(),__repr__()? Currently it prints the type of the array and its location in ram, which are adequately served by >>> type(a) <class 'scitbx_array_family_flex_ext.double'> >>> print id(a) 4451627968 If I print an array I more often want to see the content and less often the storage class and location (I humbly assert) Thoughts? I know this tends to confuse and annoy people who are new to flex arrays (and veterans who are busy debugging stuff…) Many thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

4 9

Python 3 status
by Graeme.Winter＠Diamond.ac.uk 06 Sep '18

06 Sep '18

Hi Folks, Are there any updates on the progress to Python 3 compatibility? I know there is a project here https://github.com/cctbx/cctbx_project/projects/2 but I have lost the thread of activities since I have been out of the office much of the summer. At the very least I know no progress has been made from this side of the pond as we’ve put this work down for now. i would guess that a lot is on hold while the PHENIX release is done - however I thought I should check in case I have missed anything? Part of why I ask is that there is a PHENIX meeting in Cambridge real soon now (mid September) and I wonder if it is worth folks from DIALS and PHENIX getting together at that time to discuss this? Any thoughts? Thanks & best wishes Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

7 12

Re: [cctbxbb] [git/cctbx] master: re-defining the shift/est convergence limit in terms of crystallographic rather than independent parameters to make it easier to relate to the convergence reported in the CIF; also caching the shifts for further analysis and reporting; (9339fd5f7)
by Nicholas Sauter 05 Sep '18

05 Sep '18

I'm worried about making changes like this to a fundamental toolbox component like normal equations solving. We use this for processing xfel data, and would like the opportunity to test any changes through a pull request, prior to having them committed to the main branch. Nick Sauter Nicholas K. Sauter, Ph. D. Senior Scientist, Molecular Biophysics & Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720 (510) 486-5713 On Wed, Sep 5, 2018 at 5:29 AM, CCTBX commit < diamondlightsource.jenkins(a)gmail.com> wrote: > Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx > On branch : master > > ------------------------------ > > > commit 9339fd5f7d240f0d6f85dd57ce42e796d19608f1 > Author: pcxod <oleg_dolomanov(a)hotmail.com> > Date: Wed Sep 5 13:29:29 2018 +0100 > > re-defining the shift/est convergence limit in terms of > crystallographic rather than independent parameters to make it easier to > relate to the convergence reported in the CIF; also caching the shifts for > further analysis and reporting; > > > ------------------------------ > > > 9339fd5f7d240f0d6f85dd57ce42e796d19608f1 > scitbx/lstbx/normal_eqns_solving.py | 17 +++++++++++------ > 1 file changed, 11 insertions(+), 6 deletions(-) > > diff --git a/scitbx/lstbx/normal_eqns_solving.py > b/scitbx/lstbx/normal_eqns_solving.py > index 660304eab..45f070622 100644 > --- a/scitbx/lstbx/normal_eqns_solving.py > +++ b/scitbx/lstbx/normal_eqns_solving.py > @@ -140,15 +140,20 @@ class iterations(object): > a = self.non_linear_ls.normal_matrix_packed_u() > a.matrix_packed_u_diagonal_add_in_place(value*a. > matrix_packed_u_diagonal()) > > - def do_scale_shifts(self, max_shift_over_esd): > + def do_scale_shifts(self, limit_shift_over_su): > x = self.non_linear_ls.step() > esd = self.non_linear_ls.covariance_matrix().matrix_packed_u_ > diagonal() > - x_over_esd = flex.abs(x/flex.sqrt(esd)) > - max_val = flex.max(x_over_esd) > - if max_val < self.convergence_as_shift_over_esd: > + ls_shifts_over_su = flex.abs(x/flex.sqrt(esd)) > + #max shift for the LS > + self.max_ls_shift_over_su = flex.max(ls_shifts_over_su) > + jac_tr = self.non_linear_ls.actual.\ > + reparametrisation.jacobian_transpose_matching_grad_fc() > + self.shifts_over_su = jac_tr.transpose() * ls_shifts_over_su > + self.max_shift_over_su = flex.max(self.shifts_over_su) > + if self.max_shift_over_su < self.convergence_as_shift_over_esd: > return True > - if max_val > max_shift_over_esd: > - shift_scale = max_shift_over_esd/max_val > + if self.max_ls_shift_over_su > limit_shift_over_su: > + shift_scale = limit_shift_over_su/self.max_ls_shift_over_su > x *= shift_scale > return False > > > ------------------------------ > > To unsubscribe from the CCTBX-COMMIT list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCTBX-COMMIT&A=1 >

2 3

Adding more parameters to xray::scatterer for anharmonic thermal displacements
by Luc J. Bourhis 22 Aug '18

22 Aug '18

Hi, we are working with Michal Chodkiewicz (Warsaw University) to add anharmonic thermal tensors to smtbx refinement. I was thinking to add one pointer to xray::scatterer, which can be NULL. When it is not so, it would point to an object which can compute the value and the derivatives for that thermal factor. That means we would have a base class with virtual methods, but I would say that is our problem! The issue for the rest of the cctbx community is the extra storage of that pointer in that class. Would that be all right? I remember quite a few years ago Ralf saying he wanted to replace the ADP's by a pointer to an external storage to save space. Our idea is along the same line. Best wishes, Luc

3 2

model selection
by Pavel Afonine 17 Aug '18

17 Aug '18

Currently model class has three selection methods: def selection(self, selstr, optional=True): def select(self, selection): def iselection(self, selstr): which is hideously confusing with zero chance to remember, and also inconsistent with the rest. Most objects in cctbx have ".select(self, selection)" methods, while being inconsistent about what they expect: iselection or boolean selection as "selection" argument. Some smart ones though can take both int or bool selections (figure out type automatically, which is trivial). Can we consolidate all three into one ".select()" that would allow taking bool or int or str? Or you think there are philosophical arguments against it? Pavel On 8/16/18 15:00, CCTBX commit wrote: > Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx > On branch : master > > ------------------------------------------------------------------------ > > > commit c98b44997e8eb7bf53ee828f510b933cd5167758 > Author: Oleg Sobolev <osobolev(a)lbl.gov> > Date: Thu Aug 16 15:00:20 2018 -0700 > > Proper use of model class > > > ------------------------------------------------------------------------ > > > c98b44997e8eb7bf53ee828f510b933cd5167758 > mmtbx/model/model.py | 3 +-- > 1 file changed, 1 insertion(+), 2 deletions(-) > > diff --git a/mmtbx/model/model.py b/mmtbx/model/model.py > index 6f2247452..87412f95c 100644 > --- a/mmtbx/model/model.py > +++ b/mmtbx/model/model.py > @@ -2312,8 +2312,7 @@ class manager(object): > sizes = flex.int() > h = self.get_hierarchy() > if(macro_molecule_only): > - asc = h.atom_selection_cache() > - s = asc.selection("protein or nucleotide") > + s = self.selection("protein or nucleotide") > h = h.select(s) > for r in h.residue_groups(): > sizes.append(r.atoms().size())

2 3

Extracting data from MTZ
by David Waterman 14 Aug '18

14 Aug '18

Hi all, I have a script that extracts Fobs and Fcalc from an MTZ file produced by Refmac for some analysis. I was just doing this in a trivial way: fobs = m.extract_reals("F").data fc = m.extract_reals("FC_ALL").data and this worked fine, until I came across a file with missing observations, so that the lengths of fobs and fc do not match. Can anybody advise a better way to do this please? I'm inexperienced with iotbx.mtz. I don't need anything other than the values of the structure factors, but I need each element of fobs and fc to contain the equivalent values. Thanks -- David

3 3

How does Jenkins work?
by Takanori Nakane 28 Jul '18

28 Jul '18

Hi, Why does Jenkins still run tests on my old commits on my deleted (dxtbx-sacla) branch? The test failure is probably caused by my commit dc17165 but I committed a fix 5c7416b and confirmed that it passes the tests. So I merged my topic branch to master and deleted the branch from GitHub. I know how git works but I am new to the practice in the cctbx repository (like using rebase instead of merge). If I am doing something wrong, could you please point it to me? Best regards, Takanori Nakane > <html><head></head><body><table style="width:100%; background-color: > #F59400; color: #FFF; border-radius: 0.5em; margin-bottom: 0.5em"><tr><td > style="margin: 0.5em 1.25em 0.5em 0.7em; font-weight: bold; font-family: > sans-serif; font-size: 110%"><a > href="http://jenkins.diamond.ac.uk:8080/job/cctbx_branch_tests/145/" > style="color:#FFF; text-decoration: none"><big>Build CCTBX Branch Tests is > unstable</big></a></td></tr></table><br/><table style="width:100%; > background-color: #F59400; color: #FFF; border-radius: 0.5em; > margin-bottom: 0.5em"><tr><td style="margin: 0.5em 1.25em 0.5em 0.7em; > font-weight: bold; font-family: sans-serif; font-size: 110%"><a > href="http://jenkins.diamond.ac.uk:8080/job/cctbx_branch_tests/145/testReport" > style="color:#FFF; text-decoration: none">3 out of 940 tests failed (132 > skipped)</a></td></tr></table><table style="margin: 0.5em 0 1em > 2em"><tr><td><pre style="font-size:80%">3 tests failed. > FAILED: > dxtbx.build_dials.modules.cctbx_project.dxtbx.tests.test_regression_images.test_no_multiple_format_understanding[SACLA_MPCCD_Cheetah/run266702-0-subset.h5] > Error Message: > exit code 1 > Stack Trace: > FAILED: > dxtbx.build_dials.modules.cctbx_project.dxtbx.tests.test_regression_images.test_no_multiple_format_understanding[putative_imgCIF_HDF5_mapping/minicbf.h5] > Error Message: > exit code 1 > Stack Trace: > FAILED: > dxtbx.build_dials.modules.cctbx_project.dxtbx.tests.test_regression_images.test_no_multiple_format_understanding[putative_imgCIF_HDF5_mapping/X4_wide_M1S4_1.nxs] > Error Message: > exit code 1 > Stack Trace: > </pre></td></tr></table><table style="width:100%; background-color: > #F59400; color: #FFF; border-radius: 0.5em; margin-bottom: 0.5em"><tr><td > style="margin: 0.5em 1.25em 0.5em 0.7em; font-weight: bold; font-family: > sans-serif; font-size: 110%">Changes since the last successful > build</td></tr></table><table style="margin: 0.5em 0 1em > 2em"><tr><td><table style="padding: 0.1em 0.4em; background-color: > #F59400; color:#fff"><tr><td>145</td></tr></table></td><td>No changes > </td></tr></table><table style="width:100%; background-color: #CCC; color: > #000; border-radius: 0.5em; margin-bottom: 0.5em"><tr><td style="margin: > 0.5em 1.25em 0.5em 0.7em; font-weight: bold; font-family: sans-serif; > font-size: 110%">Build environment</td></tr></table><table style="margin: > 0.5em 0 1em 2em"><tr><td><span > style="font-size:130%"> [ </span>cctbx <strong>71a4ab94a > 2018-07-25 06:47:08 +0900</strong><span > style="font-size:130%"> | </span>DIALS regression > <strong>1924</strong>, PHENIX regression <strong>4419</strong>, DLS > Jenkins scripts <strong>8de8b58 2018-07-04 23:29:22 +0100</strong><span > style="font-size:130%"> ] </span></td></tr></table><span > style="color:#CCC;font-size:80%">mail template revision > 2487</span></body></html> > <br> > <hr> > <p align="center">To unsubscribe from the CCTBX-COMMIT list, click the > following link:<br> > <a > href="https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCTBX-COMMIT&A=1" > target="_blank">https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCTBX-COMMIT&A=1</a> > </p>

4 5

mmtbx wilson_scaling in a small molecule context
by ben.williams＠diamond.ac.uk 23 Jul '18

23 Jul '18

Hello any and all, I was looking to make use of the functionality of wilson_scaling (Wilson plot, etc.) from mmtbx.scaling.data_statistics, in the context of small molecule data. I'm new to mmtbx, so here are two (groups of) questions for the enlightened: 1. Designed, as it is, with MX in mind, wilson_scaling's __init__ demands as an argument the number of protein residues, in order to form an estimate of the contents of the asymmetric unit. If this class were to be used in a small molecule context, what would be a sensible value to use? Would spoofing n_residues with an arbitrary small number (1?) be a robust bodge? Is it ever likely to howl? 2. Being crystals of small molecules, the samples in question may have inversion centres, mirror planes. Having been designed as a tool for MX, will wilson_scaling throw a wobbly when it sees my data? Many thanks, Ben Williams Diamond Light Source -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

subset of reflections from mtz object
by Robert Oeffner 18 Jul '18

18 Jul '18

Hi, I'm not as well versed using iotbx.mtz as I ought to be. I'd like to make a subset of reflections from an mtz file bounded by a chosen dmin and dmax, i.e. a resolution shell of reflections. What is the best way of doing this in python? Many thanks, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

2 1