- cctbxbb - phenix-online.org

use_internal_variance in iotbx.merging_statistics
by Keitaro Yamashita 02 Nov '16

02 Nov '16

Dear Phenix/CCTBX developers, iotbx/merging_statistics.py is used by phenix.merging_statistics, phenix.table_one, and so on. By upgrading phenix from 1.10.1 to 1.11, merging statistics-related codes were significantly changed. Previously, miller.array.merge_equivalents() was always called with argument use_internal_variance=False, which is consistent with XDS, Aimless and so on. Currently, use_internal_variance=True is default, and cannot be changed by users (see below). These changes were made by @afonine and @rjgildea in rev. 22973 (Sep 26, 2015) and 23961 (Mar 8, 2016). Could anyone explain why these changes were introduced? https://sourceforge.net/p/cctbx/code/22973 https://sourceforge.net/p/cctbx/code/23961 My points are: - We actually cannot control use_internal_variance= parameter because it is not passed to merge_equivalents() in class filter_intensities_by_sigma. - In previous versions, if I gave XDS output to phenix.merging_statistics, <I/sigma> values calculated in the same way (as XDS does) were shown; but not in the current version. - For (for example) phenix.table_one users who expect this behavior, it can give inconsistency. The statistics would not be consistent with the data used in refinement. cf. the related discussion in cctbxbb: http://phenix-online.org/pipermail/cctbxbb/2012-October/000611.html Best regards, Keitaro

4 6

Monthly "how far are we from moving to git" message
by Graeme.Winter＠diamond.ac.uk 21 Oct '16

21 Oct '16

Good afternoon / morning / evening (depending on your locale) I'm just running a bootstrap build and having to check out the whole of cctbx one ... file ... at ... a ... time from svn & it reminded me we were going to move over to git at some point - ISTR was determined by the PHENIX release. Just wondering how this is going & how long it is likely to be until we can get this ball rolling :) Thanks & best wishes Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 6

sginfo and sgtbx
by Pascal 21 Oct '16

21 Oct '16

Hi, Just out of curiosity, sginfo website mentions that it has been superseeded by sgtbx. Is that a brand new implementation? I don't recognize sginfo code at all. Also, anyone can confirm that sgtbx is tightly embedded is cctbx and cannot be used standalone? Regards, Pascal

4 6

flex.int.as_double() performance
by richard.gildea＠diamond.ac.uk 11 Oct '16

11 Oct '16

Hi, Whilst looking into performance of various DIALS routines recently, I noticed that the flex.int.as_double() conversion takes roughly 3x as long as the equivalent functionality in numpy: (the choice of a 2527x2463 array is simply because this is the size of the raw data array for a Pilatus 6M detector) import time from dials.array_family import flex n = 100 a = flex.int(flex.grid(2527, 2463)) t0 = time.time() for i in range(n): b = a.as_double() t1 = time.time() t = t1-t0 print "flex.int.as_double(): %.2fs (%.3fs/call)" %(t, t/n) import numpy as np a = np.random.randint(100, size=a.all()) t0 = time.time() for i in range(n): b = a.astype(np.float64) t1 = time.time() t = t1-t0 print "numpy.astype(np.float64): %.2fs (%.3fs/call)" %(t, t/n) $ libtbx.python time_as_double.py flex.int.as_double(): 5.76s (0.058s/call) numpy.astype(np.float64): 1.86s (0.019s/call) This is the current version of the as_double function: static flex_double as_double(f_t const& a) { shared_plain<double> result(a.size(), init_functor_null<double>()); for(std::size_t i=0;i<a.size();i++) result[i] = a[i]; return flex_double(result, a.accessor()); } Replacing the above code with the following, I see a significant runtime improvement, now flex.int.as_double() is marginally quicker than the equivalent numpy routine: static flex_double as_double(f_t const& a) { shared_plain<double> result(a.begin(), a.end()); return flex_double(result, a.accessor()); } $ libtbx.python time_as_double.py flex.int.as_double(): 1.64s (0.016s/call) numpy.astype(np.float64): 1.82s (0.018s/call) Is there any obvious reason why making this change would be A Bad Thing? In the context of dials spot-finding, this change results in a 22s saving (out of a total 172s with the original code) when running dials.find_spots on a 540 image Pilatus 6M dataset using a single processor, so it is definitely a noticeable speed-up. All timings done on a macbook pro. Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 3

"import pickle"
by markus.gerstel＠diamond.ac.uk 23 Sep '16

23 Sep '16

Hi everyone, I note that we are using "import pickle" on some 57 occasions. Is there a particular reason we don't import cPickle? For example: --- libtbx/env_config.py (revision 25451) +++ libtbx/env_config.py (working copy) @@ -7,7 +7,10 @@ from libtbx import adopt_init_args import platform import shutil -import pickle +try: + import cPickle as pickle +except: + import pickle from cStringIO import StringIO import re import sys, os This should still work on all systems without issue. According to python docs the only difference is that cPickle can't be subclassed. If it's alright with you I would go through those imports and update them to use cPickle where possible. -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 2

bootstrap / svn method / say reason not anonymous && http
by Graeme.Winter＠diamond.ac.uk 21 Sep '16

21 Sep '16

Hello folks, In bootstrap.py is there any logical reason why we can’t use %sfmethod for anonymous as well as authenticated access to the cctbx_project sources i.e. in # Core CCTBX repositories # These must all provide anonymous access. class cctbx_module(SourceModule): module = 'cctbx_project' anonymous = ['svn','svn://svn.code.sf.net/p/cctbx/code/trunk'] authenticated = ['svn', '%(sfmethod)s://%(sfuser)[email protected]/p/cctbx/code/trunk'] also respect %(sfmethod) in place of svn as transport layer. Will help people behind firewalls. Appreciate that this will also need changing when we move to git anyhow… but meantimes there are people out there this would help. Not keen to make this change myself without checking first…! Thanks & best wishes Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

rpath problem on Mac system
by Haiguang Liu 21 Sep '16

21 Sep '16

Dear CCTBX developing team, Is there already a solution for the run path problem in the new Mac OS? The System Integrity Protection <https://support.apple.com/en-us/HT204899> does not allow relative run path. I temporarily disabled this feature to run the programs that requires boost (for example). If there is already a solution, please let me know. Thanks, Haiguang

3 4

Python 2.4 compatibility
by richard.gildea＠diamond.ac.uk 16 Sep '16

16 Sep '16

This commit uses the "with" statement, which was introduced in Python 2.5, to libtbx/__init__.py, preventing installation of DIALS from sources on systems with Python 2.4: https://sourceforge.net/p/cctbx/code/25101/ Could this code be changed to remove the use of the "with" statement? Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 5

failing test with CCP4 7.0.18
by markus.gerstel＠diamond.ac.uk 16 Sep '16

16 Sep '16

Hi, The test libtbx.python $(libtbx.find_in_repositories cctbx/development/run_shelx.py) P31 works with CCP4 7.0.17, but fails when run with SHELXL from CCP4 7.0.18. The test apparently involves creating a small random crystal structure, calculating the structure factors and solving the structure using SHELXL. The updated SHELXL version creates deterministic, but slightly different anisotropic temperature factors. I am not sure if this is a problem with the test or with the SHELXL update. Could anybody have a look into that? -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

4 4

Re: [cctbxbb] [Cctbx-cvs] SF.net SVN: cctbx:[25333] trunk/libtbx/env_config.py
by markus.gerstel＠diamond.ac.uk 09 Sep '16

09 Sep '16

Hi, I just spent some time tracking software crashes to this change. Is setting the default to en_US really appropriate and what we want? In particular it affects the output of downstream, external software we run from within python. What is the unicode issue you hint at in the commit message? -Markus Dr Markus Gerstel MBCS Postdoctoral Research Associate Tel: +44 1235 778698 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE -----Original Message----- From: bkpoon(a)users.sourceforge.net [mailto:[email protected]] Sent: 07 September 2016 00:54 To: cctbx-cvs(a)lists.sourceforge.net Subject: [Cctbx-cvs] SF.net SVN: cctbx:[25333] trunk/libtbx/env_config.py Revision: 25333 http://sourceforge.net/p/cctbx/code/25333 Author: bkpoon Date: 2016-09-06 23:54:29 +0000 (Tue, 06 Sep 2016) Log Message: ----------- Unicode support: set LC_ALL in dispatchers to the one in the user's environment (if available, and supports UTF-8), otherwise use the default setting of en_US.UTF-8; fixes unicode issue with python in Linux (e.g. os.path functions do not work correctly with unicode if LC_ALL=C Modified Paths: -------------- trunk/libtbx/env_config.py Modified: trunk/libtbx/env_config.py =================================================================== --- trunk/libtbx/env_config.py 2016-09-06 21:15:34 UTC (rev 25332) +++ trunk/libtbx/env_config.py 2016-09-06 23:54:29 UTC (rev 25333) @@ -945,6 +945,15 @@ def write_bin_sh_dispatcher(self, source_file, target_file, source_is_python_exe=False): + + # determine LC_ALL from environment (Python UTF-8 compatibility in Linux) + LC_ALL = os.environ.get('LC_ALL') # user setting + if (LC_ALL is not None): + if ( ('UTF-8' not in LC_ALL) and ('utf8' not in LC_ALL) ): + LC_ALL = None + if (LC_ALL is None): + LC_ALL = 'en_US.UTF-8' # default + f = target_file.open("w") if (source_file is not None): print >> f, '#! /bin/sh' @@ -975,7 +984,7 @@ print >> f, '#' print >> f, _SHELLREALPATH_CODE print >> f, 'unset PYTHONHOME' - print >> f, 'LC_ALL=C' + print >> f, 'LC_ALL=' + LC_ALL print >> f, 'export LC_ALL' print >> f, 'LIBTBX_BUILD="$(shellrealpath "$0" && cd "$(dirname "$RESULT")/.." && pwd)"' print >> f, 'export LIBTBX_BUILD' This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. ------------------------------------------------------------------------------ _______________________________________________ Cctbx-cvs mailing list Cctbx-cvs(a)lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/cctbx-cvs -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 3