- cctbxbb - phenix-online.org

Misuse of libtbx/configure.py
by Robert Oeffner 09 Apr '18

09 Apr '18

Hi, I'm not sure when this has been put into configure.py as I have had taken my eyes of the ball lately regarding Windows builds. But it seems that during bootstrap build the configure.py step is now doing more than just configuring modules. On my PC the excerpt below from stdout shows an error during the configure.py step Processing: "c:\busers\oeffner\nightlybuild\x32\current\modules\dials\libtbx_refresh.py" requires mock>=2.0, has 2.0.0 requirement orderedset is not currently met, package not installed attempting install of orderedset... Collecting orderedset Using cached orderedset-2.0.1.tar.gz Installing collected packages: orderedset Running setup.py install for orderedset: started Running setup.py install for orderedset: finished with status 'error' Complete output from command C:\Busers\oeffner\NightlyBuild\x32\Current\base\bin\python\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\oeffner\\appdata\\local\\temp\\pip-build-smhifw\\orderedset\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record c:\users\oeffner\appdata\local\temp\pip-0seb8q-record\install-record.txt --single-version-externally-managed --compile: running install running build running build_py creating build creating build\lib.win32-2.7 creating build\lib.win32-2.7\orderedset copying lib\orderedset\__init__.py -> build\lib.win32-2.7\orderedset running build_ext building 'orderedset._orderedset' extension error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat). Get it from http://aka.ms/vcpython27 ---------------------------------------- Command "C:\Busers\oeffner\NightlyBuild\x32\Current\base\bin\python\python.exe -u -c "import setuptools, tokenize;__file__='c:\\users\\oeffner\\appdata\\local\\temp\\pip-build-smhifw\\orderedset\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record c:\users\oeffner\appdata\local\temp\pip-0seb8q-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in c:\users\oeffner\appdata\local\temp\pip-build-smhifw\orderedset\ You are using pip version 9.0.1, however version 9.0.3 is available. You should consider upgrading via the 'python -m pip install --upgrade pip' command. install failed. please check manually My first concern with this approach is that the bootstrap process doesn't stop. It carries on and eventually builds the Phenix software suite. This may or may not be a broken build as it is not clear whether the above failure to build orderedset was a showstopper for something. Secondly, I thought that modules would not be installed anywhere else than when building base components or when downloading sources. For a small project it is a nice and sneaky solution to have certain build steps second guessing what dependencies are missing and then installing them "on the fly", if not covertly. But for a large project such as CCTBX with several teams using that as a common framework this becomes messy I would argue. A quick and dirty fix for the above error might be just to put vcvarsall.bat in the PATH environment. But that is not a solution since bootstrap.py has been fashioned to use different versions of Visual C++, some of which are not invoked through a vcvarsall.bat file. I appreciate the need to install additional component modules for python. But would it not be more tidy to install these during the base step of the bootstrap process? Is there any reason why this cannot happen? Regards, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

5 5

simtbx - Pilatus example available
by James Holton 08 Apr '18

08 Apr '18

Took most of last week, but I finally got it to work. tst_nanoBragg_pilatus.py will create a Pilatus-like image, complete with bad pixels, window panes, thickness effects, module-to-module misalignment and 0.1 deg of phi rotation. It takes one optional argument, which is the image number in a sequence of 3600. I've processed this simulated data set with xia2 and XDS and found the anomalous signal has the right sign. This required some edits and additions to the simTBX code as well as FormatCBFMini.as_file() that I think fixed a beam center consistency bug. Now if you dxtbx.load() and FormatCBFMiniPilatus.as_file() you get the same beam center in both input and output headers. I also enhanced the header to make it look more like the ones in DIALS regression. A new "SIM.F000" value is now available for marking the "direct beam" on the image itself. I spent most of Friday and all of this morning trying to follow the check-in procedure we laid out during the DIALS workshop. Not sure how, but it suddenly started working just now. I believe I have checked in this code into the simtbx_sandbox branch of cctbx_project. Is Jenkins looking at this yet? Is anyone looking at this yet? -James Holton MAD Scientist

3 3

print dxtbx.model.Scan()
by James Holton 04 Apr '18

04 Apr '18

The following sequence of commands: % source build/setpaths.csh % libtbx.python >>> import dxtbx >>> from dxtbx import model >>> print dxtbx.model.Scan() seems to cause Python itself to core dump. Is this expected? show_stack(1): <stdin>(1) <module> libc backtrace (25 frames, most recent call last): /home/jamesh/projects/cctbx-dev/build/../base/bin/python2.7() [0x400659] /lib64/libc.so.6(__libc_start_main+0xfd) [0x34e961ed1d] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(Py_Main+0xc74) [0x7f93eb5c2b74] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyRun_AnyFileExFlags+0x4c) [0x7f93eb5ad78c] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyRun_InteractiveLoopFlags+0x4e) [0x7f93eb5ad04e] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyRun_InteractiveOneFlags+0x195) [0x7f93eb5ace35] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32) [0x7f93eb58d4e2] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x88e) [0x7f93eb58d3ce] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3ef0) [0x7f93eb589de0] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyFile_WriteObject+0x78) [0x7f93eb503b58] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(+0x958f2) [0x7f93eb5288f2] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyObject_Str+0xa) [0x7f93eb52879a] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(_PyObject_Str+0x7b) [0x7f93eb5286ab] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(+0xb5a1e) [0x7f93eb548a1e] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x43) [0x7f93eb584fd3] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f93eb4dc093] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(+0x5b6cf) [0x7f93eb4ee6cf] /home/jamesh/projects/cctbx-dev/base/bin/../lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f93eb4dc093] /home/jamesh/projects/cctbx-dev/build/lib/libboost_python.so(+0x2d088) [0x7f93e3f23088] /home/jamesh/projects/cctbx-dev/build/lib/libboost_python.so(boost::python::handle_exception_impl(boost::function0<void>)+0x43) [0x7f93e3f34013] /home/jamesh/projects/cctbx-dev/build/lib/libboost_python.so(+0x2fe08) [0x7f93e3f25e08] /home/jamesh/projects/cctbx-dev/build/lib/libboost_python.so(boost::python::objects::function::call(_object*, _object*) const+0xef) [0x7f93e3f251ff] /home/jamesh/projects/cctbx-dev/build/lib/dxtbx_model_ext.so(boost::python::objects::caller_py_function_impl<boost::python::detail::caller<std::string (*)(dxtbx::model::Scan const&), boost::python::default_call_policies, boost::mpl::vector2<std::string, dxtbx::model::Scan const&> > >::operator()(_object*, _object*)+0x83) [0x7f93db88a7a3] /home/jamesh/projects/cctbx-dev/build/lib/dxtbx_model_ext.so(dxtbx::model::boost_python::scan_to_string(dxtbx::model::Scan const&)+0x319) [0x7f93db871929] /lib64/libc.so.6() [0x34e9632510] Segmentation fault (Python and libc call stacks above)

2 1

Exceptions squashed by easy_mp (revenge of)
by Graeme.Winter＠Diamond.ac.uk 03 Apr '18

03 Apr '18

Folks, Following up on user reports again of errors within easy_mp - all that gets logged is “something went wrong” i.e. Using multiprocessing with 10 parallel job(s) Traceback (most recent call last): File "/home/user/bin/dials-installer/build/../modules/dials/command_line/integrate.py", line 613, in <module> halraiser(e) File "/home/user/bin/dials-installer/build/../modules/dials/command_line/integrate.py", line 611, in <module> script.run() File "/home/user/bin/dials-installer/build/../modules/dials/command_line/integrate.py", line 341, in run reflections = integrator.integrate() File "/home/user/bin/dials-installer/modules/dials/algorithms/integration/integrator.py", line 1214, in integrate self.reflections, _, time_info = processor.process() File "/home/user/bin/dials-installer/modules/dials/algorithms/integration/processor.py", line 271, in process preserve_exception_message = True) File "/home/user/bin/dials-installer/modules/dials/util/mp.py", line 171, in multi_node_parallel_map preserve_exception_message = preserve_exception_message) File "/home/user/bin/dials-installer/modules/dials/util/mp.py", line 53, in parallel_map preserve_exception_message = preserve_exception_message) File "/home/user/bin/dials-installer/modules/cctbx_project/libtbx/easy_mp.py", line 627, in parallel_map result = res() File "/home/user/bin/dials-installer/modules/cctbx_project/libtbx/scheduling/result.py", line 119, in __call__ self.traceback( exception = self.exception() ) File "/home/user/bin/dials-installer/modules/cctbx_project/libtbx/scheduling/stacktrace.py", line 115, in __call__ self.raise_handler( exception = exception ) File "/home/user/bin/dials-installer/modules/cctbx_project/libtbx/scheduling/mainthread.py", line 100, in poll value = target( *args, **kwargs ) File "/home/user/bin/dials-installer/modules/dials/util/mp.py", line 91, in __call__ preserve_exception_message = self.preserve_exception_message) File "/home/user/bin/dials-installer/modules/cctbx_project/libtbx/easy_mp.py", line 627, in parallel_map result = res() File "/home/user/bin/dials-installer/modules/cctbx_project/libtbx/scheduling/result.py", line 119, in __call__ self.traceback( exception = self.exception() ) File "/home/user/bin/dials-installer/modules/cctbx_project/libtbx/scheduling/stacktrace.py", line 86, in __call__ raise exception RuntimeError: Please report this error to dials-support(a)lists.sourceforge.net: exit code = -9 I forget why it was decided that keeping the proper stack trace was a bad thing, but could this be revisited? It would greatly help to see it in the output of the program (if as is the case here I do not have the user data) My email-fu is not strong enough to dig out the previous conversation Cheers Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 6

Re: [cctbxbb] [git/cctbx] master: New command: libtbx.install (914c74bf1)
by Pavel Afonine 13 Mar '18

13 Mar '18

Just out of curiosity: what's your definition of "real world"? Pavel On 3/13/18 03:49, CCTBX commit wrote: > Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx > On branch : master > > ------------------------------------------------------------------------ > > > commit 914c74bf195b5e5d4fa2c4145da3e5f89c7e5872 > Author: Markus Gerstel <markus.gerstel(a)diamond.ac.uk> > Date: Tue Mar 13 10:43:22 2018 +0000 > > New command: libtbx.install > > This downloads and configures from a select list of cctbx/libtbxish > modules in a similar manner to pip in the real world. This will not > update or install any libraries in the base build, but assuming > those libraries are present libtbx.install should work. > > > ------------------------------------------------------------------------ > > > 914c74bf195b5e5d4fa2c4145da3e5f89c7e5872 > libtbx/command_line/install.py | 118 > +++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 118 insertions(+) > > diff --git a/libtbx/command_line/install.py > b/libtbx/command_line/install.py > new file mode 100644 > index 000000000..47ad94eb5 > --- /dev/null > +++ b/libtbx/command_line/install.py > @@ -0,0 +1,118 @@ > +from __future__ import absolute_import, division, print_function > + > +import collections > +import os > +import sys > +from optparse import SUPPRESS_HELP, OptionParser > + > +import procrunner > + > +import libtbx.load_env > +from libtbx.auto_build.bootstrap import Toolbox > + > +# Basically 'pip' for selected libtbx/cctbx modules. > + > + > +def is_source_repository(path): > + return (path / '.git').isdir() or (path / '.svn').isdir() > + > +def run(args): > + parser = OptionParser(usage="libtbx.install [package]", > + description="Installs an additional cctbx > package") > + parser.add_option("-?", action="help", help=SUPPRESS_HELP) > + options, args = parser.parse_args(args) > + > + modules_directory = list(filter(lambda dir: not > is_source_repository(dir), libtbx.env.repository_paths)) > + if not modules_directory: > + sys.exit("No repository path candidate found. Can't install > modules without an installation root.") > + if len(modules_directory) > 1: > + print("More than one repository path candidate found.") > + installation_root = modules_directory[0] > + print("Using %s as installation root" % abs(installation_root)) > + > + packages_to_configure = set() > + > + errors = False > + for package in args: > + if (installation_root / package).isdir(): > + print("Skipping package %s: Directory already exists in > installation root" % package) > + errors = True > + continue > + if package not in warehouse: > + print("Skipping package %s: Never heard of this before" % package) > + errors = True > + continue > + > + downloaded = False > + for mech, call in mechanisms.items(): > + if mech in warehouse[package]: > + print("Attempting to obtain %s using %s..." % (package, > mech), end='') > + sys.stdout.flush() > + if call(abs(installation_root / package), > warehouse[package][mech]): > + assert (installation_root / package).isdir(), "Installation > failed" > + downloaded = True > + print("success") > + break > + else: > + assert not (installation_root / package).isdir(), "Install > mechanism %s did not fail cleanly" % mech > + print("failed") > + if not downloaded: > + print("Skipping package %s: Could not install" % package) > + errors = True > + continue > + > + packages_to_configure.add(package) > + > + if packages_to_configure: > + packages_to_configure = sorted(packages_to_configure) > + print("Configuring packages %s" % ", ".join(packages_to_configure)) > + os.chdir(abs(libtbx.env.build_path)) > + result = procrunner.run_process(['libtbx.configure'] + > packages_to_configure) > + if result['exitcode']: errors = True > + > + if errors: > + sys.exit(1) > + > +def install_git(location, source): > + try: > + result = procrunner.run_process(['git', 'clone', '--recursive', > source, location], print_stdout=False, print_stderr=False) > + if result['exitcode']: > + return False > + Toolbox.set_git_repository_config_to_rebase(os.path.join(location, > '.git', 'config')) > + return True > + except OSError: > + return False # git may not be installed > + > +def install_zip(location, source): > + os.mkdir(location) > + tempfile = os.path.join(location, '.tmp.zip') > + etagfile = os.path.join(location, '..tmp.zip.etag') > + def cleanup(): > + try: os.remove(tempfile) > + except OSError: pass > + try: os.remove(etagfile) > + except OSError: pass > + if Toolbox.download_to_file(source['url'], tempfile) <= 0: > + cleanup() > + os.rmdir(location) > + return False > + Toolbox.unzip(tempfile, location, trim_directory=source.get('trim', 0)) > + cleanup() > + return True > + > +mechanisms = collections.OrderedDict(( > + ('git-auth', install_git), > + ('git-anon', install_git), > + ('http-zip', install_zip), > +)) > + > +warehouse = { > + 'i19': { > + 'git-auth': '[email protected]:/xia2/i19', > + 'git-anon': 'https://github.com/xia2/i19.git', > + 'http-zip': { 'url': > 'https://github.com/xia2/i19/archive/master.zip', 'trim': 1 }, > + }, > +} > + > +if __name__ == '__main__': > + run(sys.argv[1:])

2 1

Re: [cctbxbb] [git/cctbx] master: rename test files, remove them from run_tests (23a4a6fe4)
by Pavel Afonine 05 Mar '18

05 Mar '18

Just to make sure: you are converting to use pytest this particular codes (fable), correct? Pavel P.S.: I'm allergic to pytest. On 3/3/18 07:46, CCTBX commit wrote: > This in preparation for pytestification.

5 8

Revisiting cctbx/iotbx/dxtbx dependencies
by Aaron Brewster 19 Feb '18

19 Feb '18

Hey folks, especially Rob and Luc. So after a quick change to iotbx/libtbx_config, it turns out there's a way to get a super minimum version of cctbx, without configuring dxtbx, and it takes advantage of existing mechanisms: - python bootstrap.py --builder=cctbx hot update - python modules/cctbx_project/libtbx/auto_build/install_base_packages.py openssl python python_compatibility certifi numpy cython png libsvm pytest pythonextra jinja2 junitxml biopython docutils sphinx pyopengl misc freetype matplotlib pillow gettext glib expat fontconfig render pixman tiff cairo gtk fonts wxpython - mkdir build; cd build - ../base/bin/python ../modules/cctbx_project/libtbx/configure.py cctbx iotbx - make Now, some details as to why this works. First, the cctbx builder in boostrap.py doesn't download cbflib so cbflib is never configured. Next, the base step of bootstrap.py is simply a call to install_base_packages.py with a list of required packages. You can just run it standalone and get the packages you need. Here, I ran "python bootstrap.py --builder=cctbx base", killed it after I saw the list of packages it was going to install, then ran install_base_packages manually after removing hdf5 and lz4 (which depends on hdf5) from the list. One could imagine removing other packages depending on your use case. With that, I could run configure.py directly. I can't use cctbx by itself as the cctbx/maptbx/SConscript file has a dependency on iotbx, but configuring cctbx and iotbx together seems to work. The key trick was to make iotbx only depend on dxtbx if dxtbx is explicitly configured. I used a keyword in libtbx_configure added by Luc a bit ago: optional_modules_only_if_explicit_request. I committed that change just now. Outstanding questions: 1. Is there interest in making this a builder? It would be straightforward to add a builder named cctbx_minimal which would just do the above steps 2. Is there further need for making cbflib and hdf5 optional in dxtbx? I purport there is not as the above procedure seems to satisfy the use case, but I can be persuaded otherwise. Thanks, -Aaron

4 5

Re: [cctbxbb] [git/cctbx] master: use larger tolerance (0b81ca799)
by Graeme.Winter＠diamond.ac.uk 11 Feb '18

11 Feb '18

Pavel, This commit did not fix test on OS X High Sierra libtbx.python "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/maptbx/tst_interpolation.py" [FAIL] 1.4s Time: 1.39 Return code: 1 OKs: 0 Standard error: Traceback (most recent call last): File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/maptbx/tst_interpolation.py", line 128, in <module> exercise(f = f, gx = gx, gy = gy, gz = gz, function_type=function_type) File "/Users/graeme/svn/cctbx/modules/cctbx_project/cctbx/maptbx/tst_interpolation.py", line 99, in exercise assert approx_equal(it, 0.0, 1.e-4) AssertionError cheers Graeme On 11 Feb 2018, at 00:14, CCTBX commit <diamondlightsource.jenkins(a)GMAIL.COM<mailto:[email protected]>> wrote: Repository : ssh://g18-sc-serv-03.diamond.ac.uk/cctbx On branch : master ________________________________ commit 0b81ca7992f9cf7730eb007aff6e8c30bd2ba71b Author: Pavel <pafonine(a)lbl.gov<mailto:[email protected]>> Date: Sat Feb 10 16:14:51 2018 -0800 use larger tolerance ________________________________ 0b81ca7992f9cf7730eb007aff6e8c30bd2ba71b cctbx/maptbx/tst_interpolation.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/cctbx/maptbx/tst_interpolation.py b/cctbx/maptbx/tst_interpolation.py index 918ba68fd..90dec8a46 100644 --- a/cctbx/maptbx/tst_interpolation.py +++ b/cctbx/maptbx/tst_interpolation.py @@ -96,15 +96,15 @@ def exercise(f,gx,gy,gz,function_type, eps = 1.e-6): if(function_type=="L"): cntr+=1 for it in [tl,gxl,gyl,gzl, tq,gxq,gyq,gzq, tt,gxt,gyt,gzt]: - assert approx_equal(it, 0.0) + assert approx_equal(it, 0.0, 1.e-4) if(function_type=="Q"): cntr+=1 for it in [tq,gxq,gyq,gzq, tt,gxt,gyt,gzt]: - assert approx_equal(it, 0.0) + assert approx_equal(it, 0.0, 1.e-4) if(function_type=="C"): cntr+=1 for it in [tt,gxt,gyt,gzt]: - assert approx_equal(it, 0.0) + assert approx_equal(it, 0.0, 1.e-4) if(abs(tl)>eps and abs(tq)>eps): assert tl >=tq if(function_type=="F"): cntr+=1 -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Re: [cctbxbb] [Dials-support] indexing success rates
by Aaron Brewster 05 Feb '18

05 Feb '18

Right, typically if I don't know the cell, I'll index a bunch of images with no target, then use cluster.unit_cell on the results to get a consensus cell, then re-index with that cell. For dials.stills_process, the default of fft1d works pretty well generally, but to get the best results, indexing.method_list=fft1d,fft3d might be better. There, if the first method fails the second is tried. real_space_grid_search can also be used. Getting a specific image in a composite file (such as hdf5) isn't a use case I've hit often, so I don't have an easy solution there. It's possible an image range parameter to dials.import would be useful? -Aaron On Sat, Feb 3, 2018 at 1:02 AM, Graeme Winter <graeme.winter(a)gmail.com> wrote: > Hi Aaron > > (and DIALS) > > For stills / single images - should dials.index switch by magic to default > fft1d? > > Of is that the responsibility of e.g. xia2 / cctbx.xfel to worry about? > > It has to be said, more often than not, particularly for screening, we do > not know cell / symmetry a priori (at diamond anyways) > > A related question > > If I have a data block / strong pickle from 1000 stills, how can I index > e.g. image 672? Do I have to split this into 1000 separate things? > > @JH - thanks for this - I think some rigorous testing will be useful here. > > Cheers Graeme > > On 3 Feb 2018, at 08:37, Aaron Brewster <asbrewster(a)lbl.gov> wrote: > > Hi James, thanks. I looked at random image 15 and after trying a bunch of > stuff it indexes fine if I use indexing.method=fft3d. The default for > stills processing is fft1d, so I find this super interesting. > > For what it's worth, I pretty much never process stills without a target > cell. I generated ~30 random images using your script and indexed them, > then I got the unit cells with "dials.show *refined*.json | grep cell". > They were all about the same. So I re-indexed random image 15 using fft1d > but I also specified known_symmetry.cell=50,60,70,90,90,90 and it indexed > fine. > > So, either indexing.method=fft3d or known_symmetry.cell=50,60,70,90,90,90 > is sufficient to index random image 15. Certainly for experimental data I > would recommend using known_symmetry as the results are always better. > However, I'd like to look deeper at fft1d vs fft3d at some point. > > Cc'ing dials-support. > > Thanks! > -Aaron > > > > On Fri, Feb 2, 2018 at 1:35 PM, James Holton <jmholton(a)lbl.gov> wrote: > >> While testing the new simTBX diffraction image simulator Aaron and I >> found that some rather beautiful-looking images still won't index. >> Subsequently, I followed up with mosflm, which does succeed in these cases, >> even without prior knowledge of the cell or space group. >> >> I have now run this 1000 times, varying only the crystal orientation, and >> found 171 cases of dials.stills_process failing to index the simulated >> image whereas mosflm had no trouble, and 5 cases where mosflm failed and >> dials.stills_process succeeded. There were no overlapping cases where both >> programs failed. >> >> I have tarballed up the relevant files here: >> http://bl831.als.lbl.gov/~jamesh/bugreports/dials_index_020218.tgz >> >> The tarball contains a "runme.com" shell script for reproducing these >> results, and also the tst_nanoBragg_forindex.py jiffy for making a test >> image given a provided random-number seed. For example: >> >> libtbx.python ./tst_nanoBragg_forindex.py random 15 >> >> will create a file called noiseimage_001.cbf that dials.stills_process >> cannot index (at least in my hands). For those of you who don't have >> mosflm installed, the tarball incluses a copy, and the "autoindex.com" >> script should run on any linux system. >> >> Any ideas as to why there is this discrepancy? Feels like an opportunity >> for a non-trivial improvement. >> >> Something to ponder over the weekend, I suppose. Note that I recently >> checked in a few bug fixes to simTBX, but this analysis is unaffected by >> them, so you shouldn't have to update your build to reproduce this. >> >> Cheers, >> >> -James Holton >> MAD Scientist >> >> >> >> _______________________________________________ >> cctbxbb mailing list >> cctbxbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> > ------------------------------------------------------------ > ------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot______ > _________________________________________ > DIALS-support mailing list > DIALS-support(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/dials-support > > >

2 1

indexing success rates
by James Holton 03 Feb '18

03 Feb '18

While testing the new simTBX diffraction image simulator Aaron and I found that some rather beautiful-looking images still won't index. Subsequently, I followed up with mosflm, which does succeed in these cases, even without prior knowledge of the cell or space group. I have now run this 1000 times, varying only the crystal orientation, and found 171 cases of dials.stills_process failing to index the simulated image whereas mosflm had no trouble, and 5 cases where mosflm failed and dials.stills_process succeeded. There were no overlapping cases where both programs failed. I have tarballed up the relevant files here: http://bl831.als.lbl.gov/~jamesh/bugreports/dials_index_020218.tgz The tarball contains a "runme.com" shell script for reproducing these results, and also the tst_nanoBragg_forindex.py jiffy for making a test image given a provided random-number seed. For example: libtbx.python ./tst_nanoBragg_forindex.py random 15 will create a file called noiseimage_001.cbf that dials.stills_process cannot index (at least in my hands). For those of you who don't have mosflm installed, the tarball incluses a copy, and the "autoindex.com" script should run on any linux system. Any ideas as to why there is this discrepancy? Feels like an opportunity for a non-trivial improvement. Something to ponder over the weekend, I suppose. Note that I recently checked in a few bug fixes to simTBX, but this analysis is unaffected by them, so you shouldn't have to update your build to reproduce this. Cheers, -James Holton MAD Scientist

2 1