- cctbxbb - phenix-online.org

Conda dropping Python 3.7 real soon now
by Winter, Graeme (DLSLtd,RAL,LSCI) 16 Sep '22

16 Sep '22

Hi All, Possibly of relevance herein - https://conda-forge.org/docs/user/announcements.html#dropping-python-3-7 "Conda-Forge has been providing support for Python 3.7 for 4 years now. Increasingly projects are moving off it (particularly in the PyData community). With Python 3.11’s release coming around the corner (October 3rd), conda-forge plans to drop Python 3.7 support when Python 3.11 comes out. This will lighten the load on conda-forge infrastructure and make room for the new versions the community would like to support.” ISTR PHENIX is pinned on 3.7 right now because of some non-3.8 compatible code? That could become more of a problem Cheerio Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

Next conda version?
by Luc Bourhis 13 Sep '22

13 Sep '22

Hi, I have just fixed a bug introduced a few years ago that broke some of our code at Bruker. We use conda to add the cctbx to our environment, hence my question: when can the next version be expected on conda-force including that fix? No pressure! I would like to know to plan ahead. Thanks! Best wishes, Luc https://github.com/cctbx/cctbx_project/commit/085f6d8cf57b91dce3a695ebbfbcf… <https://github.com/cctbx/cctbx_project/commit/085f6d8cf57b91dce3a695ebbfbcf…>

4 4

boost python exports
by David Waterman 02 Mar '22

02 Mar '22

Hi folks, I'm trying to create a boost python class that works with either vec3 or mat3 arrays (and I only need those two types). I made a templated class and then exported it twice with different names, for the vec3 and mat3 versions. For example, for mat3: class_<ReconstituteDerivatives< mat3<double> > >("ReconstituteDerivativesMat3", no_init) .def("get_data", &ReconstituteDerivatives< mat3<double> >::get_data) .def("get_indices", &ReconstituteDerivatives< std::size_t >::get_indices); I can create one of these objects from Python >>> foo <dials_refinement_helpers_ext.ReconstituteDerivativesMat3 object at 0x7f3ffa421ac0> but the accessors are not exported properly: >>> foo.get_data() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: No to_python (by-value) converter found for C++ type: scitbx::af::shared<scitbx::mat3<double> > I expected a scitbx_array_family_flex_ext.mat3_double here >>> foo.get_indices() Traceback (most recent call last): File "<stdin>", line 1, in <module> Boost.Python.ArgumentError: Python argument types in ReconstituteDerivativesMat3.get_indices(ReconstituteDerivativesMat3) did not match C++ signature: get_indices(dials::refinement::ReconstituteDerivatives<unsigned long> {lvalue}) and I expected a scitbx_array_family_flex_ext.size_t here. Is it likely there is something simple I am missing, like a #include? I do include flex_wrapper.h. Or perhaps I am just doing something wrong with templated classes and boost python. Looking at other code isn't helping to reveal the mysteries to me yet. Cheers -- David

1 2

DYLD_FALLBACK_LIBRARY_PATH
by Winter, Graeme (DLSLtd,RAL,LSCI) 04 Jan '22

04 Jan '22

Hi Folks, By default cctbx dispatchers on macOS clobber DYLD_FALLBACK_LIBRARY_PATH with some stuff which means that from napari import viewer fails with a huge stack trace ending in OSError: dlopen(libobjc.dylib, 6): image not found If I comment out the clobbering in the dispatcher then everything works fine so I am fairly sure this is the problem Some questions - - why do we do this? - is there a way I can make a dispatcher not to this? or maybe leave the system DYLD… as it was and _append_ to rather than prepend? Seems like a thing I am surprised I have not broken a toe on before now Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 8

Re: [cctbxbb] [git/cctbx_project] master: Reduce2: place NH3 only at residue with resseq 1. (765881e87b)
by Pavel Afonine 10 Nov '21

10 Nov '21

What if chain starts with a number different than 1 (people tend to number chain residues in many different ways, even using negative numbers, and they claim they have valid reasons to do so)? iotbx.pdb.hierarchy has a clear definition of the chain. So I'd suggest adding NH3 to the fist residue of the chain, no matter what that residue number is. Pavel On 11/10/21 14:03, CCTBX commit wrote: > Repository : https://github.com/cctbx/cctbx_project/ > On branch : master > Link : > https://github.com/cctbx/cctbx_project/commit/765881e87b5982c3b68e8dcd8c0e0… > > ------------------------------------------------------------------------ > > > commit 765881e87b5982c3b68e8dcd8c0e00c61c5b5600 > Author: dcliebschner <dcliebschner(a)lbl.gov> > Date: Wed Nov 10 14:03:06 2021 -0800 > > Reduce2: place NH3 only at residue with resseq 1. > > > ------------------------------------------------------------------------ > > > 765881e87b5982c3b68e8dcd8c0e00c61c5b5600 > mmtbx/hydrogens/reduce_hydrogen.py | 19 +- > mmtbx/hydrogens/tst_add_hydrogen.py | 196 ++++++++--------- > mmtbx/hydrogens/tst_add_hydrogen_2.py | 383 > ++++++++++++++++------------------ > 3 files changed, 282 insertions(+), 316 deletions(-) > > Diff suppressed because of size. To see it, use: > > git diff-tree --root --patch-with-stat --no-color > --ignore-space-at-eol --textconv --ext-diff --cc > 765881e87b5982c3b68e8dcd8c0e00c61c5b5600 > ------------------------------------------------------------------------ > > To unsubscribe from the CCTBX-COMMIT list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCTBX-COMMIT&A=1 > <https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCTBX-COMMIT&A=1> >

1 0

accessing attributes of PHIL parameters
by Robert Oeffner 15 Oct '21

15 Oct '21

Hi, Most likely a trivial question but if I have a PHIL object such as one based on the string arrayinfo_phil_str = """ selected_info { span = True .type = bool .short_caption = "Span" minmax_data = True .type = bool .short_caption = "min,max data" } """ I can then get the PHIL definitions span and minmax_data as python values like: from libtbx.phil import parse parse(arrayinfo_phil_str).extract().selected_info.span parse(arrayinfo_phil_str).extract().selected_info.minmax_data But how do I access the values of the short_caption attribute of these definitions? Does that make sense or have not understood correctly how PHIL works? Thanks, Rob -- This email has been checked for viruses by AVG. https://www.avg.com

2 1

Project Scientist and Postdoctoral research positions in XFEL, ML, and material science computational methods
by Aaron Brewster 18 Aug '21

18 Aug '21

Hello, I'm excited to announce that two positions are immediately open in my group in the fields of material science, serial X-ray crystallography, and machine learning. Links to postings: Computational Project Scientist - https://lbl.taleo.net/careersection/2/jobdetail.ftl?job=93507 Computational Postdoctoral Scholar - https://lbl.taleo.net/careersection/2/jobdetail.ftl?job=93506 In these exciting roles, you will work in the Molecular Biophysics and Integrated Bioimaging Division on a project that will research adding advanced spatial and symmetry aware neural network training methods to small-molecule chemical crystallography and material science data processing. This will use AI methods developed as part of a close collaboration between teams at Lawrence Berkeley National Labs (LBNL), University of Connecticut, and MIT. The larger group is researching crystal engineering of hybrid coordination polymers towards achieving rational control over the optoelectronic properties of low-dimensional nanostructures. These positions are focused on the chemical crystallography aspect of the program, but the successful candidates will be expected to participate in all aspects of the program, including data collection at X-ray free electron lasers worldwide, data analysis on supercomputing facilities, structure solution, research on AI driven structure prediction, paper writing, and software management. Both roles are 3 year appointments. This work will be part of the Department of Energy Integrated Computational and Data Infrastructure for Science <https://insidehpc.com/2021/08/doe-funds-15m-for-integrated-computational-an…> Research program. Specific details follow. Berkeley Lab is committed to Inclusion, Diversity, Equity and Accountability (IDEA) <https://diversity.lbl.gov/ideaberkeleylab/> and strives to continue building community with these shared values and commitments. Berkeley Lab is an Equal Opportunity and Affirmative Action Employer. We heartily welcome applications from women, minorities, veterans, and all who would contribute to the Lab’s mission of leading scientific discovery, inclusion, and professionalism. In support of our diverse global community, all qualified applicants will be considered for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, age, or protected veteran status. *Computational Project Scientist* *What You Will Do:* - Design and train neural networks that allow determination of the unit cell and crystal orientations of small molecule crystals from sparse, randomly oriented diffraction patterns. This will involve curating training sets from simulated datasets and from existing and new experimental data that will be collected. - Research methods to scale training and use of these neural networks on multiple nodes in high performance computing environments. - Lead data collection and processing at X-ray free electron sources. - Maintain a detailed record of work performed. - Document and manage code developed using standard source control methods. - Publish in peer-reviewed journals. - Present research at seminars and conferences. - Assist Principal Investigator in developing grant proposals. - Document and maintain existing and new data processing software, including the cctbx build system. *What is Required:* - 2-5 years of relevant experience beyond the highest degree in Physics, Statistics, Crystallography, Artificial Intelligence, or a related field. - A strong background in Python, C++, or an equivalent platform for scientific computing. - A track record of critical thinking, problem solving, and mathematical intuition, demonstrated through publications and presentations. *Desired Qualifications:* - Familiarity with general Python scientific data packages such as NumPy, SciPy, and pandas. - Versed in modern coding practices for scientific computing, including algorithm design and evaluation, source code management under a revision control system like git, and incorporation of readable documentation and comments. - Expertise in computational approaches in at least one of the areas related to the research is required, including but not limited to small-molecular crystallography, structure solution and refinement, unit cell determination, neural network design, training, and evaluation, high speed computing, image analysis, large dataset analysis, graph theory, error propagation and statistics, signal processing, and GPU acceleration. - Familiarity with small-molecule material design and synthesis. - Strong analytical and organizational skills. - Excellent interpersonal, oral, and written communication skills. - Ability to work independently as well as part of a diverse team. *Computational **Postdoctoral Scholar* *What You Will Do:* - Design and train neural networks to model diffraction from small molecule crystals. This will involve curating training sets from simulated datasets and from existing and new experimental data that will be collected. - Assist in data collection and processing at X-ray free electron sources. - Maintain a detailed record of work performed. - Document and manage code developed using standard source control methods. - Publish in peer-reviewed journals. - Present research at seminars and conferences. * What is Required:* - PhD degree in Physics, Statistics, Crystallography, Artificial Intelligence, or a related field. - Experience in computational approaches in at least one of the areas related to the research, including but not limited to small-molecular crystallography, structure solution and refinement, unit cell determination, neural network design, training, and evaluation, high speed computing, image analysis, large dataset analysis, graph theory, error propagation and statistics, signal processing, and GPU acceleration. - Strong analytical and organizational skills. - Excellent interpersonal, oral, and written communication skills. - Ability to work independently as well as part of a diverse team. * Desired Qualifications:* - Familiarity with Python, C++, and general scientific data packages such as NumPy, SciPy, and pandas. - Familiarity with small-molecule material design and synthesis. Please do not hesitate to reach out to me for more information. Thank you, Aaron S. Brewster, PhD Computational Research Scientist, Molecular Biophysics and Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720

1 0

DIALS releases in PHENIX
by Winter, Graeme (DLSLtd,RAL,LSCI) 15 Jul '21

15 Jul '21

Hi Folks Following from - # select dials-3.5 branch + # select dials-3.2 branch def _add_git(self, module, parameters, destination=None): super(PhenixBuilder, self)._add_git(module, parameters, destination) if (module == 'dials' or module == 'dxtbx' or module == 'xia2') and self.python3: @@ -2339,9 +2339,9 @@ class PhenixBuilder(CCIBuilder): if module == 'dxtbx': self.add_step(self.shell(command=['git', 'remote', 'set-url', 'origin', 'https://github.com/dials/dxtbx.git'], workdir=workdir)) self.add_step(self.shell(command=['git', 'fetch', 'origin'], workdir=workdir)) - self.add_step(self.shell(command=['git', 'checkout', 'dials-2.2'], workdir=workdir)) + self.add_step(self.shell(command=['git', 'checkout', 'dials-3.2'], workdir=workdir)) self.add_step(self.shell( - command=['git', 'branch', '--set-upstream-to=origin/dials-2.2', 'dials-2.2'], + command=['git', 'branch', '--set-upstream-to=origin/dials-3.2', 'dials-3.2'], workdir=workdir)) I was wondering - what are the constraints in PHENIX builds which mean bouncing around between extremely different DIALS releases is necessary? I note the shift from 2.2 to 3.2 would give a significant change in behaviour to the end user (who ultimately we may end up having to help with debugging) If we knew better why these changes were coming through, maybe we could help to keep PHENIX up to date? I would draw comparisons with CCP4 however they have been stuck in 2.2-series forever now as they’re still not able to support Python3 - will be in CCP4 8.x which will be ready any time now. I only mention this as it would be really good to start eliminating obsolete versions from distribution as “please go to dials.github.io and install a non-ancient version” is annoying advice to an end user with a fresh install of PHENIX or CCP4. Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 2

Is there a schedule for the python3 migration or can cctbx.python already be built to be/use a python3 interpreter?
by Jan M. Simons 12 Jul '21

12 Jul '21

Cheers, I'm getting more and more concerned with the state of cctbx and the impending removal of python2 from quite a lot of major linux distributions in the near future. So far I have not managed to find documentation on how to build the cctbx to then use a python3 interpreter for the cctbx.python script, but I would really like to have this, as I'd have to check/port my code that's using cctbx modules (mostly smtbx and xrax) to python3 as well, so that it does not get obsoleted. So is it possible to build cctbx as a python3 module by now and if so how would one be able to do this? Thanks for your support Jan

5 10

WSL2 (was: Windows icc builds)
by Kay Diederichs 27 May '21

27 May '21

Hi Graeme, I have no solution for this problem but I want to bring up the fact that WSL2 is very easy to install on current Windows10, and lets one run the Linux binaries (including graphics if you install MobaXterm which has an Xserver) quite efficiently. I have not run into a single problem when trying WSL2 with XDS and XDSGUI; my writeup is at https://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Installation#WS… . I think those who struggle with developing for Windows, including you, the Phenix team and the CCP4 team, might be able to save a lot of time/effort (and those who just use Windows would benefit in the long run by getting used to a Linux environment). Microsoft announced WSLg (the g stands for graphics) which provides an Xserver, but this currently requires Windows-Build 21362+ (Windows Insider Program). Give it a try! Just my 2 cents, Kay > Date: Wed, 26 May 2021 18:57:27 +0000 > From: "Winter, Graeme (DLSLtd,RAL,LSCI)" <Graeme.Winter(a)diamond.ac.uk> > To: cctbx mailing list <cctbxbb(a)phenix-online.org> > Subject: [cctbxbb] Windows icc builds > Message-ID: <6210D39B-96F7-493F-98D4-D1EDFCBE09A5(a)diamond.ac.uk> > Content-Type: text/plain; charset="utf-8" > > HI Folks > > Following from a DIALS issue about it being slow on Windows > > https://github.com/dials/dials/issues/1666 > > I have fallen into a rabbit hole and I am trying to build DIALS (and hence cctbx) on Windows with the intel compilers from oneAPI - however I keep stumbling over linking errors with e.g. undefined symbol __floor > > Should I abandon hope or has someone done this? > > I know this is not a new path to tread as windows_icc was a defined combination on the SConscripts > > I had had to hack some bits around a little presumably to accommodate 2021 versions of icc compilers, but this felt like I was going down a bad path > > Would welcome input from anyone who can help - currently windows DIALS is a pretty poor relation > > It is noteworthy however that parallel operation does now appear to work on Windows with easy mp :-) > > In other news - has anyone considered adding clang support with clang coming from conda? Could make for more consistency in the builds? I had a look at how one may go about doing this but rapidly found myself in SCons hell. > > Thanks in advance for any help & all the best Graeme > -- Kay Diederichs http://strucbio.biologie.uni-konstanz.de email: Kay.Diederichs(a)uni-konstanz.de Tel +49 7531 88 4049 Fachbereich Biologie, Universität Konstanz, Box M647, D-78457 Konstanz This e-mail is digitally signed. If your e-mail client does not have the necessary capabilities, just ignore the attached signature "smime.p7s".

2 1