- cctbxbb - phenix-online.org

Windows icc builds
by Winter, Graeme (DLSLtd,RAL,LSCI) 27 May '21

27 May '21

HI Folks Following from a DIALS issue about it being slow on Windows https://github.com/dials/dials/issues/1666 I have fallen into a rabbit hole and I am trying to build DIALS (and hence cctbx) on Windows with the intel compilers from oneAPI - however I keep stumbling over linking errors with e.g. undefined symbol __floor Should I abandon hope or has someone done this? I know this is not a new path to tread as windows_icc was a defined combination on the SConscripts I had had to hack some bits around a little presumably to accommodate 2021 versions of icc compilers, but this felt like I was going down a bad path Would welcome input from anyone who can help - currently windows DIALS is a pretty poor relation It is noteworthy however that parallel operation does now appear to work on Windows with easy mp :-) In other news - has anyone considered adding clang support with clang coming from conda? Could make for more consistency in the builds… I had a look at how one may go about doing this but rapidly found myself in SCons hell. Thanks in advance for any help & all the best Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

3 2

Building with system compilers
by Winter, Graeme (DLSLtd,RAL,LSCI) 04 May '21

04 May '21

Hi Folks Recent macOS / xcode update broke the version of clang in conda so I can no longer build (SCons decides that the compilers don’t support C++11) I can build just fine, and everything works and all the tests pass However if I try and recompile anything SCons swears at me and dies as it decides that the perfectly valid C++11 compilers I have don’t compile C++11 because … some reason which makes no sense to me. Here is content of config.log Grey-Area build :( $ cat config.log file /Users/graeme/git/dials/modules/cctbx_project/libtbx/SConscript,line 865: Configure(confdir = .sconf_temp) scons: Configure: ".sconf_temp/conftest_171626668f451c9275a8effd06a8fea3_0.cpp" is up to date. scons: Configure: The original builder output was: |.sconf_temp/conftest_171626668f451c9275a8effd06a8fea3_0.cpp <- | |#include <iostream> | |int main() { | | std::cout << "{"; | | #ifndef __llvm__ | | #define __llvm__ "" | | #endif | | std::cout << "\"llvm\":" << __llvm__ << ", "; | | #ifndef __clang__ | | #define __clang__ "" | | #endif | | std::cout << "\"clang\":" << __clang__ << ", "; | | #ifndef __clang_major__ | | #define __clang_major__ "" | | #endif | | std::cout << "\"clang_major\":" << __clang_major__ << ", "; | | #ifndef __clang_minor__ | | #define __clang_minor__ "" | | #endif | | std::cout << "\"clang_minor\":" << __clang_minor__ << ", "; | | #ifndef __clang_patchlevel__ | | #define __clang_patchlevel__ "" | | #endif | | std::cout << "\"clang_patchlevel\":" << __clang_patchlevel__ << ", "; | | #ifndef __GNUC__ | | #define __GNUC__ "" | | #endif | | std::cout << "\"GNUC\":" << __GNUC__ << ", "; | | #ifndef __GNUC_MINOR__ | | #define __GNUC_MINOR__ "" | | #endif | | std::cout << "\"GNUC_MINOR\":" << __GNUC_MINOR__ << ", "; | | #ifndef __GNUC_PATCHLEVEL__ | | #define __GNUC_PATCHLEVEL__ "" | | #endif | | std::cout << "\"GNUC_PATCHLEVEL\":" << __GNUC_PATCHLEVEL__ << ", "; | | #ifndef __clang_version__ | | #define __clang_version__ "" | | #endif | | std::cout << "\"clang_version\": \"" << __clang_version__ << "\", "; | | #ifndef __VERSION__ | | #define __VERSION__ "" | | #endif | | std::cout << "\"VERSION\": \"" << __VERSION__ << "\", "; | | std::cout << "}"; | | return 0; | |} | scons: Configure: ".sconf_temp/conftest_171626668f451c9275a8effd06a8fea3_0.o" is up to date. scons: Configure: The original builder output was: |/usr/bin/c++ -o .sconf_temp/conftest_171626668f451c9275a8effd06a8fea3_0.o -c .sconf_temp/conftest_171626668f451c9275a8effd06a8fea3_0.cpp | scons: Configure: ".sconf_temp/conftest_171626668f451c9275a8effd06a8fea3_0_79477a053ab4616bb93a6d1a8e51fb1f" is up to date. scons: Configure: The original builder output was: |/usr/bin/c++ -o .sconf_temp/conftest_171626668f451c9275a8effd06a8fea3_0_79477a053ab4616bb93a6d1a8e51fb1f .sconf_temp/conftest_171626668f451c9275a8effd06a8fea3_0.o | scons: Configure: ".sconf_temp/conftest_171626668f451c9275a8effd06a8fea3_0_79477a053ab4616bb93a6d1a8e51fb1f.out" is up to date. scons: Configure: The original builder output was: |.sconf_temp/conftest_171626668f451c9275a8effd06a8fea3_0_79477a053ab4616bb93a6d1a8e51fb1f > .sconf_temp/conftest_171626668f451c9275a8effd06a8fea3_0_79477a053ab4616bb93a6d1a8e51fb1f.out | file /Users/graeme/git/dials/modules/cctbx_project/libtbx/SConscript,line 1143: Configure(confdir = .sconf_temp) scons: Configure: ".sconf_temp/conftest_e73942b1e04111dec1918a18e43ee693_0.cpp" is up to date. scons: Configure: The original builder output was: |.sconf_temp/conftest_e73942b1e04111dec1918a18e43ee693_0.cpp <- | |#include <iostream> | |#include <vector> | |#include <array> | |#include <utility> | |std::vector<int> func(std::vector<int> && in) {std::vector<int> r = std::move(in); return r;} | |int main() {std::vector<int> x(9); for(auto & i : x) {i=7; std::cout << i << std::endl;} | | x = func(std::vector<int>(2)); return 0;} | | | scons: Configure: ".sconf_temp/conftest_e73942b1e04111dec1918a18e43ee693_0.o" is up to date. scons: Configure: The original builder output was: |/usr/bin/c++ -o .sconf_temp/conftest_e73942b1e04111dec1918a18e43ee693_0.o -c -std=c++11 -stdlib=libc++ -mmacosx-version-min=10.9 -I/Users/graeme/git/dials/conda_base/include -I/Users/graeme/git/dials/modules -I/Users/graeme/git/dials/modules/cctbx_project -Iannlib_adaptbx/include -I/Users/graeme/git/dials/modules/annlib_adaptbx/include -I/Users/graeme/git/dials/modules/cctbx_project/annlib_adaptbx/include .sconf_temp/conftest_e73942b1e04111dec1918a18e43ee693_0.cpp | /usr/bin/c++ -o .sconf_temp/conftest_e73942b1e04111dec1918a18e43ee693_0_3b63d4473a8f41665bd62af890bf3763 -w -stdlib=libc++ -mmacosx-version-min=10.9 -std=c++11 -stdlib=libc++ -mmacosx-version-min=10.9 .sconf_temp/conftest_e73942b1e04111dec1918a18e43ee693_0.o -L/Users/graeme/git/dials/conda_base/lib -Llib -L/Users/graeme/git/dials/modules/lib -L/Users/graeme/git/dials/modules/cctbx_project/lib clang: error: no such file or directory: '.sconf_temp/conftest_e73942b1e04111dec1918a18e43ee693_0.o' clang: error: no input files Key point is this: I can blow away the build directory and it all compiles correctly, but if I touch a file and type make I get build failures as so Has anyone a clue about this? Should I add an issue on GitHub? Thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

Question/a possible typo in sampling backrub and side chain conformations in mmtbx
by Swati Jain 22 Apr '21

22 Apr '21

Hi, Not sure if this message was posted earlier or not, that is why I am reposting it. I had a couple of questions about sampling backrub and sidechains using the building or refinement modules in cctbx_project/mmtbx. 1) In the file: mmtbx.building.alternate_conformations.conformer_generation.py In the function def __call__: the if statements for checking whether the sampled side chain conformation is a rotamer outlier or not, the check in the if statement is for "selfUTLIER". Also in the function *set_up_backrub*(self): the atom "O" for the previous residue is not included in the backrun motion, as a result this atom is not moved when the backrub is implemented. I was just wondering if this is a mistake (possibly all instances of "O" being replaced by "self" which would explain both these things, and some other instances throughout the file), or is this code old and not used anymore? 2) When trying to run fit_residues.py and tst_fit_residues* scripts in mmtbx.refinement.real_space, I encountered errors mainly related to chem_data/rotamer_chi_angles/ not being present. Looks like this data is not distributed with cctbx. Is this correct, or maybe there is some other known issue ? I look forward to hearing from you. Thanks, Swati

5 7

Question/a possible typo in sampling backrub and side chain conformations in mmtbx
by Swati Jain 20 Apr '21

20 Apr '21

Hi, I had a couple of questions about sampling backrub and sidechains using the building or refinement modules in cctbx_project/mmtbx. 1) In the file: mmtbx.building.alternate_conformations.conformer_generation.py In the function def __call__: the if statements for checking whether the sampled side chain conformation is a rotamer outlier or not, the check in the if statement is for "selfUTLIER". Also in the function *set_up_backrub*(self): the atom "O" for the previous residue is not included in the backrun motion, as a result this atom is not moved when the backrub is implemented. I was just wondering if this is a mistake (possibly all instances of "O" being replaced by "self" which would explain both these things, and some other instances throughout the file), or is this code old and not used anymore? 2) When trying to run fit_residues.py and tst_fit_residues* scripts in mmtbx.refinement.real_space, I encountered errors mainly related to chem_data/rotamer_chi_angles/ not being present. Looks like this data is not distributed with cctbx. Is this correct, or maybe there is some other known issue ? I look forward to hearing from you. Thanks, Swati

2 1

MTZ object creation bug or feature?
by Robert Oeffner 30 Mar '21

30 Mar '21

I wonder if anyone has noticed the behaviour that the creation of columns in an mtz object seems to suffer from a problem. The script below creates an mtz object from three miller arrays of intensities, amplitudes and anomalous differences together with their sigmas on 5 reflections. The intention is that the mtz object should also consist of three pairs of columns ( I,SIGI and F,SIGF and DANO,SIGDANO). However the script shows that if the miller array with anomalous differences is added as the last miller array then the resulting mtz object is messed up: It has grouped F,SIGF and DANO,SIGDANO together as 10 reflections instead of just 5. Only if the miller array with anomalous differences is added as the second miller array after the intensities does the resulting mtz object contain the three distinct pairs of columns I,SIGI and F,SIGF and DANO,SIGDANO as intended each with 5 reflections. from cctbx.xray import observation_types from cctbx.array_family import flex from cctbx import miller from cctbx import crystal xs = crystal.symmetry(unit_cell=(50,50,40, 90,90,120), space_group_symbol="P3 1") mi = flex.miller_index([(1,-2,3), (0,0,-3), (1, 2, 3), (0, 1, 2), (1, 0, 2)]) ma1 = miller.array( miller.set(xs, mi), flex.double( [ 11.2, 6.3, 26.1, 14.9, 2.4 ] ), sigmas=flex.double( [ 13.6, 6.3, 24.9, 6.2, 11.1 ] ) ) ma2 = miller.array( miller.set(xs, mi), flex.double( [ 8.2, 26.3, 2.1, 4.9, 32.4 ] ), sigmas=flex.double( [ 3.6, 76.3, 2.9, 61.2, 1.1 ] ) ) ma3 = miller.array( miller.set(xs, mi), flex.double( [ 8.2, 26.3, 2.1, 4.9, 32.4 ] ), sigmas=flex.double( [ 3.6, 76.3, 2.9, 61.2, 1.1 ] ) ) mtz1 = ma1.as_mtz_dataset(column_root_label = "I", column_types = "JQ") mtz1.add_miller_array(ma2, column_root_label = "F", column_types = "FQ") mtz1.add_miller_array(ma3, column_root_label = "DANO", column_types = "DQ") mtz1.mtz_object().as_miller_arrays() # unexpected F,SIGF,DANO,SIGDANO array looks messed up mtz2 = ma1.as_mtz_dataset(column_root_label = "I", column_types = "JQ") mtz2.add_miller_array(ma3, column_root_label = "DANO", column_types = "DQ") mtz2.add_miller_array(ma2, column_root_label = "F", column_types = "FQ") mtz2.mtz_object().as_miller_arrays() # looks correct Is this a bug, a well known feature or am I simply doing something wrong? Many thanks, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus The Keith Peters Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 -- This email has been checked for viruses by AVG. https://www.avg.com

3 3

Generate mtz from pdb
by Academic Research 06 Mar '21

06 Mar '21

Dear All, Is it possible to generate an .mtz FFT file from a .pdb file in cctbx? the equivalent of "sfall XYZIN filename.pdb HKLOUT filename.mtz" in CCP4? Regards, AC

2 1

testing newer versions of boost
by Robert Oeffner 12 Feb '21

12 Feb '21

Hi, I'm testing newer versions of boost than 1.70 currently distributed with CCTBX. I can't remember if I have to do something special other than unpacking the zip file, say boost_1_74_0.zip, that I download from boost.org and put it into the modules directory for my cctbx build. But I get several errors during build like: C:\Users\Oeffner\OeffnerStuff\Work\PhenixVoyager\modules\boost\libs\filesystem\src\codecvt_error_category.cpp(71): error C2491: 'boost::filesystem::codecvt_error_category': definition of dllimport function not allowed I'm have tried with the VS2015, VS2017 and the VS2019 compiler and they all complain. So I'm wondering if there are some other magic invocations that have to be done before building with a new version of boost. Thanks in advance for any suggestions, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus The Keith Peters Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 -- This email has been checked for viruses by AVG. https://www.avg.com

2 2

Find nearest orthonormal matrix
by Robert Oeffner 25 Jan '21

25 Jan '21

Does the CCTBX has a routine for that? I didn't find anything in the scitbx module. But I came across a procedure online here: http://people.csail.mit.edu/bkph/articles/Nearest_Orthonormal_Matrix.pdf If not already in the CCTBX I plan to include a pythonic version of it within the next few days. Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus The Keith Peters Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 -- This email has been checked for viruses by AVG. https://www.avg.com

3 2

(not really cctbx: PHENIX release bundle content: .o files)
by Winter, Graeme (DLSLtd,RAL,LSCI) 25 Jan '21

25 Jan '21

Hi All, Could not think of a better place to ask this question Installed a PHENIX update this morning and noticed it is quite a substantial download - hard to avoid, in this modern day. While unpacking I noticed it contained a lot of object files - totting it up seems to be ~ 450 MB of .o files which are probably not useful for the average end user. Just wondered, are these worth stripping out before making the release bundles? I note the same is true for DIALS so I am not claiming some moral high ground here & also https://github.com/dials/dials/discussions/1563 … looks like my colleagues decided to pick up on this & happened to agree Just thought worth asking Cheers Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Space-saving tips
by Tristan Croll 25 Jan '21

25 Jan '21

Hi all, Following on from Graeme's email about .o files, another way to quite dramatically reduce the size of the distribution (with little impact on code complexity or performance) would be to keep all the various text-based data files in .zip format. With modern Python fetching the data from these is little different from working with uncompressed files anyway (1-2 extra lines of code, generally negligible runtime cost). A couple of snippets as an example from ISOLDE (where I keep all MD ligand definition files in a .zip): To get a list of the file contents: def _ligand_db_from_zip(self, ligand_zip): from zipfile import ZipFile import os namelist = [] with ZipFile(ligand_zip) as zf: for fname in zf.namelist(): name, ext = os.path.splitext(fname) if ext.lower() == '.xml': namelist.append(name) return namelist To read a given file from the zip as needed: if ligand_db is not None: zip, namelist = ligand_db if name in namelist: from zipfile import ZipFile logger.info('Loading residue template for {} from internal database'.format(name)) with ZipFile(zip) as zf: with zf.open(name+'.xml') as xf: forcefield.loadFile(xf) Even just doing this for the CaBLAM contour data cuts about 160 MB off the size of the uncompressed distribution. Best regards, Tristan

1 0