- cctbxbb - phenix-online.org

Re: [cctbxbb] sequential revision number from git repo
by markus.gerstel＠diamond.ac.uk 06 Apr '16

06 Apr '16

Hi Johan, No, not really. .gitversion is not in the repository and only kept (and updated) locally. Basically the logic is: Can I determine the current version number (ie do I have git installed and the repository present) - if so, write .gitversion. Otherwise, attempt to read .gitversion and use the information therein (this covers the route via the installer). Hence, if you download the zip file frm github you will not have a .gitversion file. But really in this case we can assume that the user knows what they're doing. -Markus On 5 Apr 2016 16:34, "Hattne, Johan" <hattnej(a)janelia.hhmi.org> wrote: Would this not mean that .gitversion is updated on every commit, at least if as in our case, it not only records the release version but also the current commit? @Rob: Apologies for my liberal use of “hook”. What our build system does is that it generates what essentially amounts to Markus’s .gitversion whenever we run “make dist” (using “git describe” or some such, it need not involve the network). There is little need for this file when compiling from a sources under git control, because it can be done on the fly. // Cheers; Johan > On Apr 5, 2016, at 09:47, markus.gerstel(a)diamond.ac.uk wrote: > > Hi Johan, > > We generate a file named '.gitversion' that lives in the main directory. That way the version number survives being packed up in an installer. > If you download the code via the "Download ZIP" button on Github there is no way (as far as I know) to recover any version information, so we fall back to a default of "DIALS 1.dev" ("DIALS 1.1" for the release branch) > > -Markus > > -----Original Message----- > From: cctbxbb-bounces(a)phenix-online.org [mailto:[email protected]] On Behalf Of Hattne, Johan > Sent: 05 April 2016 14:21 > To: cctbx mailing list > Subject: Re: [cctbxbb] sequential revision number from git repo > > Hi Markus; > > I didn’t actually read the code, but it looks like we’re doing something similar over here, and I find the the tricky bit is how to deal with any kind of releases that don’t have all the little git bits. We sort that out by having hooks in the “make dist” equivalent, which basically generate a version file that is included in the release tarball. > > However, it was recently pointed out to us that that all falls apart if someone uses the "Download ZIP” button from the code tab, because then the hooks are not run. I haven’t even attempted to RTFM around this, and if you or anybody else here has a solution I won’t have to. > > Do you by any chance? > > // Cheers; Johan > > >> On Apr 5, 2016, at 08:55, markus.gerstel(a)diamond.ac.uk wrote: >> >> Hi Rob, >> >> This is close to what we do with DIALS. >> If you run dials.version you may get e.g. >> DIALS 1.dev.132-g7042805 >> So this is the 132nd commit after the commit we tagged "1.dev", which >> happened when we branched for the 1.1 stable release so we could keep >> the version numbers sufficiently different >> >> If you run DIALS 1.1, for example the version from CCP4, you might get >> DIALS 1.1.3-gb5528a5-release The 3rd commit on the 1.1 release branch. >> >> Same with xia2: >> XIA2 0.4.0.301-g7f16ab6 >> >> In xia2 we also add the branch name if it is not 'master', so the version distributed with DIALS 1.1 is: >> XIA2 0.4.0.289-g2a1e557-dials-1.1 >> >> We kept the commit ID in - and I would recommend to keep it with cctbx, too. >> The reason for this is: >> - The number on its own does not uniquely identify a commit, as you >> might be on a different branch (although we do put in the branch >> names, too) >> - Because then I can go to >> https://github.com/dials/dials/commit/b5528a5 and see everything about >> that commit >> - Because then I can do git checkout b55528a5 and instantly have that particular code version on my machine to fiddle with things. >> >> To see the magic happen, check >> https://github.com/dials/dials/blob/master/util/version.py or >> https://github.com/xia2/xia2/blob/master/XIA2Version.py >> >> -Markus >> >> >> -----Original Message----- >> From: cctbxbb-bounces(a)phenix-online.org >> [mailto:[email protected]] On Behalf Of R.D. Oeffner >> Sent: 05 April 2016 13:38 >> To: cctbxbb(a)phenix-online.org >> Subject: [cctbxbb] sequential revision number from git repo >> >> Hi, >> >> We are slowly moving phaser svn to git. One issue that we would like to address is if there is a practical way of conveying a linear commit history from the git repo. >> >> One approach is to just count the number of revisions that was made to the master branch in the git repo with "git rev-list --count HEAD" >> and then pretend that number is a revision number. This enables us to tell users "This bug was fixed after revision 6456 of phaser" rather than "This bug was fixed after commit 29b0a1c... of phaser". Is there any pitfalls with this approach? >> >> What workarounds do other people using git come up with? >> >> >> >> Rob >> >> >> -- >> Robert Oeffner, Ph.D. >> Research Associate, The Read Group >> Department of Haematology, >> Cambridge Institute for Medical Research University of Cambridge >> Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road >> Cambridge CB2 0XY >> >> www.cimr.cam.ac.uk/investigators/read/index.html<http://www.cimr.cam.ac.uk/investigators/read/index.html> >> tel: +44(0)1223 763234 >> mobile: +44(0)7712 887162 >> _______________________________________________ >> cctbxbb mailing list >> cctbxbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> -- >> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. >> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. >> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. >> Diamond Light Source Limited (company no. 4375679). Registered in >> England and Wales with its registered office at Diamond House, Harwell >> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United >> Kingdom >> >> >> _______________________________________________ >> cctbxbb mailing list >> cctbxbb(a)phenix-online.org >> http://phenix-online.org/mailman/listinfo/cctbxbb > > Research Specialist @ Gonen Lab ____________________________________________________ > Janelia Research Campus * 19700 Helix Drive Ashburn, VA 20147 * +1 (571) 209-4000 extension 3376 > > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/cctbxbb > > -- > This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. > Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. > Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. > Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom > > _______________________________________________ > cctbxbb mailing list > cctbxbb(a)phenix-online.org > http://phenix-online.org/mailman/listinfo/cctbxbb Research Specialist @ Gonen Lab ____________________________________________________ Janelia Research Campus * 19700 Helix Drive Ashburn, VA 20147 * +1 (571) 209-4000 extension 3376 _______________________________________________ cctbxbb mailing list cctbxbb(a)phenix-online.org http://phenix-online.org/mailman/listinfo/cctbxbb

3 3

CCTBX.XFEL problems with synchrotron stills
by Schulz, Eike-Christian 06 Apr '16

06 Apr '16

Dear all, I am new to serial crystallography and trying to process still images from a synchrotron source (Pilatus 6M detector) using CCTBX. Obviously the process is not glitch-free and I was hoping to get some help here. The indexing routine starts, indexes the images correctly and and produces a log file but no .pickle files are created in the results/000/ directory as described in the wiki entry. I assume that there is a software module that is not properly loaded during the process since I get following error message during indexing: ---------BEGIN Integrate one frame 75 warsaw0048_00075.pickle EFFECTIVE TILING 0 0 2282 2282 Indexing error: No module named cxi_user Is there a way to find out if all dependencies are matched ? I suppose CCTBX.XFEL depends on some (specific) python libraries … ? Or am I completely wrong and the error is something different ? With best regards Eike

3 4

sequential revision number from git repo
by R.D. Oeffner 05 Apr '16

05 Apr '16

Hi, We are slowly moving phaser svn to git. One issue that we would like to address is if there is a practical way of conveying a linear commit history from the git repo. One approach is to just count the number of revisions that was made to the master branch in the git repo with "git rev-list --count HEAD" and then pretend that number is a revision number. This enables us to tell users "This bug was fixed after revision 6456 of phaser" rather than "This bug was fixed after commit 29b0a1c... of phaser". Is there any pitfalls with this approach? What workarounds do other people using git come up with? Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 mobile: +44(0)7712 887162

3 6

changes to cctbx we have in ccp4
by Marcin Wojdyr 04 Apr '16

04 Apr '16

Hi All, we have a bunch of changes to cctbx in ccp4. If any of them are of general interest I can push them to svn. They'd need some cleanup before applying, so I'm not attaching any patches for now. - option to use external Boost. It's the same what debian and gentoo distros are doing when packaging cctbx. We change a few lines in the scons scripts and rm cctbx_project/boost/* (but the boost directory itself must be present). We could add an option for it (--use-system-boost ?). - removing -funroll-loops - this option must be a historical leftover. Generally it's better to let compiler decide because unrolling loops may make the program slower as well as faster. I tested it on Phaser a couple years ago. The only noticeable effect of this flag was much bigger binary. - extend --use-environment-flags to handle also CC and CXX. With most of makefiles / build scripts, if the user has multiple compilers installed he selects one with CC=gcc-mp-5 etc. Then if the build script really needs to differentiate between compilers it tries to determine what kind of compiler it is. In cctbx it's the other way around - the --compiler flag tells what kind of compiler it is and the build script guesses executable names. For ccp4 I added in two places in libtbx/SConscript: + if libtbx.env.build_options.use_environment_flags: + if 'CC' in os.environ: cc = os.environ['CC'] + if 'CXX' in os.environ: cxx = os.environ['CXX'] so for _some_ platform/compiler combinations it's possible to use compilers with non-default names. Useful for systems with multiple GCC versions. - probe and reduce are called in mmtbx through phenix wrappers (phenix.probe and phenix.reduce). We prefer to call them directly, but I guess it's not to be changed in cctbx. But after changing the name I also had to remove error checking in mmtbx/validation/clashscore.py that checks return code from phenix.reduce. Reduce itself quite often returns non-zero, it's just that the phenix.reduce wrapper ignores it and always returns 0 (probably not intentionally). - cctbx_project/spotfinder/SConscript - on Windows we have to link dxtbxspotfinder with spotfinder. That's the only such problem when compiling cctbx. OTOH it apparently works for everyone else. I don't know why. It must have something to do with us using MinGW. - A couple of #ifdefs for compiling with MinGW (annlib and gltbx) Marcin

2 2

Berkeley Server Downtime (3/30/2016)
by Billy Poon 30 Mar '16

30 Mar '16

Dear all, On Wednesday, March 30, our servers will be shutdown starting at 7:00 am Pacific for scheduled maintenance in our server room. This will affect, - The Phenix web site (http://phenix-online.org/) - The DIALS West web site (http://dials.lbl.gov/) - The CCI web site (http://cci.lbl.gov/) - The Structural Biology Technology Portal (http://technology.sbkb.org/) We expect that the servers will be available by 5 pm Pacific at the latest. Bug reports and emails requesting help should be queued and we will receive them once the servers are back online. We apologize for any inconvenience. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 1

new parallel_map implementation
by Gabor Bunkoczi 29 Mar '16

29 Mar '16

Hi, I have switched over libtbx.easy_mp.parallel_map to use new code that is not susceptible to deadlock when large amounts of data are returned from a worker process. This also means that parameters added to parallel_map to prevent exactly this are no longer necessary (but will be respected), although unlikely to influence performance so much to warrant changing existing code. I have run all tests, but unsure how well this gets tested by regression test, so please let me know if you come across something that is amiss. BW, Gabor

1 0

Troublesome Hall Symbols in CIF
by Luc Bourhis 21 Mar '16

21 Mar '16

Hi, Consider the following two Hall symbols: (1) -C 2y (x,y,-x+z) (2) -I 2y They represent the same setting of the same spacegroup, so in a perfect world the fact that the sgtbx comes with (1) from some list of operators should not matter. Unfortunately, (1) is rejected by CheckCIF right now. Reminder: every single small molecule publication in Acta Cryst. must pass CheckCIF. On the contrary, CheckCIF accepts (2). Yes, I know, this is a bug in CheckCIF. But the IUCR is very slow at fixing bug in that kerberos of a program and we do therefore put our Olex2 users in trouble when they use the smtbx to refine their structure, as opposed to ShelXL, which does output (2). Besides, (2) is clearly more pleasing to the human mind than (1). Thus I reckon there is definitively a case to make a change in the cctbx here. The question is at which level? I don’t think we want to touch the sgtbx. I am of the opinion of patching iotbx.cif. Would that cause any trouble in Phenix and consort? Any remark or suggestion? Best wishes, Luc

3 2

New top-level module: fast_linalg
by Luc Bourhis 07 Mar '16

07 Mar '16

Hi, this is an advanced warning for the introduction of a new top-level module named fast_linalg. In a couple of days, I will push it in one go as 25 commits but don’t be scared, nothing is changed outside the top directory fast_linalg, which also means it is not configured by default. Thus there cannot be any disruption but I thought it would be polite to warn all cctbx developers. The module features the automatic installation and integration of OpenBLAS which I have used to speed up smtbx refinement. I will commit that work later. Best wishes, Luc J. Bourhis

2 2

Dead link for old builds on cctbx_build page
by Luc Bourhis 06 Mar '16

06 Mar '16

Hi, on http://cci.lbl.gov/cctbx_build/, in the phrase “ Old code bundles prior to 2015 may be accessed here.”, the word “here" is a link toward http://cci.lbl.gov/cctbx_legacy_build but this is a dead link. Is this a typo or are those builds truly gone? Best wishes, Luc

2 1

libtbx.easy_mp semantics
by markus.gerstel＠diamond.ac.uk 23 Feb '16

23 Feb '16

Hello everyone, Could someone please explain the libtbx.easy_mp semantics to me? Currently, if you run easy_mp with multiple processes, objects are 'reset' back to their original state, ie. the state of the object in the main thread is retained, changes in the processing threads are dropped. However if you run easy_mp with the number of processes set to 1, object states are modified, because the main thread is the executing thread. How it this supposed to work? For a more detailed test please run $ libtbx.python libtbx/tst_easy_mp_state.py The source code explains the test in more detail. This behaviour currently causes dials.integrate to fail on bzipped input files. Given that this code is used in many places and this particular feature is not obvious, it may explain other anecdotal emergent behaviour, too. -Markus -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 3