Python-based
Hierarchical
ENvironment
for
Integrated
Xtallography
Contents
This document details how to create zip file bundles of python and the HDF5 library used for the Windows build of Phenix as described here.
These two base components do not have to be compiled from scratch but can be bundled as zip files to be downloaded when building for Windows. This is possible because unlike Unix operating systems binary compatibility of executables with Windows has been retained over successive versions of Windows apart from the distinction between the 32 and 64 bit platforms. These components are only necessary to create once. Bootstrap.py will then download them from http://cci.lbl.gov/cctbx_dependencies during the first stages of the build. This happens around lines 136 - 156 in cctbx_project\libtbx\auto_build\install_base_packages.py, svn revision 24805. The python bundle is uncompressed into base\bin\python and the HDF5 bundle is expanded into base\HDF5-1.8.16 within the directory from where the bootstrap build is being done. Unlike the directory location on Unix the respective library and header files of these two bundles are stored as sub directories within these directories and not in common base\include and base\lib directories.
The first base component bundled is python 2.7.9 with installed modules such as matplotlib, numpy, biopython, wxpython, etc. Version 2.7.9 is the minimum version that provides tools like "pip install" which makes it much easier to enhance python with 3rd party modules from https://pypi.python.org.
The first step is to download and install python2.7.9 (or higher) to your PC. Unfortunately the Python for Windows installer is much too clever for its own good. It may overwrite any existing python installation on your PC as well as the registry key that specifies the existing installation: the key;
HKEY_CURRENT_USER\Software\Python\PythonCore\2.7\InstallPath
To deal with this the easiest thing may be to either uninstall the existing python or to make a backup of the registry key
HKEY_CURRENT_USER\Software\Python
and then delete it. After that, install python in a folder that has no blank spaces in its name.
Warning
Entering wrong values in the registry or accidentally deleting values may render Windows bricked. Always ensure you have a recent "System Restore" point in your Windows backup. This will allow you to boot Windows to "Last known good configuration" if the registry gets corrupted.
Next we need to install wxPython package for the GUI. This is available as a binary installer from http://www.wxpython.org. Get version 3.0.2 and install it into your python installation. It should default to live in <your python folder>\Lib\site-packages.
The list of additional 3rd party python modules and their versions necessary for the python for Windows distributed with Phenix is:
Cython==0.22 PIL==1.1.7 Pillow==2.8.1 PyOpenGL==3.1.1a1 PyRTF==0.45 Pygments==2.1.3 biopython==1.65 comtypes==1.1.1 docutils==0.12 h5py==2.5.0 libsvm==3.20 matplotlib==1.4.3 numpy==1.9.2 psutil==2.2.1 py2exe==0.6.9 pyparsing==2.0.3 python-dateutil==2.4.2 pytz==2015.2 reportlab==3.1.44 scipy==0.15.1 six==1.9.0 jinja2==2.8
These can be added to your python installation as follows: In a command prompt change directory to the toplevel directory of your python installation and type:
python -m pip install moduleXYZ
where "moduleXYZ" is one of the above. Modules not found through the online database of python modules or those which have not been compiled with Visual C++ 2008 may be found on Christoph Gohlke's site. Make sure that the python version and compiler version of the modules matches 2.7 and Visual Studio 2008 respectively.
Now your customised python is ready to be bundled into a zip file. Open Windows Explorer, locate the folder above your python, right-click on your python folder and select "Send to Compressed (zipped) Folder". This will create a zip file named python.zip which contains everything in your customised python. Depending on the platform you are targetting rename this file to python2.7.9_x86_64_plus_relocatable.zip or python2.7.9_x86_32_plus_relocatable.zip. Once it has been uploaded to http://cci.lbl.gov/cctbx_dependencies bootstrap.py will download it during the build and uncompress it in the base subdirectory of your build location. This currently happens onwards from line 131 in the file cctbx_project\libtbx\auto_build\install_base_packages.py.
The second base component bundled is the HDF5 library which is necesary for Dials and Xia2. On Windows the build tool CMake from https://cmake.org/ must be installed first.
On this occassion it seems to be necessary to have the full Visual Studio 2008 installed. The free version of the compiler in the Windows 7 SDK seems to confuse CMake if other and newer Visual Studio compilers are present. This leads to error messages when attempting to build the library.
HDF5 can be downloaded from http://www.hdfgroup.org/ftp/HDF5/current/src/ such as a zipped file like CMake-hdf5-1.8.16.zip. Uncompress it on your PC.
In the folder CMake-hdf5-1.8.16 there should be cmd scripts such as build-VS2012-32.bat, build-VS2013-32.bat etc. but none targetting Visual Studio 2008, i.e. with "VS2008" as part of the filename. We need to adapt HDF5 to build with Visual Studio 2008. To do this open one of the existing cmd scripts such as build-VS2012-32.bat and replace the line
ctest -S HDF518config.cmake,BUILD_GENERATOR=VS2012 -C Release -V -O hdf518.log
with
ctest -S HDF518config.cmake,BUILD_GENERATOR=VS2008 -C Release -V -O hdf518.log
Save this file with the name "build-VS2008-32.bat". Now open the file "HDF518config.cmake". Locate the section:
# build generator must be defined
if(NOT DEFINED BUILD_GENERATOR)
message(FATAL_ERROR "BUILD_GENERATOR must be defined - Unix, VS200864, VS2013, VS201364, VS2012, or VS201264")
else()
if(${BUILD_GENERATOR} STREQUAL "Unix")
set(CTEST_CMAKE_GENERATOR "Unix Makefiles")
elseif(${BUILD_GENERATOR} STREQUAL "VS2015")
set(CTEST_CMAKE_GENERATOR "Visual Studio 14 2015")
elseif(${BUILD_GENERATOR} STREQUAL "VS201564")
set(CTEST_CMAKE_GENERATOR "Visual Studio 14 2015 Win64")
elseif(${BUILD_GENERATOR} STREQUAL "VS2013")
set(CTEST_CMAKE_GENERATOR "Visual Studio 12 2013")
elseif(${BUILD_GENERATOR} STREQUAL "VS201364")
set(CTEST_CMAKE_GENERATOR "Visual Studio 12 2013 Win64")
elseif(${BUILD_GENERATOR} STREQUAL "VS2012")
set(CTEST_CMAKE_GENERATOR "Visual Studio 11 2012")
elseif(${BUILD_GENERATOR} STREQUAL "VS201264")
set(CTEST_CMAKE_GENERATOR "Visual Studio 11 2012 Win64")
else()
message(FATAL_ERROR "Invalid BUILD_GENERATOR must be - Unix, VS2013, VS201364, VS2012, or VS201264")
endif()
endif()
and add entries for VS2008 like:
# build generator must be defined
if(NOT DEFINED BUILD_GENERATOR)
message(FATAL_ERROR "BUILD_GENERATOR must be defined - Unix, VS2008, VS200864, VS2013, VS201364, VS2012, or VS201264")
else()
if(${BUILD_GENERATOR} STREQUAL "Unix")
set(CTEST_CMAKE_GENERATOR "Unix Makefiles")
elseif(${BUILD_GENERATOR} STREQUAL "VS2015")
set(CTEST_CMAKE_GENERATOR "Visual Studio 14 2015")
elseif(${BUILD_GENERATOR} STREQUAL "VS201564")
set(CTEST_CMAKE_GENERATOR "Visual Studio 14 2015 Win64")
elseif(${BUILD_GENERATOR} STREQUAL "VS2013")
set(CTEST_CMAKE_GENERATOR "Visual Studio 12 2013")
elseif(${BUILD_GENERATOR} STREQUAL "VS201364")
set(CTEST_CMAKE_GENERATOR "Visual Studio 12 2013 Win64")
elseif(${BUILD_GENERATOR} STREQUAL "VS2012")
set(CTEST_CMAKE_GENERATOR "Visual Studio 11 2012")
elseif(${BUILD_GENERATOR} STREQUAL "VS201264")
set(CTEST_CMAKE_GENERATOR "Visual Studio 11 2012 Win64")
elseif(${BUILD_GENERATOR} STREQUAL "VS2008")
set(CTEST_CMAKE_GENERATOR "Visual Studio 9 2008")
elseif(${BUILD_GENERATOR} STREQUAL "VS200864")
set(CTEST_CMAKE_GENERATOR "Visual Studio 9 2008 Win64")
else()
message(FATAL_ERROR "Invalid BUILD_GENERATOR must be - Unix, VS2013, VS201364, VS2012, or VS201264")
endif()
endif()
Now locate the section:
######### Following describes compiler ############
if(WIN32)
set(SITE_OS_NAME "Windows")
set(SITE_OS_VERSION "WIN7")
if(${BUILD_GENERATOR} STREQUAL "VS201564")
set(SITE_OS_BITS "64")
set(SITE_COMPILER_NAME "vs2015")
set(SITE_COMPILER_VERSION "14")
elseif(${BUILD_GENERATOR} STREQUAL "VS2015")
set(SITE_OS_BITS "32")
set(SITE_COMPILER_NAME "vs2015")
set(SITE_COMPILER_VERSION "14")
elseif(${BUILD_GENERATOR} STREQUAL "VS201364")
set(SITE_OS_BITS "64")
set(SITE_COMPILER_NAME "vs2013")
set(SITE_COMPILER_VERSION "12")
elseif(${BUILD_GENERATOR} STREQUAL "VS2013")
set(SITE_OS_BITS "32")
set(SITE_COMPILER_NAME "vs2013")
set(SITE_COMPILER_VERSION "12")
elseif(${BUILD_GENERATOR} STREQUAL "VS201264")
set(SITE_OS_BITS "64")
set(SITE_COMPILER_NAME "vs2012")
set(SITE_COMPILER_VERSION "11")
elseif(${BUILD_GENERATOR} STREQUAL "VS2012")
set(SITE_OS_BITS "32")
set(SITE_COMPILER_NAME "vs2012")
set(SITE_COMPILER_VERSION "11")
endif()
## Set the following to unique id your computer ##
and add entries for the VS2008 compiler like:
######### Following describes compiler ############
if(WIN32)
set(SITE_OS_NAME "Windows")
set(SITE_OS_VERSION "WIN7")
if(${BUILD_GENERATOR} STREQUAL "VS201564")
set(SITE_OS_BITS "64")
set(SITE_COMPILER_NAME "vs2015")
set(SITE_COMPILER_VERSION "14")
elseif(${BUILD_GENERATOR} STREQUAL "VS2015")
set(SITE_OS_BITS "32")
set(SITE_COMPILER_NAME "vs2015")
set(SITE_COMPILER_VERSION "14")
elseif(${BUILD_GENERATOR} STREQUAL "VS201364")
set(SITE_OS_BITS "64")
set(SITE_COMPILER_NAME "vs2013")
set(SITE_COMPILER_VERSION "12")
elseif(${BUILD_GENERATOR} STREQUAL "VS2013")
set(SITE_OS_BITS "32")
set(SITE_COMPILER_NAME "vs2013")
set(SITE_COMPILER_VERSION "12")
elseif(${BUILD_GENERATOR} STREQUAL "VS201264")
set(SITE_OS_BITS "64")
set(SITE_COMPILER_NAME "vs2012")
set(SITE_COMPILER_VERSION "11")
elseif(${BUILD_GENERATOR} STREQUAL "VS2012")
set(SITE_OS_BITS "32")
set(SITE_COMPILER_NAME "vs2012")
set(SITE_COMPILER_VERSION "11")
elseif(${BUILD_GENERATOR} STREQUAL "VS200864")
set(SITE_OS_BITS "64")
set(SITE_COMPILER_NAME "vs2008")
set(SITE_COMPILER_VERSION "9")
elseif(${BUILD_GENERATOR} STREQUAL "VS2008")
set(SITE_OS_BITS "32")
set(SITE_COMPILER_NAME "vs2008")
set(SITE_COMPILER_VERSION "9")
endif()
## Set the following to unique id your computer ##
Finally due to licensing terms of the optionally compiled SZIP library as detailed here: https://www.hdfgroup.org/doc_resource/SZIP/ this needs to be excluded from builds. To do that find the lines:
### disable ext libs building
#set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=OFF")
#set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF")
#set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_SZIP_ENCODING:BOOL=OFF")
and uncomment them:
### disable ext libs building
set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=OFF")
set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF")
set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DHDF5_ENABLE_SZIP_ENCODING:BOOL=OFF")
The configuration is now good to build. Open a command prompt and change directory to the CMake-hdf5-1.8.16 folder with the amended build script files. Initialise the compilers environment variables by running
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvars32.bat
or
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvars64.bat
according to the platform you are targetting. CMake installed earlier may not be in the executable path in which case you can add it temporarily to your command prompt by entering:
set PATH=%PATH%;C:\Program Files\CMake\bin
Then run the build script by entering:
build-VS2008-32.bat
This should build the library, bundle it into a compressed zip file located in CMake-hdf5-1.8.16\build and named HDF5-1.8.16-win32.zip or HDF5-1.8.16-win64.zip depending on the platform. As with the python bundles these files should be uploaded to http://cci.lbl.gov/cctbx_dependencies so that bootstrap.py may fetch it during a build of Phenix.