[phenixbb] Autobuild on parallel

[Terwilliger, Thomas Charles]

Hi Kaushik,

I'm sorry for the problem with parallel autobuilding!

I think that parallel_autobuild is unfortunately not going to do what you want. It is for running many parallel jobs with the same starting model and combining all the results.  It doesn't work with multiple starting models as I think you want it to.

On the error you got, I'm not quite sure what happened but it seems that perhaps the command "qsub" was not found:

RuntimeError: child process stderr output:
  command: 'qsub background=False nproc=20 "~phenixWorkingDir/1yya/part1/8tim/AutoBuild_run_5_/TEMP0/RUN_FILE_1"'
  /bin/sh: qsub: command not found

It is a little surprising that this would happen in parallel_autobuild and not in autobuild.  However perhaps in autobuild you specified that the multiple jobs are to be run with "sh" (run_command=sh) and in parallel_autobuild it was with qsub (run_command=qsub).

Let me know if that does not help,

All the best,
Tom T



________________________________
From: phenixbb-bounces at phenix-online.org [phenixbb-bounces at phenix-online.org] on behalf of Kaushik Hatti [hskaushik at gmail.com]
Sent: Monday, April 27, 2015 12:05 AM
To: phenixbb at phenix-online.org
Subject: [phenixbb] Autobuild on parallel

Hi,

What is the best way to run Autobuild if I need to run it on hundreds of datasets (through command-line)?  I have a single machine with 24 cores and access to a cluster with 128 cores (on parallel; accessible only through PBS script).

Each Autobuild job with nproc=4 on a single core machine takes ~12 hours to run (nproc > 4 wouldn't make any difference in the speed of computation?!).  However, I read that I could set nproc=10 (or more) and run on parallel (https://www.phenix-online.org/documentation/reference/autobuild.html#parallel-jobs-nproc-nbatch-number-of-parallel-models-and-how-autobuild-works-in-parallel).  When I tried the same through PBS script, I got the following error,

XXXXXXXXXXXXXX
Running up to  1  jobs in parallel... with total of  3  jobs

Splitting work into 3 jobs and running with 1 processors using qsub background=False nproc=20
background=False in ~phenixWorkingDir/1yya/part1/8tim/AutoBuild_run_5_/TEMP0
Final job will be run with sh  with background=True
Starting job 1...Log will be: ~phenixWorkingDir/1yya/part1/8tim/AutoBuild_run_5_/TEMP0/RUN_FILE_1.log
Traceback (most recent call last):
  File "~opt/phenix-1.9-1692/phenix/phenix/autosol/AutoBaseExtend.py", line 989, in DoNextMethod
    self.CarryOutBest()     # to be obtained after it is finished
  File "~opt/phenix-1.9-1692/phenix/phenix/autosol/AutoBaseExtend.py", line 2147, in CarryOutBest
    getattr(self,str(self.application_method))()  # call this fn
  File "~opt/phenix-1.9-1692/phenix/phenix/wizards/AutoBuild.py", line 2364, in AutoBuild_build_cycle
    self.AutoBuild_rebuild_cycle_run()
  File "~opt/phenix-1.9-1692/phenix/phenix/wizards/AutoBuild.py", line 4647, in AutoBuild_rebuild_cycle_run
    always_reuse_model=always_reuse_model)
  File "~opt/phenix-1.9-1692/phenix/phenix/wizards/AutoBuild.py", line 7452, in AutoBuild_build_refine
    mtz_file=mtz_file,mtz_ref_file=mtz_ref_file)
  File "~opt/phenix-1.9-1692/phenix/phenix/wizards/AutoBuild.py", line 7696, in run_standard_build_in_parallel
    r.run(out=sys.stdout)
  File "~opt/phenix-1.9-1692/phenix/phenix/autosol/run_group_of_wizards.py", line 294, in run
    self.start_run(run_file,last=is_last)
  File "~opt/phenix-1.9-1692/phenix/phenix/autosol/run_group_of_wizards.py", line 604, in start_run
    cmd+" "+self.add_double_quote(run_file,escape_space=False)).raise_if_errors()
  File "~opt/phenix-1.9-1692/cctbx_project/libtbx/easy_run.py", line 37, in raise_if_errors
    raise Error(msg)
RuntimeError: child process stderr output:
  command: 'qsub background=False nproc=20 "~phenixWorkingDir/1yya/part1/8tim/AutoBuild_run_5_/TEMP0/RUN_FILE_1"'
  /bin/sh: qsub: command not found

XXXXXXXXXXXXXX

Please let me know if I can share eff and log files of the run.

Please advice,
Kaushik


--
People living deeply have no fear of death - Anais Nin
Caution: I am still the dumbest person I have ever known :-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://phenix-online.org/pipermail/phenixbb/attachments/20150427/65ec248a/attachment-0001.htm>