History for BioBrewClusterSetup

changed:
-
I'm testing biobrew, a Rocks based cluster distro with bioinformatics tools added (EMBOSS, BLAST, phylip, ...).

The test rig is puya.hpcf.upr.edu (136.145.54.188), the bottom 5 nodes in the left rack of espresso (all on the same powerbar, and the same Dell switch).

I installed biobrew v 0.9 from http://bioinformatics.org/ftp/biobrew/ following the readme. No surprises other than the fact that PXE and DHCP didn't work on the switch I set up on. I used the cheap FE switch on the GB ports to boot the compute node installations.

The biobrew docs don't say how to add users, but you can just run 'useradd' on the master. Rocks has good documentation online at http://www.rocksclusters.org/Rocks/ just remember OpenPBS and maui are out, and SGE is in.

Unlike stock Rocks or Oscar, biobrew uses SunGridEngine instead of OpenPBS. Sun has info at: http://gridengine.sunsource.net/ but the short info is man submit, and look at the docs for qsub, qconf, qstat.

You can get info on parallel environments and available hosts using::

  qconf -spl
  qconf -sel
  qstat -f

Example qsub script to run MPI jobs in parallel::

 # Test script for sun grid engine
 
 #$ -M humberto@hpcf.upr.edu
 #$ -m abe
 #$ -pe mpi 2
 
 echo Script ran on `hostname`
 echo Using nodes:
 cat $TMPDIR/machines
 echo NSLOTS = $NSLOTS
 
 /opt/mpich/ethernet/gcc/bin/mpirun -np $NSLOTS  \
    -machinefile $TMPDIR/machines \
    /opt/hpl-eth/bin/gcc/xhpl

Wierd note #1. Did you know you can reinstall the entire Rocks cluster from the frontend command line? The errata for OpenSSH explains how, and it worked (modulo DHCP issues, I swapped out the networking).

http://www.rocksclusters.org/errata/Security/ssh-alert.html

The step labled "7. Reinstall all compute nodes"::

      # cluster-fork /boot/kickstart/cluster-kickstart

triggers the reinstall, cool huh? 'cluster-fork' isn't quite as nice as 'cexec', but it's written in python, so it has hope.


From humberto Wed Dec 10 22:46:00 GMT-4 2003
From: humberto
Date: 2003/12/10 22:46 AST
Subject: MPI jobs on same node?
Message-ID: <20031210224652AST@plone.hpcf.upr.edu>

Asking SGE for 2 cpus results in it scheduling 2 nodes with 1 cpu each, how to ask for 1 2 CPU node?

From humberto Mon Dec 15 14:22:00 GMT-4 2003
From: humberto
Date: 2003/12/15 14:22 AST
Subject: Environment
Message-ID: <20031215142204AST@plone.hpcf.upr.edu>

You can't just set environmental parameters in qsub scripts like in OpenPBS. In order to set variables you need to specify them on the qsub command line::

  qsub -v P4_GLOBMEMSIZE=10485760 double.sge

From humberto Mon Dec 15 14:24:00 GMT-4 2003
From: humberto
Date: 2003/12/15 14:24 AST
Subject: MPI scheduling.
Message-ID: <20031215142438AST@plone.hpcf.upr.edu>

Glen Otero told me the way to request sge to allocate 2 processes per node, change the allocation_policy from '$round_robin' to '$fill_up'.

See 'qconf -sp mpi' to view the current policy, and 'qconf -mp mpi' to change it.