BioBrewClusterSetup

I'm testing biobrew, a Rocks based cluster distro with bioinformatics tools added (EMBOSS, BLAST, phylip, ...).

The test rig is puya.hpcf.upr.edu (136.145.54.188), the bottom 5 nodes in the left rack of espresso (all on the same powerbar, and the same Dell switch).

I installed biobrew v 0.9 from http://bioinformatics.org/ftp/biobrew/ following the readme. No surprises other than the fact that PXE and DHCP didn't work on the switch I set up on. I used the cheap FE switch on the GB ports to boot the compute node installations.

The biobrew docs don't say how to add users, but you can just run useradd on the master. Rocks has good documentation online at http://www.rocksclusters.org/Rocks/ just remember OpenPBS? and maui are out, and SGE is in.

Unlike stock Rocks or Oscar, biobrew uses SunGridEngine? instead of OpenPBS?. Sun has info at: http://gridengine.sunsource.net/ but the short info is man submit, and look at the docs for qsub, qconf, qstat.

You can get info on parallel environments and available hosts using:

  qconf -spl
  qconf -sel
  qstat -f

Example qsub script to run MPI jobs in parallel:

 # Test script for sun grid engine

 #$ -M humberto@hpcf.upr.edu
 #$ -m abe
 #$ -pe mpi 2

 echo Script ran on `hostname`
 echo Using nodes:
 cat $TMPDIR/machines
 echo NSLOTS = $NSLOTS

 /opt/mpich/ethernet/gcc/bin/mpirun -np $NSLOTS  \
    -machinefile $TMPDIR/machines \
    /opt/hpl-eth/bin/gcc/xhpl

Wierd note #1. Did you know you can reinstall the entire Rocks cluster from the frontend command line? The errata for OpenSSH? explains how, and it worked (modulo DHCP issues, I swapped out the networking).

http://www.rocksclusters.org/errata/Security/ssh-alert.html

The step labled "7. Reinstall all compute nodes":

      # cluster-fork /boot/kickstart/cluster-kickstart

triggers the reinstall, cool huh? cluster-fork isn't quite as nice as cexec, but it's written in python, so it has hope.

comments:

MPI jobs on same node? --humberto, 2003/12/10 22:46 AST reply
Asking SGE for 2 cpus results in it scheduling 2 nodes with 1 cpu each, how to ask for 1 2 CPU node?

Environment --humberto, 2003/12/15 14:22 AST reply
You can't just set environmental parameters in qsub scripts like in OpenPBS?. In order to set variables you need to specify them on the qsub command line:

  qsub -v P4_GLOBMEMSIZE=10485760 double.sge

MPI scheduling. --humberto, 2003/12/15 14:24 AST reply
Glen Otero told me the way to request sge to allocate 2 processes per node, change the allocation_policy from $round_robin to $fill_up.

See qconf -sp mpi to view the current policy, and qconf -mp mpi to change it.