NewBioBrewClusterSetup

I'm now testing BioBrew? 3 as a roll on top of Rocks 3.1.

Test rig is puya, bottom 5 nodes on left rack.

Production cluster is espresso, master + 12 nodes at top of left rack.

PXE is disabled (or better, put PXE after CD + HD).

Installed Rocks 3.1 base + java, hpc, sge, biobrew rolls.

Compute node installs complain that Fasta is not available, just select continue. (Carlos fixed this using Glen Otero's instructions http://bioinformatics.org/pipermail/biobrew-discuss/2004-September/000164.html)

Touch /etc/USESGE to enable sun grid engine.

After install, fix the /etc/profile.d/z-biomodules.csh script:

 # Additional modules for biobrew
 #

 if ( ${?MODULEPATH} ) then
    setenv MODULEPATH "${MODULEPATH}:/opt/BioBrew/modulefiles"
 else
    setenv MODULEPATH /opt/BioBrew/modulefiles
 endif

As user:

 cluster-fork cp z-biomodules.csh /tmp

As root:

 cluster-fork cp /tmp/biomodules.csh /etc/profile.d

(root can't read users home directories)

I sent Glen the file, and he shipped back a new modulefiles rpm that fixes the problem. It is now installed on espresso, same as the fasta rpm and the new 411 rpm.

qlogin is also broken follow tips at:

https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/2004-January/004413.html

But I have the same problem with closed connections.

Neat trick is to set qlogin_command to the following script '/usr/local/bin/qssh':

  #!/bin/sh
  exec ssh $1

That ignores the port number that sge passes to the qlogin_command, so it just does a straight ssh to the compute node.

Mail on the nodes is disabled by default.

I chkconfiged and started mail on the master and the nodes (sge needs mail to send notices to users). Mail is still busted, the nodes need to set up espresso as a smart relay. espresso needs to accept connections from the 10.0.0.0/8 network and relay them.

Hector Mendez asked for larger /tmp space on the nodes. ROCKS leaves 33 GB free on /state/partition1 on all the nodes. I made a /state/partition1/tmp directory mode 1777, and set the Sun Grid Engine tmpdir to point at it:

  # qconf -rattr queue tmpdir /state/partition1/tmp compute-0-0.q

That needs to be done for each compute node.

We've made a customization configuration file to apply the same configuration to all newly installed nodes. Carlos and Cheo did the initial implementation, I made some changes to support the $TMPDIR. Attached is the current version of the file. You can place a file in export/home/install/profiles/3.1.0/site-nodes/extend-compute.xml and it will be preserved across frontend upgrades and applied to the nodes.

extend-compute.xml

One liner to change the queue tmpdir on all nodes:

  for q in `qconf -sql`; do qconf -rattr queue tmpdir /state/partition1/tmp $q ; done

comments:

compute-0-11 broken CD --humberto, Fri, 08 Oct 2004 20:24:52 -0400 reply
The CDROM on compute-0-11 is broken, the tray doesn't open.

qmake --humberto, Fri, 08 Oct 2004 21:02:10 -0400 reply
Check out the qmake command and the make parallel environment.