Coordinated Laboratory for Computational Genomics Home

1.  To get an account on AA please contact tscheetz@eng.uiowa.edu.  In the body of the message please give us your Academic Institution, Department and a brief description of the types of applications you would like to run.

2.  After you've received your account, you can ssh to aa.eng.uiowa.edu to login to the head node.  The cluster itself is made up of sixteen nodes.  Each node (aa101 - aa116) consists of local disk space and NFS mounted raid partitions.  Generally, these partitions will house the blast db's and binaries.  Also, each node has 2GBytes of RAM and dual 2.2Ghz Athlon MP processors.  They run RedHat Linux 7.3 (i686).  As I mention below, our interconnect is standard gigabit ethernet with jumbo frame support.    

3.  Currently, AA has been soley configured for batch processing.  Our scheduler for batch processing is PBSPro or Portable Batch System.  Below are some links to the user and administrator guide.  I also included some basic PBS commands to get you started.  Message Passing Interfaces (MPI) programs can be run, via the gigabit ethernet, however, substantial overhead will be encountered since our MTU size ~ 8500 bytes (jumbo frames).  



PBS Basics

Creating shell scripts to be submitted to PBS

#!/bin/csh
#PBS -N HMMRJOB
#PBS -o aa01:/pbs/junk
#PBS -e aa02:/pbs/junk
#PBS -l nodes="1:ppn=2"
program executions

This is a typical script that could be sent to the PBS server.  Note that none of these commands other than the program executions are essential, but are recommended for daily use.

#!/bin/csh  --This specifies which shell script to use, others include /bin/bash /bin/tcsh

#PBS -N HMMRJOB  --This specifies the PBS name given to the job, we will see this again when we look at qstat.

#PBS -o aa01:/pbs/junk  --This will sent the output to /pbs/junk on aa01.  We do this because with the latest versions of Linux, we can no longer rcp to /dev/null, so we send it to /pbs/junk where we delete the output files weekly.

#PBS -e aa01:/pbs/junk  --This is the same as the above but for error output.

#PBS -lnodes=1:ppn=2  --This tells the scheduler that you want 1 node and you are going to use both processors (ppn=2).  If you were to say ppn=1, then another job that also uses one processor could be allocated to the same node.  If you are going to be running a single process program but require all 2GBytes of memory, say ppn=2.

Note, that all of these options specified in the shell script can also be specified on the command:
bash$ qsub -N HMMRJOB -o aa01:/pbs/junk -e aa01:/pbs/junk -lnodes=1:ppn=2 program executions


Submitting shell scripts to the PBS server:
qsub:  This command submits jobs to the scheduler.  This command is generally followed by the name of a shell script you would like to run.  Other options include -lnodes=X:properties where X is the number of nodes you would like to request (useful for allocating nodes for MPIrun), and properties are the specific properties of nodes in a cluster.  Currently, AA is a homogenous cluster so properties probably won't be of much use.  The -q queue@server option specifies which queue and server you are sending your shell script to, the workq is the default queue.  The web queue should only be used by our online web applications for priority reasons.  Finally, the -I option can be very useful in debugging your shell scripts.  This option, gives you an interactive rlogin to one of the nodes.  qsub will return a job id for each shell script you submit.

Trivial qsub example:
bash$ qsub shellScript.csh

This example assumes your options are in the shell script itself, as shown above.  It's up to you if you want to put these options on the command line, the shell script or not at all.

Checking on the status of your jobs:
qstat:  This command gives you the status of the jobs running on the cluster and there associated PBS name, queue and user.  A useful option is -n which also shows you the node that a job is running on and the number of cpu's it's allocated from a particular node or nodes.


For more information on PBS please check out the user and administrator guide.
PBS User Guide
PBS Admin Guide