HHPC - BenLangmead/jhu-compute GitHub Wiki

The Homewood High Performance Cluster (HHPC) has the main computing and storage resources available to us.

There are two ways to use the HHPC's computing resources.

  • batch mode: you submit jobs, perhaps many of them, to the worker nodes
  • interactive mode: you log directly in to a dedicated node that is shared lab-wide

It's best to do as much computation in batch mode as possible, and only log in to a dedicated node for development or testing purposes.

See also: HHPC wiki.

Worker nodes

Langmead Lab owns a "share" of the cluster's worker nodes. This gives us priority access to 8 workers (8 times 12 = 96 cores). We can use a much larger fraction of the cluster if we need to, but with reduced priority. Programs and scripts ("jobs") can be run on the worker nodes by submitting them via the batch scheduler. HHPC uses TORQUE/Maui for this purpose. This works like other batch schedulers you may be familiar with, like Sun Grid Engine (SGE). See the HHPC Cluster Information page for details on software installed on the workers and how to submit jobs.

Batch jobs cannot live for longer than 4 or 4.5 days or so.

Jobs spanning many cores might wait in the queue for longer. Jobs that are currently using all of our "share" (and possibly beyond) will block other jobs from being scheduled on the share, which increases the chance the later-submitted jobs will wait, perhaps for a long time.

Dedicated nodes

Besides the worker nodes, the lab owns two dedicated nodes that aren't part of the worker pool. If you need to do "interactive" work (experimental, on the command line) -- as opposed to "batch" work (using MPI or running the same long-running program or script repeatedly on one or more datasets) -- then you can log into one of these nodes:

  • [langmead-login](HHPC: langmead-login) This is the analysis node.
  • [langmead-fs1](HHPC: langmead-fs1). This is the file server. Please only use this in an emergency and only with Ben's permission. Please don't run any resource-intensive tasks on this. This node is serving filesystems that are shared lab-wide, so using up its resources can harm I/O performance across the entire lab.
    • Dell PowerEdge R720xd (225-2110)
    • 4 x Intel Xeon E5-2665 2.4 GHz, each with 8 cores
    • 128 GB RAM

Anyone from the lab can log into these nodes at any time. If you want to do something very resource-intensive, very long-running, or both, you'll have to coordinate with other lab members to ensure you're not inconveniencing others.

When working interactively on cluster nodes, it's always a good idea to use screen. If you're not already familiar with screen, read up on it now. man screen to learn more.

Use top to monitor CPU usage on these nodes. man top to learn more.

Storage

See HHPC Storage.

Software

See HHPC shared software.

Getting help

Here's Jason Williams ([email protected]) describing the best way to ask for help with HHPC:

it would be best to send all HHPC support requests and account creations through the help desk system used over here in Physics and Astronomy. If you are emailing me regarding any HHPC support requests or account creations or anything of that nature, please make sure you CC [email protected] and clearly indicate in your subject line that it is an HHPC related request. This will automatically create a tracking ticket for your request, allow the wonderful help desk folks over here to correctly route the request, and allow me to better organize all the incoming requests and prioritize them accordingly.
This simple little modification will help both you and I more efficiently serve you and your needs on the cluster(s).

Be kind

A few principles for how you can be a good citizen on HHPC. Please adhere to these if at all possible.

  1. If you are submitting a job to the queue that spawns child processes, ensure you do not use more than your fair share of CPUs by binding each "root" process to a core with MPI. In brief, invoke mpirun -machinefile $PBS_NODEFILE -n [argument of nodes= from PBS file] --bind-to-core --by-socket.
  2. Clean up after yourself, even in the event your scripts don't run to completion. E.g. when writing to scratch disk in a Python script you can schedule a temporary directory for deletion after normal program termination with atexit.register as well as in case of an aborted job arising from signals like SIGTERM using signal.signal. See this recipe for comprehensively handling program termination in Python.
  3. Do not run any commands besides those for job scheduling/monitoring on the head node.

Here's a gallery of things you might accidentally do to inconvenience other HHPC users. Please be a good citizen. Try not to use an excessive amount of resources, and don't run any work-intensive programs on the head node.

From Jason Williams, December 13, 2012:

I have had reports from the sys admins that run the email servers of large amounts of emails coming from the clusters. After looking into it, it seems that people are submitting large numbers of single processor jobs (in the thousands) that all generate a single email when they either start, abort and/or finish.

According to the sys. admins, this is bringing the email server down. If you are going to submit large amounts of jobs to either cluster, please make sure that you set your mail options in your PBS job script to disable mails. This is done by setting #PBS -m n Or passing -m n to the qsub command.

From Jason Williams, June 4 2013:

Recently I have gotten reports from quite a few users that over the past 3 days, they have had issues with access to their home directories. After quite a bit of digging yesterday, last night and this morning, I found out that people have been running rather large computing processes on the main login node (login.hhpc.jhu.edu) It is unclear if these are processes that should be run on the cluster or not, but I would like to remind everyone that login.hhpc.jhu.edu is for logging in, compiling code and submitting jobs. It is not intended to be used to run any compute processing or post-processing programs. With the amount of users that use the system, this will cause massive disruptions in access to the cluster for everyone.

I issued a reboot this morning to clear out all of the processes that have been causing the problem. Please note that anyone running compute processing or post-processing programs on the main login node may be subject to having their account disabled without notice. If you are unsure of a place to run your post-processing jobs, the best place to run them is on the compute nodes through scheduler or on an analysis node owned by your group. If you are unsure if you group owns an analysis node, please contact me by emailing [email protected] and I will let you know.

From Jason Williams, September 9, 2013:

At about 3:30pm today, a rouge script running on the login node submitted 60,000+ jobs to the cluster within a matter of moments. This bogged down the scheduler while it tried to process those requests. The user account in question has been disabled until the user contacts me to clarify the issue. I am in the process of deleting the jobs which may take quite some time. While the scheduler processes those delete requests, you may notice slow responses from it and possibly periods of complete loss of connectivity to the scheduler through the qstat, showq and other such commands.