ISAAC intro - mestato/EPP622_2026 GitHub Wiki

List from EPP 622 2026:

• Basic intro to ISAAC, what it is, how its managed.
  • available resources
  • dtn1/dtn2/login/login1/login2
  • When it's okay to use commands on the login nodes vs sbatching
  • how to find help on the oit webpages
  • where and how to file a ticket
  • filesystems - home dir, scratch, project space. what you get, how to see how much you’ve used up, how to request more (edited) 
  • how to buy in
  • Tennessee AI Initiative (ai-tenn)
• SLURM
  • sphinx - do not need SLURM, use google calendar
  • how does scheduling work?
  • SLURM the header - all the things
  • how to find the right partition and qos
  • if its a your computer issue vs a server issue
  • how to see if your jobs are running
  • sshing into specific nodes to check on your jobs
    • side note for GPUs - monitor GPU resources with nvidia smi
  • other important slurm commands to know
• Interactive resources including ISAAC on demand
  • jupyter and rstudio

1. Request an account (You've already done this)
2. Go to the Submit a HPSC Request link and ask for a new project (this may only be performed by PIs). By default this will produce a 1TB directory in addition to the designated home and scratch directories. If more space is required consider submitting additional help tickets.

Let's check out the UTK HPSC page.

On the bottom left side navigation, there is a expandable link for ISAAC Next Gen:

This is where the useful information lives:

Click on File Systems to see the default user storage, split into the home, scratch, and proj categories:

Let's log on to isaac to see our storage limits!

• The acronym:
  • Infrastructure for Scientific Applications and Advanced Computing
• ISAAC is a high performance computing (HPC) cluster managed by the High Performance & Scientific Computing (HPSC) group of the Office of Innovative Technologies (OIT)
• Currently there are 259 nodes with 18,168 total cores

Example image of HPC from ornl.gov.

Log into Isaac Next Gen. After your password, it will send you a Duo push.

ssh <yourusername>@login.isaac.utk.edu

Notes

• You can actually provide login1 or login2 at this step to always go to a specific login node.
• You should never run jobs on the login node! It is only to set up your scripts to launch your jobs through the scheduler.
• Keep the documentation handy!
• The user portal will enable you to see what projects you are a part of and where you can store data and how much
• You should be a part of ISAAC-UTK0505 and see storage at /lustre/isaac/proj/UTK0505
• By default, SLURM scheduler assumes the working directory to be the directory from which the jobs are being submitted

This command should give you a breakdown, per project, of your disk use footprint (as well as the maximum limits based on your account).

checkquotas24

• Simple, low memory file checking/organization commands:

  • cd / mkdir / htop
  • mv / cp (as long as files aren't enormous, then use data transfer nodes)
  • head / tail / cat / zcat
  • nano / vim (as long as files aren't enormous)

• Avoid:

  • Running bioinformatics tools
  • About any tool you'd import / load

If in doubt, check htop (hit u and start typing your username).

Okay, so we know how to access the login node of ISAAC-NG, and we understand the limitations of storage space in home, scratch, and proj. Suppose we want to actually do some work.

Every job on the cluster requires (as a bare minimum) three pieces of information:

• Account (you'll likely have 2+ right now, you must specify which one)
• Partition
• QOS

Running jobs documentation

Opportunistic partitions as of November 2024:

This command will list all of your projects and which QOS each has access to:

sacctmgr -p show assoc

If you want to learn more about a given qos governing a partition, you can use sacctmgr to learn more:

sacctmgr show qos where name=campus

You can use a comma-separated list of qos to return multiple qos summaries and edit the format of the output:

sacctmgr show qos where name=campus format=Name,Priority,MaxTRES,MaxWall,MaxTRESPU,MaxJobsPU,MaxSubmitPU

PU means per user.

• MaxTRES : resources for a single job
• MaxWall : walltime for a single job
• MaxTRESPU : total resources across running jobs
• MaxJobsPU : number of running jobs
• MaxSubmitPU : number of pending + running jobs

You can also return a summary of the entire partition using scontrol.

scontrol show PartitionName=campus

You can also get an enormous list of partitions on the cluster:

showpartitions

But what if we want to learn a bit more about the actual nodes underlying these partitions?

sinfo \
--Format statecompact,gres,GresUsed,cpusstate,partition,nodes,nodelist \
--partition=campus-gpu,campus-gpu-bigmem,campus-gpu-large,long-gpu,ai-tenn

• A: Allocated (in use by jobs)
• I: Idle (available)
• O: Other (unavailable, e.g. node is down, drained, or otherwise not usable)
• T: Total

The OIT HPSC ISAAC-NG Running Jobs link has some very helpful additional commands, including some custom to ISAAC.

isaac-sinfo | grep gpu

This is overkill, but lots of info you can parse through about the nodes/partitions of ISAAC-NG.

cat /etc/slurm/slurm.conf

Wow, we've talked a lot about the resources, but what is the scheduler and queue, and how can we just run a job?

Here is the queue:

squeue -o "%.20i %.11P %.8j %.1u %.10M %.12l %.6D %.5C %.7m %.12q %.12R %.2t"

Note the R and PD status of the running and pending jobs, respectively.

Go to the project directory and make a practice directory:

cd /lustre/isaac24/proj/UTK0505/$USER
mkdir -p isaac_ng_practice
cd isaac_ng_practice

Lets just get a quick test command going

echo Worked! > output.txt

Lets try to run exact same command through the scheduler.

In first.sbatch, add:

#!/bin/bash
#SBATCH -J intro_to_isaac
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH -A ISAAC-UTK0505
#SBATCH -p short
#SBATCH -q short
#SBATCH -t 00:10:00

zcat /lustre/isaac24/proj/UTK0505/module3_data/AppSpring2.fq.gz | wc -l

The directives at the top beginning with #SBATCH tell the job scheduler about this job, including what kind of resources it needs. This can be tricky - you need to know how many threads and how much RAM your job will take, so that you request a sufficient amount. Here, we requested a single cpu for 10 minutes. We provide the class account ISAAC-UTK0505 and the short qos and partition. The -J field is a job name, which is quite useful.

Submit the script:

sbatch first.sbatch

To see where your job is in the queue

squeue -u <yourusername>

...or

squeue --me

Did it work? Do you see sbatch_output.txt? What is the slurm file that got created?

In general, all SBATCH scripts will be set up this way:

#!/bin/bash
#SBATCH -J intro_to_isaac
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH -A ISAAC-UTK0505
#SBATCH -p short
#SBATCH -q short
#SBATCH -t 00:10:00

< your commands go here >

Other common SBATCH header params include:

#SBATCH --error <path_to_file>  # where to write stderr
#SBATCH --output <path_to_file> # where to write stdout

A note on memory:

#SBATCH --mem 20G

From the HPSC page on running jobs:

Memory is tied to number of cores. If you require more memory for your job, then you will need to request more tasks or cores. Bigmem nodes pin a minimum of 32 GB of RAM per core, whereas non-bigmem nodes pin a minumum of 3.8 GB of RAM per core.

ISAAC-NG has many pre-installed software options available. Always check the documentation for a given tool to see what the newest version is, but it never hurts to see what's already available in the software system.

module avail

BWA is something we use a lot, let's see if its installed

module avail bwa
module load bwa
bwa

You can manually unload software:

module unload bwa

Given that software may be out of date, it's worth noting that ISAAC-NG has a dedicated apptainer install:

apptainer --version

Extending the sbatch scripts we've used above, we can load modules or call commands in our script, as long as they're available.

#!/bin/bash
#SBATCH -J load_module
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH -A ISAAC-UTK0505
#SBATCH -p short
#SBATCH -q short
#SBATCH -t 00:10:00

# We can load bwa within an sbatch script
module load bwa

# Then we can call the command "bwa"
bwa

The basic way to see jobs in the queue is:

squeue -u <yourusername>
squeue --me #same as above, less typing

Using the job id from above, kill a job in the queue using:

scancel <job_id>

To get a nice summary of all of your currently running jobs including job id (i), partition (P), job name (j), cpus (C), memory (m), and qos (q).

squeue -o "%i %P %j %C %m %q" --me

Get a lot of information about a job in the queue.

scontrol show job <job_id>

Modify the sbatch header to create explicit log files for stdout and stderr. You can also modify --mail-type to include "BEGIN".

#SBATCH --error=job.e%J
#SBATCH --output=job.o%J
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=<YOUR EMAIL>@<YOUR EMAIL>

For an immediate interactive node in your current directory, srun will accomplish the exact same as salloc (explained below) but will connect you immediately with a node without manually connecting and disconnecting.

srun --account ISAAC-UTK0505 --partition=short --qos=short --nodes=1 --cpus-per-task=2 --mem=8G --time=0-01:00:00 --pty /bin/bash

Another way to get an interactive session is via salloc where slurm will reserve a node for you:

salloc --account ISAAC-UTK0505 --partition=short --qos=short --nodes=1 --cpus-per-task=2 --mem=8G --time=0-01:00:00

This will produce a message similar to Nodes il1340 are ready for job, but the node will be different depending on what's available. To connect you can use:

ssh <node name from above>

Once you connect, you'll see the prompt shows your username@node. To exit simply type exit. You may have to type exit a second time or scancel the allocation to free up the resource.

If you have any job running in the queue, it is also possible to ssh into it from the login node:

ssh <name of node>

How to decide which settings to use for a job:

• Check documentation for recommendations on num CPUs, memory, etc.
• Check syntax, run on short qos
• Estimate start time sbatch --test
• Factors to consider before deciding on a partition/qos:
  • Read github issues/documentation for recommended max memory needed.
  • See if there's any documentation for how long the job will take.
  • Does the tool have the ability to resume a failed job?
• Parsing common slurmstepd error messages and log files

If we want to see a summary of the nodes and the CPUs on each that are Allocated, Idle, Other, Total (A/I/O/T).

sinfo --Format statecompact,cpusstate,partition,nodelist --partition=campus,short

Make a new sbatch file called test_short.sbatch with the following content:

#!/bin/bash
#SBATCH -J bwa
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH -A ACF-UTK0011
#SBATCH -p short
#SBATCH -q short
#SBATCH -t 01:00:00

sleep 600

[!NOTE] Don't run this with sbatch just yet!

Now make another file called test_campus.sbatch. It might be helpful to just use cp from the previous file.

#!/bin/bash
#SBATCH -J bwa
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH -A ACF-UTK0011
#SBATCH -p campus
#SBATCH -q campus
#SBATCH -t 01:00:00

sleep 600

[!NOTE] Don't run this with sbatch just yet!

Now, using the --test-only flag, let's see which partition will run first with the exact same resource requests:

sbatch --test-only test_short.sbatch
sbatch --test-only test_campus.sbatch

CHALLENGE

Pick out a few more partitions/qos combinations (remember you can see your partitions using sacctmgr command above). Using the sbatch --test-only approach, which partition is going to take the absolute longest to begin?

Go to this link and select the "Submit a HPSC Request" button:

Fill out your department name and title for the issue, then select ISAAC-NG and choose the most sensible category for your issue:

General guidance on what to type:

• What were you trying to do?
  • Run a tool? Which version? How was it installed/used?
  • Use an HPC resource? Which one?
• What was the exact command(s) that you tried?
• What was the exact error or behavior that happened?

OIT will be able to solve your issue 10x faster if you provide these baseline troubleshooting starting points.

Return to your analysis directory from last lab.

cd /lustre/isaac24/proj/UTK0505/$USER/isaac_ng_practice

Task arrays are great but the scheduler assumes all data can be referred to by a convenient range of numbers, in this case 1 to 2. Let's see how this works. In array.sbatch, put

#!/bin/bash
#SBATCH -J array_test
#SBATCH --nodes=1
#SBATCH --cpus-per-task=4
#SBATCH -A ISAAC-UTK0505
#SBATCH -p short
#SBATCH -q short
#SBATCH -t 00:10:00
#SBATCH --array=1-2

echo "${SLURM_ARRAY_TASK_ID}" > ${SLURM_ARRAY_TASK_ID}.txt

What is SLURM_ARRAY_TASK_ID in this script and what is it doing?

Note we are using --cpus-per-task=4, we'll investigate later if these cpus are split amongst the tasks or if each task receives 4 cpus.

Submit the script

sbatch array.sbatch

New files are created, showing the array of 2 jobs. And new slurm output file names. Check them out. What do you see?

But our actual files aren't named 1-2. We can make aliases OR we can use some more clever sed.

Start by putting those names in a file:

ls /lustre/isaac24/proj/UTK0505/module3_data/*.fasta > filenames.txt

You can check it worked with nano or cat.

Now tweaking the submission script. First, we'll use sed to take the number provided from ${SLURM_ARRAY_TASK_ID} and pull out the corresponding line number in filenames.txt.

Try it out quick and see how it behaves (sed is printing the line number 2):

sed -n -e "2 p" filenames.txt

So, we'll use sed inside our sbatch script to create the output filename (using a regex and the input filename). In array.sbatch, put:

#!/bin/bash
#SBATCH -J bwa
#SBATCH --nodes=1
#SBATCH --cpus-per-task=4
#SBATCH -A ISAAC-UTK0505
#SBATCH -p short
#SBATCH -q short
#SBATCH -t 00:10:00
#SBATCH --array=1-2

infile=$(sed -n -e "${SLURM_ARRAY_TASK_ID} p" filenames.txt)

outfile=$( basename $infile | sed 's/.fasta/.stats/' )

grep -c "^>" $infile > $outfile

This will request 2 simultaneous jobs, each with --cpus-per-task=4.

Submit and see if it worked.

What happens when you use squeue --me?

A list of useful commands and topics to mention:

• transferring data with dtn[12]
• tour of resources
  • Open OnDemand
  • Open OnDemand RStudio
    • (don't forget .libPaths(c("/lustre/isaac24/scratch//R", .libPaths())) )
  • Open OnDemand Jupyter
  • Open OnDemand shell
  • Open OnDemand file browser
  • Globus