Our lab's research computing subscription supports a directory at /blue/juliemeyer/ on HiPerGator. Each lab member will have a home directory at /blue/juliemeyer/[user]. Your user name is the same as your gatorlink id.

When connecting via FTP, select SFTP (SSH Secure File Transfer Protocol). The server name will be sftp.rc.ufl.edu, use your own gatorlink id and password. The first time you use Cyberduck you will need to navigate to the lab directory. In the main menu select Go > Go to folder and then type in the full path to the lab directory: /blue/juliemeyer/. You can create a bookmark that will reopen every time you open the program.

First, you will need to request a UFRC (UF Research Computing) account. You will need a terminal program to talk to hpg and you will likely want an SFTP (secure file transfer protocol) program to look at the files on hpg. On a mac, I use iTerm for my terminal and Cyberduck for my sftp, but you may feel free to use other programs if you are more familiar with them. Options on windows machines will be different.

Once you have an account and the programs, you will login to hpg with your gatorlink credentials:

ssh [user]@hpg.rc.ufl.edu

You will be prompted to enter your password and will see a welcome message.

Your prompt will now look something like:

[juliemeyer@login3 ~]$

You are now connected to a "login" node. From the login node, you can change directories, look at lists of files, and submit jobs to the SLURM scheduler. You are not allowed to do computational work on the login nodes -- for that you will need to switch to a development node.

1. Raw sequencing reads from ICBR: the sequencing center sends the lab's data to our directory on hipergator - I usually select /blue/juliemeyer/share. This directory will accessible to everyone in the lab. Be aware that other people may have data on the same run as you so don't delete anything from /ufrc/juliemeyer/share without checking with them first. I will backup all raw sequencing reads on an external hard drive.

2. Moving large amounts of data to and from HiPerGator to either your desktop computer or an external hard drive: this is best done with globus. Amplicon data (Miseq runs) are small enough to use something like cyberduck. Metagenomic data (NextSeq runs) will probably need globus, which will continue the upload even if the internet connection is disrupted during the transfer. Sign up for a free globus account through UF, see UFRC wiki on globus for more information.

You can see files and directories from the terminal (aka "command line") using commands like ls or ls -l for "list". Alternatively, you can install a file transfer protocol like Cyberduck (my preference) or FileZilla. Using SSH file transfer protocol, the server name is "hpg.rc.ufl.edu", use your gatorlink username and password to login. You should now have a window showing the files and directories you have.

 sbatch script.txt

 squeue -A juliemeyer

Shows the IDs for all jobs run by the entire lab as denoted by the -A flag (where juliemeyer is the lab directory name)

 squeue -u [user]

Shows the job IDs for an individual user as denoted by the -u flag, replace [user] with your own username

 scancel [jobid]

Where the [jobid] is the job you want to cancel

When running a SLURM script on a Windows computer, you must run a Dos to Unix command to make sure that the computer can read your script. When you save your code on a Windows computer, it adds a few extra components that make it unreadable to HiPerGator. Before you submit your sbatch, first run:

dos2unix script.txt

Our research computing investment has limits on the amount of memory we can use. If one member is using the full memory ~800gb allotted for our investment on a running script, no new jobs for our lab group will start until the first job is finished. Thus, communication and collaboration among the lab members is important. The maximum wall time (on one script) for our group investment is 31 days.

To check the qos limits our lab group pays for, use showQos. As of writing (May 2024), the output is:

Name                 Descr                          GrpTRES                                        GrpCPUs 
-------------------- ------------------------------ --------------------------------------------- -------- 
juliemeyer           juliemeyer qos                 cpu=100,gres/gpu=0,mem=800000M                     100 

If you need to run scripts and the full investment allocation is being used OR if you have memory intensive jobs like metagenome assembly with >50 million reads and binning MAGs on merged assemblies, you can use "burst", which allows you up to 7,000 gb (!!) of memory, but a max wall time of 4 days. For more information, see UFRC page on quality of service. ***I have found that I can submit multiple jobs using 100gb memory on the burst QOS and they will run simultaneously as long as there isn't high demand at the time. To date, I have only ever had to wait a few minutes for burst scripts to start.

To check our subscription to burst, use showQos juliemeyer-b. Right now (May 2024) it is:

Name                 Descr                          GrpTRES                                        GrpCPUs 
-------------------- ------------------------------ --------------------------------------------- -------- 
juliemeyer-b         juliemeyer burst qos           cpu=900,gres/gpu=0,mem=7200000M                    900 

It is important to communicate with your peers about usage, and a quick way to check who is using what from our lab group is with the code:

slurmInfo -pu -g juliemeyer

With that information, you can help allocate resources for your given job.
An example of SLURM script header for using our regular investment with maximum memory and wall time is below. You can customize the job-name to fit the task that you are performing (here it is "map_". The "%j" will insert the unique jobid so each of your output files (the log and error files) with have the jobid. Replace the user email with your email. You can choose getting emails when scripts begin, end, and/or fail. Take out notifications that you aren't interested in.

#!/bin/bash
#SBATCH --job-name=map_%j
#SBATCH --output=map_%j.log
#SBATCH --error=map_%j.err
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH [email protected]
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=2
#SBATCH --mem-per-cpu=30gb

An example of SLURM script header for using the "burst" quality of service with maximum memory and wall time:

#!/bin/bash
#SBATCH --job-name=map_%j
#SBATCH --output=map_%j.log
#SBATCH --error=map_%j.err
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH [email protected]
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=2
#SBATCH --mem-per-cpu=50gb
#SBATCH --account=juliemeyer
#SBATCH --qos=juliemeyer-b

Sometimes, like for co-assemblies, they may require an enormous amount of memory, so to make them go more quickly, you can request the bigmem partition on hipergator. You do this by adding:

#SBATCH --partition=bigmem

An example of resources for a large metagenome co-assembly Cynthia did with MegaHit is below. Note, this used 112 days of CPU wall hours in only a day and a half! It can be a huge time saver to request large amounts of memory/cpus on the burst.

#!/bin/bash
#SBATCH --job-name=megahit_%j                # add the name of a job. %j will add the job ID
#SBATCH --output=megahit_%j.log              # add the log output
#SBATCH --error=megahit_%j.err               # add a file name for anything printed to standard error
#SBATCH --mail-type=BEGIN,END,FAIL           # when you get an email
#SBATCH --mail-user=[username]@ufl.edu       # add your personal email
#SBATCH --mem-per-cpu=10gb                   # How much RAM you want for each CPU
#SBATCH --cpus-per-task=100                  # How many CPUs, often referred to as "threads" you request
#SBATCH --partition=bigmem                   # For 1000 gb of ram, you have to request bigmem
#SBATCH --time=4-00:00:00                    # The burst has a limit of 4 days
#SBATCH --account=juliemeyer                 # Lab account
#SBATCH --qos=juliemeyer-b                   # Request the "burst" with juliemeyer-b

export OMP_NUM_THREADS=100                   # You need this if you request multiple cpus (threads)

Above, this script requests a total of 1000 gb (1 TB) of computer memory (10 gb per cpu, 100 cpus). That is way over what is typically requested, so the automatic computer partitions will not work. Hence, why you must request bigmem.

We have a storage limit on HiPerGator of 8 TB for the entire group. It is very important not to use the hpg as long term storage! When we get raw sequencing data from ICBR, it will be backed up on the lab's external hard-drive and on dropbox. If you need to keep important intermediate files, please back them up on dropbox and an external hard drive when you no longer need immediate access to them. The best way to move large files is with globus. You can set up an external hard drive as a globus endpoint and transfer files directly from hpg to the external hard drive, without downloading to your computer. See UFRC wiki on how to use globus (you get a free account with your gatorlink id). It is fine to keep files on hpg if you are still working on them, but use compression whenever possible. Many bioinformatics tools will use .fastq.gz files as both input and output, so don't gunzip if you don't need to do so.

The following script will give a summary of the storage being used by the group, our total storage allotment, and the percentage of storage being used.

To see our group's allocation of both computing cores and storage:

showAllocation

To see what storage we are using as a group:

module load dibig_tools
qu juliemeyer

For a quick look at your and the group's usage, use:

blue_quota

We can see how much time we spend running jobs on hipergator. You need to specify the time frame by changing the Start and End times in the code below.

sreport -t Hours cluster AccountUtilizationByUser Start=2023-03-01T00:00:00 End=2023-03-31T23:59:59 Accounts=juliemeyer

Here is an example of the output for March 2023 (using the line of code above):

--------------------------------------------------------------------------------
Cluster/Account/User Utilization 2023-03-01T00:00:00 - 2023-03-31T23:59:59 (2674800 secs)
Usage reported in CPU Hours
--------------------------------------------------------------------------------
  Cluster         Account     Login     Proper Name     Used   Energy
--------- --------------- --------- --------------- -------- --------
hipergat+      juliemeyer                              24120        0
hipergat+      juliemeyer   acauvin     Alli Cauvin      923        0
hipergat+      juliemeyer e.quinta+ Elena Quintani+       68        0
hipergat+      juliemeyer ewelinar+   Ewelina Rubin    23054        0
hipergat+      juliemeyer juliemey+     Julie Meyer       75        0

The development node is for trying out new code line by line. This is NOT for the submission of bash scripts. I also switch to it if I have jobs in SLURM and I just need to do something relatively simple, like fastqc, quast, or gzipping. You must be logged into a login node first. Then:

module load ufrc
srundev --time=1:00:00

You will get a message like this:

srun: job 37521830 queued and waiting for resources
srun: job 37521830 has been allocated resources
[juliemeyer@dev1 ~]$

You will now see "@dev" next to your name in the prompt. The default time for a development node is 10 minutes. The example here shows 1 hour. You should set the time to something reasonable for your task. I have set it for as much as 10 hours and just stayed logged in all day to run misc tasks. Now that you are logged into the dev node, load the module you want and submit commands line by line (not a whole script). You can also do tasks like gzip, gunzip, mv, sed, grep, awk from the terminal in the dev node.

If you are done with the dev node and want to go back to the login node, just type "exit" and hit enter and you should now see "@login" next to your user name in the prompt.