Using the GPUs at EI - TGAC/knowledge_base GitHub Wiki

At EI we have 4 NVIDIA A100 GPU nodes. Each node has 4 GPU processors, 56 CPUs and 1024Gb of RAM. Each GPU has 80Gb of SSD RAM.

One way to make use of the GPUs is to use ParaBricks, NVIDIA's genomics software suite for secondary analysis that provides GPU-accelerated versions of trusted, open-source tools. This includes tools like BWA and minimap2 for alignment and HaplotypeCaller from GATK. There are many examples in the Parabricks documentation.

This example will use GATK's HaplotypeCaller, used to call SNPs and indels from a BAM file.

Parabricks v4.4.0-1 is installed on the HPC. An sbatch script similar to this can be used to run it.

#!/bin/bash -e
#SBATCH -p ei-gpu                 # specify the GPU partition
#SBATCH -J pb_gatk                # name your job
#SBATCH -N 1                      # number of nodes
#SBATCH -n 1                      # number of tasks
#SBATCH -c 5                      # number of CPUs
#SBATCH --mem 100G                # memory pool for all cores
#SBATCH --time=0-02:00:00         # format is days-hours:minutes:seconds
#SBATCH -o logs/%x.%j.%N.out      # log STDOUT (job name (-J), job id, node name))
#SBATCH -e logs/%x.%j.%N.err      # log STDERR
#SBATCH --gres=gpu:a100:1         # number of GPUs

source package 40b030a5-9757-4084-a0f8-379ed3146631

# HaplotypeCaller is optimised to run on 1 GPU, and defaults to 5 CPUs
pbrun haplotypecaller \
  --num-gpus 1 \
  --ref reference.fa \
  --in-bam aligned_reads.bam \
  --out-variants variants.vcf

Note: HaplotypeCaller uses a tmp directory which by default is created in the current working directory. No speed increase was observed if the local SSD was specified for this. Other pipelines may benefit from using the SSD for temp files if they are particularly i/o heavy.

Submit the script in the usual way, eg.

sbatch <script.sh>

You can check jobs running on the ei-gpu partition using;

squeue -p ei-gpu

And check the resource usage of a GPU node (eg. t1024n1) using;

scontrol show node t1024n1
⚠️ **GitHub.com Fallback** ⚠️