Julia on LUMI - gher-uliege/Documentation GitHub Wiki
See this document to connect to LUMI via SSH: https://docs.lumi-supercomputer.eu/firststeps/loggingin/
Also possible to access via a web GUI: https://www.lumi.csc.fi/pun/sys/dashboard/
It is recommended to add the following environment variable to your .bashrc
export SLURM_ACCOUNT=project_465003122Alternatively, you can use the slurm option --account=project_465003122.
List the partitions with sinfo:
$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
debug up 30:00 1 comp nid002500
...
dev-g up 3:00:00 20 mix nid[005003,005007-005010,007954-007955,007965-007977]
dev-g up 3:00:00 15 alloc nid[005004,005006,005015,005018,005020-005022,005024-005025,007956-007961]
dev-g up 3:00:00 12 idle nid[005002,005005,005011-005014,005016-005017,005023,007962-007964]
small-g up 3-00:00:00 5 comp nid[005034,005066,007881,007929,007945]
...
We reverse 1 GPU and 8 CPU cores on the dev-g partition with 70G memory for 2 hours:
srun --interactive --pty --partition dev-g --nodes=1 --gpus=1 --ntasks=1 --cpus-per-task=8 --time=2:00:00 --mem=70G bashmodule load Local-CSC julia/1.12.0
export JULIA_DEPOT_PATH=/tmp/julia-depot-$USER-$SLURM_JOBID:/appl/local/csc/soft/math/julia/1.12.0/share/julia
juliaInstall AMDGPU
]add AMDGPU
Some quick tests:
using AMDGPU # v2.5.1
# Create random array of 10 000 by 10 000 elements in single precision
A = randn(Float32,10_000,10_000);
# Compute the product A A^T on the CPU
B = @time A*A';
# 5.052050 seconds (12 allocations: 381.473 MiB, 0.08% gc time)
# The first time can be slower due to compilation overhead
B = @time A*A';
# 5.046070 seconds (12 allocations: 381.473 MiB, 0.02% gc time)
# Transfer to GPU
A_d = AMDGPU.roc(A);
# Check the type
typeof(A_d)
# ROCArray{Float32, 2, AMDGPU.Runtime.Mem.HIPBuffer}
# Repeat the computation
B_d = @time A_d*A_d';
# 3.473769 seconds (1.41 M allocations: 71.980 MiB, 14.92% compilation time)
# A second time
B_d = @time A_d*A_d';
# 0.000217 seconds (28 allocations: 912 bytes)
maximum(abs.(Array(B_d) - B))
# 0.0703125f0
maximum(abs.(B))
# 10598.61f0
# type ≈ as \approx<TAB>
B ≈ Array(B_d)
# true
More information about the hardware and the used library.
julia> AMDGPU.versioninfo()
[ Info: AMDGPU versioninfo
┌───────────┬──────────────────┬───────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Available │ Name │ Version │ Path │
├───────────┼──────────────────┼───────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ + │ LLD │ - │ /opt/rocm-6.3.4/lib/llvm/bin/ld.lld │
│ + │ Device Libraries │ - │ /tmp/julia-depot-barthale-19521177/artifacts/0a0cad46b7dacc66387ed19e8c0f45b097f46691/amdgcn/bitcode │
│ + │ HIP │ 6.3.42134 │ /opt/rocm-6.3.4/lib/libamdhip64.so │
│ + │ rocBLAS │ 4.3.0 │ /opt/rocm-6.3.4/lib/librocblas.so │
│ + │ rocSOLVER │ 3.27.0 │ /opt/rocm-6.3.4/lib/librocsolver.so │
│ + │ rocSPARSE │ 3.3.0 │ /opt/rocm-6.3.4/lib/librocsparse.so │
│ + │ rocRAND │ 4.2.0 │ /opt/rocm-6.3.4/lib/librocrand.so │
│ + │ rocFFT │ 1.0.31 │ /opt/rocm-6.3.4/lib/librocfft.so │
│ + │ MIOpen │ 3.3.0 │ /opt/rocm-6.3.4/lib/libMIOpen.so │
└───────────┴──────────────────┴───────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────┘
[ Info: AMDGPU devices
┌────┬─────────────────────┬────────────────────────┬───────────┬────────────┬───────────────┐
│ Id │ Name │ GCN arch │ Wavefront │ Memory │ Shared Memory │
├────┼─────────────────────┼────────────────────────┼───────────┼────────────┼───────────────┤
│ 1 │ AMD Instinct MI250X │ gfx90a:sramecc+:xnack- │ 64 │ 63.984 GiB │ 64.000 KiB │
└────┴─────────────────────┴────────────────────────┴───────────┴────────────┴───────────────┘
Exercise: run this snipped on your laptop:
A = randn(Float32,10_000,10_000);
B = @time A*A';What is going on??
A simple convolution:
using Flux, AMDGPU; m = gpu(Conv((3,3),3 => 8)); x = gpu(randn(Float32,16,16,3,1)); m(x);fails with the error:
MIOpen(HIP): Warning [Find] /long_pathname_so_that_rpms_can_package_the_debug_info/src/extlibs/MLOpen/src/include/miopen/sqlite_db.hpp:260: Cannot open database file:/tmp/gfx90a6e_1.1.0.udb
MIOpen(HIP): Warning [Find] /long_pathname_so_that_rpms_can_package_the_debug_info/src/extlibs/MLOpen/src/include/miopen/sqlite_db.hpp:260: Cannot open database file:/tmp/gfx90a6e_1.1.0.udb
MIOpen(HIP): Warning [Find] /long_pathname_so_that_rpms_can_package_the_debug_info/src/extlibs/MLOpen/src/include/miopen/sqlite_db.hpp:260: Cannot open database file:/tmp/gfx90a6e_1.1.0.udb
MIOpen(HIP): Warning [Find] /long_pathname_so_that_rpms_can_package_the_debug_info/src/extlibs/MLOpen/src/include/miopen/sqlite_db.hpp:260: Cannot open database file:/tmp/gfx90a6e_1.1.0.udb
MIOpen(HIP): Warning [Find] /long_pathname_so_that_rpms_can_package_the_debug_info/src/extlibs/MLOpen/src/include/miopen/sqlite_db.hpp:260: Cannot open database file:/tmp/gfx90a6e_1.1.0.udb
MIOpen Error: /long_pathname_so_that_rpms_can_package_the_debug_info/src/extlibs/MLOpen/src/ocl/convolutionocl.cpp:299: No suitable algorithm was found to execute the required convolution
ERROR: MIOpenException:
- status: miopenStatusUnknownError
- description: Unknown error
export MIOPEN_USER_DB_PATH="/tmp/my-miopen-cache"
export MIOPEN_CUSTOM_CACHE_DIR=${MIOPEN_USER_DB_PATH}
rm -rf ${MIOPEN_USER_DB_PATH}
mkdir -p ${MIOPEN_USER_DB_PATH}
Replace <node_name> with the actual node identifier and <jobid> with the ID of your SLURM job
srun --overlap --pty --jobid=<jobid> -w <node_name> rocm-smi --showuse
srun --overlap --pty --jobid=13486921 top
See ~/.julia/dev/FlowMatching/examples/training.sh
salloc --account=project_465001568 --partition dev-g --nodes=1 --gpus=1 --ntasks=1 --time=2:00:00 --mem-per-cpu=25G
Prepare interactive session:
export MIOPEN_USER_DB_PATH="/tmp/my-miopen-cache-$USER-$$"
export MIOPEN_CUSTOM_CACHE_DIR="$MIOPEN_USER_DB_PATH"
rm -rf "$MIOPEN_USER_DB_PATH"
mkdir -p "$MIOPEN_USER_DB_PATH"
DEPOT_FILE=$HOME/julia-depot-FlowMatching.tar.xz
DEPOT="/tmp/$(basename "${DEPOT_FILE%.tar.xz}")-$USER"
export JULIA_NUM_THREADS="$SLURM_CPUS_PER_TASK"
export JULIA_DEPOT_PATH="$HOME/.julia:$DEPOT:$JULIA_DEPOT_PATH"
export JULIA_HISTORY="$HOME/.julia/logs/repl_history.jl"
if [ -e Project.toml ]; then
export JULIA_PROJECT="$PWD"
fi
# https://docs.lumi-supercomputer.eu/development/compiling/prgenv/#gpu-aware-mpi
export MPICH_GPU_SUPPORT_ENABLED=1
# on each node but only once per node
echo "extracting $DEPOT_FILE"
srun flock --nonblock --conflict-exit-code=0 "/tmp/julia-depot-lock-$SLURM_JOBID" tar -xf $DEPOT_FILE -C /tmp
srun ls $DEPOT
if [ "$SLURM_NTASKS" -gt "1" ]; then
export PARALLEL=true
else
export PARALLEL=false
fi
#export ENABLE_JITPROFILING=1
#srun rocprofv2 --plugin perfetto --hip-trace --hsa-trace --kernel-trace -o prof julia training.jl
srun --interactive --pty julia
# or
# srun julia some_script.jl
sbatch ~/projects/bin/mkdepot
$ module li
Currently Loaded Modules:
- craype-x86-rome 4) perftools-base/24.03.0 7) craype/2.7.31.11 10) cray-libsci/24.03.0 13) lumi-tools/24.05 (S) 16) julia/1.11.2
- libfabric/1.15.2.0 5) xpmem/2.8.2-1.0_5.1__g84a27a5.shasta 8) cray-dsmml/0.3.0 11) PrgEnv-cray/8.5.0 14) init-lumi/0.2 (S)
- craype-network-ofi 6) cce/17.0.1 9) cray-mpich/8.1.29 12) ModuleLabel/label (S) 15) Local-CSC/default (S)
cc --cray-print-opts=all
$ mpicc --cray-print-opts=all -L/opt/cray/pe/cce/17.0.1/cce/x86_64/lib/pkgconfig/../ -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup
$ lumi-check-quota $ lumi-quota lumi-workspaces
https://docs.lumi-supercomputer.eu/runjobs/lumi_env/dailymanagement/
Keep track of the allocated resources at:
The resources are allocated for the group (GHER).