Running DL software in singularity containers - twongjirad/LArLiteSoftCookBook GitHub Wiki

Just some quick notes for now. A more thorough explainer will follow.

• Use NVIDIA docker images as the base to the container stack

• Build docker container with DL software. For us: caffe-larbys, dllee-ssnet.

• Convert docker container into singularity container. During this conversion, use gpu4singularity to install drivers matching the system to be run on

• when running the container, make sure to use the singularity argument --nv

• if one needs to run multiple copies on one machine, define the environment variables in the script running inside the container

  export MKL_NUM_THREADS=1
  export OMP_NUM_THREADS=1
  

Note that the last fix comes when the following error is seen:

OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 1031988 max

I followed this page to fix it.