slurm_quick - OpenNebula/one-apps GitHub Wiki

Quick Start

This guide provides the minimum CLI steps to deploy a Slurm cluster with one Controller and an arbitrary number of Workers.

Before deploying the service, make sure that OneGate and OneFlow are running in the OpenNebula cloud. OneGate is required because the Controller publishes the cluster coordination data used by Workers, including controller readiness, the Munge key, and optional LDAP metadata. The OneGate endpoint must be reachable by both Controller and Worker VMs, either directly or through OpenNebula transparent proxies.

OneGate must also be configured to reach the active OneFlow API endpoint. From a running service VM, onegate service show must return the OneFlow service document. If it returns Service <id> not found, check the :oneflow_server: setting in /etc/one/onegate-server.conf.

The Marketplace service enables ready_status_gate. OneFlow keeps the Worker role pending until the Controller publishes READY=YES through OneGate. The Controller appliance sets that attribute when configuration succeeds, and the imported Controller and Worker templates include REPORT_READY=YES so one-context can also report readiness after a successful appliance bootstrap.

Workers must also be able to reach the Controller on TCP port 6817, which is used by slurmctld and configless Slurm.

Deploy the Slurm Service

  1. Download the Service OneSlurm appliance from the OpenNebula Marketplace. This imports the service definition, VM templates, and disk images needed for the Controller and Worker roles:

    $ onemarketapp export 'Service OneSlurm' 'Service OneSlurm' --datastore default
IMAGE
    ID: 34
    ID: 35
VMTEMPLATE
    ID: 62
    ID: 63
SERVICE_TEMPLATE
    ID: 64

  2. (Optional.) Adjust the Flow and VM templates. Common changes are Worker CPU and memory, NUMA pinning on pinned hosts, GPU PCI devices, InfiniBand PCI devices, and shared NFS home or scratch exports.

    $ oneflow-template update 64
$ onetemplate update 62
$ onetemplate update 63

    To run GPU jobs, attach a GPU PCI device or GPU PCI profile to the Worker VM template. For example, an NVIDIA H100 PCIe request:

    PCI = [
  VENDOR = "10de",
  DEVICE = "2331",
  CLASS  = "0302" ]

    The exact GPU device ID and available PCI profile names are deployment-specific.

    To run MPI/RDMA jobs over InfiniBand, attach an InfiniBand HCA or PCI profile to the Worker VM template. The InfiniBand fabric and subnet manager must already be configured outside the appliance.

    To use shared in-guest storage, prepare NFS exports that are reachable from the Controller and Worker VMs. The service can mount one export at /home and one export at /scratch.

  3. Instantiate the Flow service:

    $ oneflow-template instantiate 'Service OneSlurm'
There are some networks that require user input. Use the string <<EDITOR>> to launch an editor (e.g. for multi-line inputs)
  * (Service) Service
    TYPE Existing(1), Create(2), Reserve(3). 
1
    VN ID. 
0
  * (ONEAPP_LDAP_ENABLE) Enable controller local LDAP server
    Press enter for default (NO). 
  * (ONEAPP_LDAP_DOMAIN) LDAP domain (base DN derived from it)
    Press enter for default (slurm.local). 
  * (ONEAPP_LDAP_ADMIN_USER) LDAP admin user
    
  * (ONEAPP_LDAP_ADMIN_PASSWORD) LDAP admin password
    Password: 
  * (ONEAPP_LDAP_URL) External LDAP URL (used only when local LDAP is disabled)
    
  * (ONEAPP_LDAP_BIND_USER) LDAP bind user (optional)
    
  * (ONEAPP_LDAP_BIND_PASSWORD) LDAP bind password (optional)
    Password: 
  * (ONEAPP_SLURM_INFINIBAND_ENABLE) Enable InfiniBand on workers
    Press enter for default (NO). 
  * (ONEAPP_SLURM_IPOIB_SUBNET) IPoIB subnet (required when InfiniBand is enabled)
    
  * (ONEAPP_SLURM_NFS_HOME) NFS export for /home (host:/export)

  * (ONEAPP_SLURM_NFS_SCRATCH) NFS export for /scratch (host:/export)

ID: 71

    Answer the prompts to select the OpenNebula virtual network for Service (existing VNet ID in the example) and to configure LDAP if needed. Accept the defaults to leave local LDAP, InfiniBand, and NFS mounts disabled. To enable InfiniBand, set ONEAPP_SLURM_INFINIBAND_ENABLE=YES and provide an IPoIB subnet such as 10.20.0.0/24. To mount shared storage, set ONEAPP_SLURM_NFS_HOME and/or ONEAPP_SLURM_NFS_SCRATCH to NFS exports such as 10.125.0.1:/srv/nfs/slurm/home and 10.125.0.1:/srv/nfs/slurm/scratch. OneFlow attaches the selected network to the Controller and Worker roles. With ready_status_gate enabled, OneFlow deploys the Worker only after the Controller reports ready.

  4. Wait until the service reaches RUNNING:

    $ oneflow list
  ID USER     GROUP    NAME                  STARTTIME STAT
  71 oneadmin oneadmin Service OneSlurm  08/03 09:18:32 RUNNING
    $ onevm list -f NAME~'service_71' -l NAME,STAT
NAME                         STAT
controller_0_(service_71)    runn
worker_0_(service_71)        runn

During Controller configuration, the appliance:

  • Sets its hostname to slurm-one-controller.
  • Generates and verifies the Munge key.
  • Starts slurmctld.
  • Mounts configured NFS exports at /home and/or /scratch.
  • Publishes READY=YES and SLURM_MUNGE_KEY through OneGate.
  • Publishes LDAP metadata when LDAP is enabled.

Worker VMs then discover the Controller through OneGate. Each Worker:

  • Waits until the Controller has published READY=YES.
  • Reads the Controller network address and SLURM_MUNGE_KEY from OneGate.
  • Maps the Controller IP to slurm-one-controller in /etc/hosts.
  • Verifies connectivity to the Controller on TCP port 6817.
  • Mounts configured NFS exports at /home and/or /scratch.
  • Installs the Munge key and starts slurmd in configless mode.
  • Publishes its SLURM_NODENAME through OneGate for lifecycle reconciliation.

Once the Workers are configured, check the registered nodes from the Slurm Controller:

$ scontrol show nodes

To add more capacity later, scale the Worker role:

$ oneflow scale <service_id> worker <cardinality>

New Workers follow the same automatic discovery and registration flow.

For GPU clusters, check that Slurm sees the GPU resource from the Controller:

$ scontrol show nodes
$ srun -N1 -n1 --gres=gpu:1 nvidia-smi -L

For InfiniBand clusters, verify the IPoIB interface on each Worker and MPI configuration from the Controller:

$ ip addr show ib0
$ ibstat
$ scontrol show config | grep -E 'MpiDefault|PropagateResourceLimitsExcept'

For NFS-backed home or scratch storage, verify the mounts from the Controller and from at least one Worker:

$ findmnt /home
$ findmnt /scratch
$ srun -N1 -n1 findmnt /home
$ srun -N1 -n1 findmnt /scratch

Compile MPI programs on the Controller with mpicc and launch them with srun. When InfiniBand is enabled, the Controller already sets MpiDefault=pmix, so Slurm-launched jobs use PMIx (and can take the InfiniBand path) without extra MPI flags:

$ mpicc mpi_program.c -o mpi_program
$ srun -N2 ./mpi_program

Next: Slurm Features and usage

⚠️ **GitHub.com Fallback** ⚠️