How it works - KasperSkytte/slurmjobpacker GitHub Wiki

How it works

A daemon, sjpd, reads the cluster every second. A small Lua module, sjp.lua, is called from your job_submit.lua and places each batch job when it is submitted:

  • it chooses the partitions where the job fits the free space best;
  • if the job can start now, it also chooses the node (in the highest PriorityTier partition with room, the node whose free memory per CPU best matches the job's);
  • it returns true when it placed the job, or false and a reason (e.g. "no room") when it could not, leaving the job to you (its AdminComment says why). Your script decides what to do then, e.g. apply its own fallback rule (see Using sjp in job_submit.lua);
  • with [policy] always_place, it places a job even when no node has room: in the partitions its shape fits best, and with [pin] wait_for_room on the node expected to have room for it first;
  • it counts the nodes it just pinned as taken, so a burst of similar jobs (e.g. from a workflow manager) is spread over nodes instead of piling onto one.
flowchart TD
    sjpd["sjpd<br/>reads each node's free CPUs and memory,<br/>i.e. its free memory per CPU"] -. "every second" .-> table[("placement table")]
    submit(["sbatch"]) --> own
    subgraph lua ["job_submit.lua (on slurmctld)"]
        own["your own rules"] --> place{"sjp.place()<br/>does a node have room<br/>for the job now?"}
        place -- "no: returns false, reason" --> fallback["your fallback rule"]
    end
    table -.-> place
    place -- "yes: returns true" --> parts["sets partition and node<br/>matching the job's memory per CPU"]
    place -- "no, with always_place" --> shape["sets partitions by shape,<br/>pins the node with room first"]
    shape --> queue
    parts --> queue(["Slurm queue"])
    fallback --> queue
    queue -- "Slurm starts it" --> running(["running"])
    subgraph pending ["sjpd, while sjp's jobs wait (enforce)"]
        release["pin not started in 60 s:<br/>drop the node"]
        recheck["every minute: choose again,<br/>move the job if there is room,<br/>or to the node with room first"]
        widen["waited too long:<br/>widen to every partition"]
        flex["held by a QOS CPU cap:<br/>flex QOS or raised caps"]
    end
    pending -. "updates" .-> queue
Loading

Slurm still schedules, orders the queue and applies fair-share. While sjp's jobs wait, sjpd keeps its choices current; each step can be switched off:

while a job waits setting
a pinned job that did not start loses its pin and gets its partitions back [pin] release_after (60 s)
sjp chooses again, against the cluster and queue as they are now, and moves the job if the choice changed and a node there has room [recheck] interval (60 s)
a job pinned to wait for room moves to the node now expected to have room first [pin] wait_for_room (off)
a job pending longer than its budget is widened to every partition that can hold it [starvation] enabled
a job held only by a QOS CPU cap that fits on idle nodes moves to a flex QOS, or the caps are raised briefly [limits] mode (off), see QOS limits and flex

The details: Decision making.

⚠️ **GitHub.com Fallback** ⚠️