Decision making - KasperSkytte/slurmjobpacker GitHub Wiki

Decision making

Placeable capacity

Free space counts only as far as typical jobs can use it. For a node with fc free CPUs and fm MB free memory:

phi = Σ over the demand mix of  weight × min(fc, fm / mem_per_cpu)

The demand mix ([policy] demand) is a weighted list of typical memory-per-CPU requests. Fit your own with python3 -m sjp.fit --since 2026-01-01. Placing a job costs the phi it destroys.

Partitions

sjpd precomputes a table: jobs bucketed by memory per CPU, CPUs and time limit ([policy] buckets); each partition scored by its best node with room. Allowed: the cheapest partition plus any within tolerance (0.25 CPU per job CPU). No room anywhere: every partition big enough. Slurm then tries them in PriorityTier order.

Idle cluster → jobs go where their ratio fits. Busy cluster → costs converge, so everything big enough is allowed. Time limit is a bucket key but doesn't affect scores yet.

Reservations

Treated as planned jobs: whole nodes or the listed cores (× threads), memory at the node's memory per CPU.

  • Active: counted as allocated (less what the reservation's own jobs use), everywhere.
  • Upcoming: counted against jobs whose time limit would reach them (all of them for jobs without a limit), in the room check, pins and flex moves.

At submission

sjp.place():

  1. skips interactive, GPU, reservation, --constraint, multi-node and --mem=0 jobs;
  2. looks the job up and drops partitions too small for it; for a job asking for nodes (--nodelist, --exclude), considers only those nodes, and partitions holding them (each holding every --nodelist node), or with [policy] user_nodes = "ignore" drops the request;
  3. if no node in them has the CPUs and memory free now: returns false, "no room", or with always_place uses the partitions the job's shape fits best on an empty cluster (marked room=no);
  4. sets partitions, and the node for a pin;
  5. writes AdminComment, returns true. A job it skips gets sjp:skipped=<reason>.

Table older than [cadence] policy_max_age (300 s) or node data older than [pin] max_age (10 s): false, "stale".

Node pins (enforce)

For a job that can start now, sjp sets --nodelist:

  • only single-node jobs with an explicit memory request; not --nodelist (the user chose, unless user_nodes = "ignore"), --gres, --exclusive, held, future --begin, or several tasks without -N 1; not when the user is at the QOS CPU cap;
  • not an array at submission: one node list would hold all its tasks on one node. The array is placed by one task's shape, and each task waiting is then a job of its own;
  • within the highest-PriorityTier partition with room;
  • among nodes within min_gain (1 CPU of phi) of the best, the closest memory per CPU wins;
  • no pin if only one node fits, or all are about equal (min_gain, min_ratio_gain).

Pins not started within release_after (60 s) are removed and partitions restored. Pending jobs tied to one node (pins or --nodelist) count as taken there, so a burst of submissions isn't pinned to the same free space.

Time-aware pins

[pin] time_aware = true adds to the node choice:

time_cost = max(0, log(job time limit / time until the node's jobs end))

Empty nodes count as busy for 1 minute, so jobs join busy nodes and long jobs gather. The plugin also counts the end times of its own pins for 30 s, since sjpd reads the queue only every 8 s.

Waiting for room (always_place, wait_for_room)

With [pin] wait_for_room, a job no node has room for is pinned to the node expected to have room for it first: its free space, less the jobs already waiting there, plus what its running jobs release by their time limits. Ties: higher PriorityTier, then name. Marked eta=. Not released after release_after; the recheck moves it, and it is released once the job is past its [starvation] budget (then widened as usual).

Rechecks (enforce)

Jobs that start, end or are cancelled change what fits where, so a choice made at submission goes stale. Every minute ([recheck] interval) sjp chooses again for each pending job it placed (None, Resources, Priority or waiting on its pinned node) and moves it if the choice changed and a node there has room, dropping any wait pin. A job that still cannot start is re-pinned to the node now expected to have room first (wait_for_room). Highest priority first, only as many as fit, so a job whose priority you raise is moved first. Jobs the plugin skipped for want of room (skipped=no room, or stale) are taken over the same way once room opens: your rules placed them only as a fallback. Not jobs skipped for other reasons, nor jobs on the user's own --nodelist. Each waiting array task is rechecked, moved and pinned on its own (Slurm then gives it a job record of its own); a waiting job stays on its node unless room is expected sooner elsewhere.

Long waits (enforce)

Jobs sjp placed that pend for Resources or Priority beyond their budget (1 h at ≥ 6000 MB/CPU, 4 h otherwise, from eligibility) are widened once to every partition big enough ([starvation] enabled); each array task on its own. Not GPU, pinned or multi-node jobs, nor jobs your own rules placed.

⚠️ **GitHub.com Fallback** ⚠️