Troubleshooting - KasperSkytte/slurmjobpacker GitHub Wiki

Troubleshooting

Start with the job's slurmctld log line (grep "sjp:") and AdminComment. More detail: sjp.config.verbose = true.

Not placed

reason check
no table sjpd running? not observe? no disable file? table_path matches state_dir?
stale sjpd hung or slow slurmctld (systemctl status sjpd, sjp.log); or a backup controller
no room expected on a full cluster; your fallback applies, or set [policy] always_place
error Lua error just above the sjp line in the slurmctld log

Other reasons are jobs sjp skips by design (Using sjp in job_submit.lua).

Upgrade has no effect

scontrol reconfigure.

No pins

The verbose line says why (several tasks without -N 1, no memory request, only one node has room…, about equally good, pinning is off).

Pins often released

Higher-priority jobs took the node within 60 s. Each release is logged with the pending reason.

Capped jobs not helped

  1. squeue -h -t PD -o "%r" | sort | uniq -c: only QOSMaxCpuPerUserLimit and MaxCpuPerAccount count.
  2. enforce + [limits] mode set, no disable file.
  3. idle_fraction in /run/sjp/status.json ≥ min_idle_share.
  4. Jobs fit one node's free space.
  5. global: ~75 s before a pulse, 5 min between pulses.
  6. flex: users without access are named in the log.

Log not written

Non-root users can't write /var/log/sjp: use --state-dir (or --log-file, --text-log).