Troubleshooting - KasperSkytte/slurmjobpacker GitHub Wiki
Troubleshooting
Start with the job's slurmctld log line (grep "sjp:") and AdminComment. More detail:
sjp.config.verbose = true.
Not placed
| reason | check |
|---|---|
no table |
sjpd running? not observe? no disable file? table_path matches state_dir? |
stale |
sjpd hung or slow slurmctld (systemctl status sjpd, sjp.log); or a backup controller |
no room |
expected on a full cluster; your fallback applies, or set [policy] always_place |
error |
Lua error just above the sjp line in the slurmctld log |
Other reasons are jobs sjp skips by design (Using sjp in job_submit.lua).
Upgrade has no effect
scontrol reconfigure.
No pins
The verbose line says why (several tasks without -N 1, no memory request, only one node has room…, about equally good, pinning is off).
Pins often released
Higher-priority jobs took the node within 60 s. Each release is logged with the pending reason.
Capped jobs not helped
squeue -h -t PD -o "%r" | sort | uniq -c: onlyQOSMaxCpuPerUserLimitandMaxCpuPerAccountcount.enforce+[limits] modeset, no disable file.idle_fractionin/run/sjp/status.json≥min_idle_share.- Jobs fit one node's free space.
global: ~75 s before a pulse, 5 min between pulses.flex: users without access are named in the log.
Log not written
Non-root users can't write /var/log/sjp: use --state-dir (or --log-file, --text-log).