Dry run and going live

Dry run

As any user, no config needed:

cd /opt/slurmjobpacker
python3 -m sjp.daemon --dry-run --state-dir ~/sjp-dry
tail -f ~/sjp-dry/sjp.log

Logs, per new job, where Slurm put it and where sjp would have, plus every action it would take (command and reason). Nothing is changed:

• --dry-run forces observe;
• outside enforce, only scontrol show, squeue, sacctmgr -nP show and sacct can run;
• the table goes to policy.dryrun.lua, which the plugin doesn't read.

Summary: python3 -m sjp.report ~/sjp-dry/decisions.jsonl --summary.

Going live

1. Run sjpd in observe (no table is written; sjp.place() returns "no table").
2. Add sjp.place() to job_submit.lua, ensure JobSubmitPlugins=lua, scontrol reconfigure.
3. mode = "advise", restart sjpd: partitions are chosen.
4. mode = "enforce", restart sjpd: pins, rechecks, widening and QOS limits too.

Backing out

• touch /etc/sjp/disable: sjp.place() returns false from the next submission.
• mode = "observe" + restart: the plugin sees stale data within seconds; raised caps are restored on stop.
• Remove the sjp.place() call.

Release remaining pins by hand:

for j in $(squeue -h -t PD -o %i); do
  c=$(scontrol show job $j | grep -o 'AdminComment=sjp:pin=[^ ]*') || continue
  from=$(echo "$c" | grep -o 'from=[^;]*'); from=${from#from=}
  scontrol update jobid=$j reqnodelist= && scontrol update jobid=$j partition=$from
done