Visualizer - KasperSkytte/slurmjobpacker GitHub Wiki

Visualizer

Live view of the cluster: python3 -m sjp.viz.

3D view of a demo cluster: three slim and three fat nodes, and waiting jobs on the left

3D

Each node is a box of CPUs × memory × time; each running job a box inside it, as deep as its time limit and filled up to its run time (white line). On the front, red is what the jobs use (CPUs × memory), green what is free. The dashed diagonal is the node's memory per CPU. Drag to turn, right-drag to move, click a job.

2D (/2d)

2D view of the same cluster: CPU and memory bars per node

The same in CPUs and memory only, as a pair of bars per node: free on the left, used / total on the right, the free memory per CPU in between.

Waiting jobs are in squeue's order (priority, then job ID), the top 50 of them. Jobs waiting for one node (sjp's pins, or --nodelist) are drawn at that node: in 2D in a column right of its card, in 3D as tiles in front of it (the top 10 per node). A job's details (click it) include its reservation and sjp's AdminComment marks (Logs and AdminComment).

A node that takes no jobs is red, with what keeps jobs off it in the middle (DOWN, DRAIN, NOT_RESPONDING, MAINTENANCE, INVALID_REG, REBOOT_ISSUED, FAIL, ...) and the reason an admin gave. sjp places no jobs there. Power saving, RESERVED and REBOOT_REQUESTED don't count.

Reservations are purple, listed per node with start, end and length. In 2D a reservation is one outlined block on the CPU bar: its own jobs, then its unused cores striped (for a whole node, its free memory too); click it for its details. In 3D only a whole-node reservation colours the node, since memory cannot be reserved on its own.

Show (top panel) picks one user (any with a job running or waiting) or one account (any in the Slurm database): the others' running jobs fade, and only theirs wait in the queue and at the nodes. It goes into the URL (?user=, ?account=), so a link shows the same.

Options

The options panel (click "options", or press ?) has a switch for each setting; the key beside it does the same. Settings are remembered in the browser; the URL parameter starts with one set.

key URL
b nodes by how busy they are (on): blue when idle, yellow at half, red when every CPU is allocated. In 2D the used / total figures are always coloured so ?busy=0
c job colours by memory per CPU, instead of fit to the node ?color=mpc
p jobs waiting for one node at that node (on), instead of in the queue ?pinned=queue
a all waiting jobs, instead of the top 50 in the queue and 10 per node ?waiting=all
l job labels ?labels=1
r rotate (3D)
h hide the panels

Running it

cd /opt/slurmjobpacker && python3 -m sjp.viz -c /etc/sjp/sjp.toml

Serves http://127.0.0.1:8650, reads Slurm every 5 s (--interval), read-only. Shows the nodes sjp places jobs on ([topology]). Localhost only by default, since the page shows users and job names:

ssh -L 8650:localhost:8650 <controller>      # then open http://localhost:8650

As a service:

sudo cp systemd/sjp-viz.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now sjp-viz

Ansible: sjp_viz_enabled: true.

Demo and videos

python3 -m sjp.viz --demo: simulated cluster (3 slim + 3 fat nodes, --slim/--fat), 900× speed (--speed), jobs placed as sjp would, half of those waiting pinned to the node expected to have room first (--pinned). No Slurm needed.

tools/record_demo.py records it frame by frame:

python3 -m venv ~/sjp-rec && ~/sjp-rec/bin/pip install playwright imageio-ffmpeg
~/sjp-rec/bin/python -m playwright install chromium
python3 -m sjp.viz --demo --port 8651 &
~/sjp-rec/bin/python tools/record_demo.py demo.mp4 --seconds 10
~/sjp-rec/bin/python tools/record_demo.py phone.mp4 --size 1080x1920 --ui 2 \
    --slim 1 --fat 1 --load 2.2 --orbit -0.2          # vertical, for phones

--view 2d records the 2D view. ?ui=2 enlarges the panels in the browser.

⚠️ **GitHub.com Fallback** ⚠️