What it is for. A field guide drawn from recent incidents. Each entry reads symptom, cause, check, fix. Version numbers say from which package version the cause is fixed in the source; on an older box, upgrade the package first (see Operations-APT-Repository).

#	Symptom	Section
1	disk full	Disk full
2	timer with no next run	Timers
3	new routes return 404 / aggregator restart	Aggregator
4	WAF no longer writes logs	sbxwaf logs
5	WAF shown "out of service" while it runs	WAF state
6	valid certificates shown expired	Certificates
7	network stats always "stale"	Network stats
8	module restart loop, stale socket	Sockets
9	nginx will not start	nginx :80

• Symptom. Services fail to write, dpkg cannot write its own state file, packages stay unconfigured, UIs return 5xx. Seen on a build host and on an undersized full image (the build sizes are now 8G for isp, 12G for full).
• Cause. On a development host: accumulated agent worktrees and build output. On a box: a small root filesystem and data on the wrong volume (media and caches belong on the data disk).
• Check. df -h, then du -xh --max-depth=1 / on the full volume. For worktrees: scripts/agent-worktree.sh list --verbose.
• Fix. Remove finished worktrees with scripts/agent-worktree.sh clean <issue> (it refuses a dirty worktree or a branch whose pull request is not merged, unless --force). Before deleting a worktree by hand, look at what git ignores (git status --ignored): an unanchored .gitignore pattern can hide real source files. Clean the apt cache with apt-get clean. Never free space by deleting under /etc/secubox or /var/lib/secubox.

• Symptom. A job never runs; systemctl list-timers --all shows n/a in the NEXT column for a SecuBox timer.
• Cause. The timer unit is not active (not enabled, stopped by a package upgrade, or failed), or a timer driven only by OnUnitActiveSec= whose service never ran has nothing to count from. Package postinst scripts must restart a unit that the upgrade stopped.
• Check. systemctl list-timers --all, systemctl status <name>.timer, and the effective cadence with systemctl cat <name>.timer (the cadence drop-ins 10-cadence.conf come from secubox-core; they first reset OnUnitActiveSec= then set the new interval, with a RandomizedDelaySec=).
• Fix. systemctl daemon-reload after any drop-in change, then systemctl enable --now <name>.timer. A package that leaves its timer stopped after an upgrade has a postinst bug: report it with the package version.

• Symptom. A new or changed route of a module answers 404, or a whole set of /api/v1/<module>/ routes answers 502.
• Cause. secubox-aggregator imports the code of every aggregated module at startup (about 110 modules in one process, no module runs its own daemon). A module's new code is only taken into account after the aggregator restarts. The cold start takes time and the sockets are unavailable meanwhile.
• Check. systemctl is-active secubox-aggregator; journalctl -u secubox-aggregator for import errors. A module on its own socket is independent of the aggregator.
• Fix. systemctl restart secubox-aggregator. Expect a cut of about 40 seconds for every aggregated module; plan it, and never restart many secubox-* daemons in parallel. The watchdog allows a startup grace of 480 s before its own auto-heal, so do not restart it again while it is still loading.

• Symptom. /var/log/secubox/waf/waf-threats.log stops growing; WAF panels go empty although traffic flows.
• Cause. The log directory lost its owner or mode. It must be secubox-waf:secubox-waf, mode 0750; the shared parent /var/log/secubox stays 0755 (setting it to 0750 breaks traversal for the other services). The cache directory /var/cache/secubox/waf has the same owner.
• Check. ls -ld /var/log/secubox /var/log/secubox/waf /var/cache/secubox/waf; journalctl -u secubox-waf-ng.
• Fix. Reinstall or upgrade secubox-waf-ng (its postinst recreates the directories with the right owner and mode; the unit prepares the cache itself since 1.27.3), then systemctl restart secubox-waf-ng. Do not use a hand-written drop-in: a corrupted one (backticks in an unquoted heredoc) once broke the unit.

• Symptom. The global health (portal, banner) says the WAF is out of service while sbxwaf is running.
• Cause. The health check still looked at the old mitmproxy container. Since secubox-metrics 1.19.1 (#2391) the state follows secubox-waf-ng (service active and port 8085 answering).
• Check. systemctl is-active secubox-waf-ng and ss -ltn for port 8085 on the loopback.
• Fix. Upgrade secubox-metrics to 1.19.1 or later. If the service is really down, systemctl restart secubox-waf-ng (a restart, not a reload signal).

• Symptom. The Certificates page lists valid certificates as expired (minus 999 days).
• Cause. Files in /data/haproxy/certs were root mode 0600: the API runs as secubox and could not read them, and an unreadable file counted as expired (#2383). secubox-certs 1.2.8 now shows such a certificate as unknown ("illisible"), and secubox-haproxy 1.10.3 writes the .pem as 0640 haproxy:secubox, like certsctl deploy.
• Check. ls -l /data/haproxy/certs.
• Fix. Upgrade both packages, then make the existing files readable by the group (chown haproxy:secubox, chmod 0640). A failed read is no longer cached, so the next scan picks them up.

• Symptom. "Live network" in the SOC and the Network tab of the Toolbox show "COLLECTOR STOPPED?" while the collector runs.
• Cause. The staleness threshold was lower than the collector cadence. The collector timer secubox-netstats.timer runs every 120 s plus up to 30 s of random delay. secubox-hub 1.9.55 (#2388) sets the threshold to 360 s (two cadences plus a margin), tied to the cadence drop-in by a test.
• Check. systemctl cat secubox-netstats.timer; compare the interval with the threshold. If you slow the cadence down, raise the threshold above two cadences.
• Fix. Upgrade secubox-hub to 1.9.55 or later. If the collector is really stopped: systemctl status secubox-netstats.timer.

• Symptom. A module restarts in a loop, with Unable to check or remove stale UNIX socket: PermissionError(1) then Address already in use; its routes answer 502. Or many modules lose their socket at once after a restart.
• Cause. /run/secubox is shared by about 95 units and is sticky (1777, root:root): a process can only remove a file it owns, so a socket left by another user blocks the daemon. The opposite fault, RuntimeDirectory=secubox in a unit, wipes the sockets of every neighbour on restart.
• Check. ls -l /run/secubox. For the effective setting (never grep, comments produce false hits): systemctl show <unit> -p RuntimeDirectory -p RuntimeDirectoryPreserve --value.
• Fix. Every unit that binds a socket there carries ExecStartPre=+/bin/rm -f /run/secubox/<module>.sock (the + runs it as root). Never chown the parent and never declare RuntimeDirectory=secubox. Rules: section 2 of the repository file RULES-CODE.md.

• Symptom. nginx fails with bind() to 0.0.0.0:80 failed (98: Address already in use), and every host vhost goes down with it. It can stay invisible for weeks because a reload does not rebind.
• Cause. On a SecuBox, HAProxy holds 0.0.0.0:80 (and :443). Any host vhost with listen 80 prevents nginx from starting; the first real restart then fails.
• Check. nginx -t, then search the enabled vhosts for listen 80. Several nginx instances can also run inside containers: check the cgroup before killing a process.
• Fix. Make the vhost listen on its module port instead (the HAProxy to sbxwaf chain already routes there) and restart nginx. Traffic must go through sbxwaf; do not bypass it by hand (only the declarative waf_bypass = true per vhost in /etc/secubox/haproxy.toml). See WAF.

Operations-APT-Repository · Operations-Release-Process · Troubleshooting · FAQ-Troubleshooting · WAF · Profils · Operations-Runbook-FR