docs backend evidence community safety SEC09D_STAGING_RUNBOOK - AlexandreZanata/Abastevo GitHub Wiki
SEC09-D — Abastevo-only staging runbook and rollback
Date: 2026-10-11. Status: LOCAL_DONE. Scope: runbook document only —
no rollout, migration, or VPS mutation. Head: maintained dev + this
task.
Plan: remediation plan
(SEC09). Contract: staging topology (infra/compose.staging.yml:
db 2 cpu/6 G, api 2 cpu/1 G, worker 2 cpu/2 G, caddy 1 cpu/512 M);
existing guards (aggregate/per-key quotas, global breaker, IntakeHealth,
sweep/audit breakers, /health/ready). Constraint: Abastevo-only —
no shared workloads/proxy/images/volumes/data changes; historical
5 req/s windows are NOT automatic budgets; spare capacity is measured
in the window, never assumed.
Validation: PASS
Preconditions (all required, authorized window only)
- Explicit user authorization naming the release tag
(
ANPFUEL_RELEASE, neverlatest) and the window. - Staging env file operator-owned, never committed; DB password set;
docker compose -f infra/compose.staging.yml configclean. - Spare capacity measured on the host first (CPU idle, free mem, disk, current staging load); rollout rate fits inside measured headroom with 50% reserve. No number below is a substitute.
Rollout (db → migrate → api/worker → caddy)
up -d db, wait healthy (pg_isready, ≤10 retries).run --rm migrate(one-shot, lock/checksum ledger); abort on any migration error — never retry blindly, never skip.- Rebuild + recreate
api/workerat the release tag; verify/health/ready200 before traffic; smoke: anonymous read 200, signed write 201/409-shape, 401 without proof. - Switch
caddylast; confirm TLS + front-only exposure (no origin-port mapping; infra gate).
Rate bounds (measured, not historical)
- Start at 1 req/s synthetic, step only while p95 mixed-route latency, error rate, and backlog stay flat; stop well below any guard trip.
- The historical 5 req/s windows are prior observations, not budgets or maximum-user estimates — each rollout re-measures.
Stop conditions (any one halts and rolls back)
- p95 breach vs. window baseline, any 5xx burst, growing job backlog, overdue-copy alarm, reserve/byte-budget trip, OOM/restart of any service, or operator judgment. Guards tripping is a stop, not a tuning prompt.
Rollback
- Repoint
caddyto the previous release tag (kept); recreateapi/worker; verify/health/ready+ smoke. Migrations are append-only and never rolled back — forward-fix only; data stays. - Record the stop reason + head + timestamp in PROGRESS; failed rollout never retries in the same window without a new cause.
Antiblast (Abastevo-only)
- Touch only
anpfuel-staging-*networks/volumes/containers; host proxy, other workloads, shared images/volumes, and production data are out of scope and untouched. DB has no published ports.
Validation (actual)
- Document-only: topology limits, guard names, and command shapes
verified against current
compose.staging.yml, health wiring, and quota/breaker code references. git diff --checkclean; no code/test/infra touched, no secrets/PII.
Limits / owed
- No rollout performed here; spare-capacity numbers belong to the authorized window, not this doc. G09/G24/live statuses distinct.
Next microtask: SEC09-E final protected delivery.