docs operator recovery - AlexandreZanata/Abastevo GitHub Wiki
Restore procedure for the backend database with privacy guarantees (BUC-007/BUC-008, B-BR-016, ADR-008). Traffic stays closed until the deletion ledger replays: no restored erased data is served before replay.
- Off-host encrypted backup artifact with checksum manifest (backup
pipeline in P08-T03; until then, the documented
pg_dumpcustom format plus SHA-256 manifest). - Snapshot timestamp of the backup (when the snapshot started).
- Operator identity (
--operatororANPFUEL_OPERATOR_ID) and the access model from access.md. - A fresh isolated database (never restore over live production).
-
Stop API/worker traffic (disable writes at the edge; keep the database reachable for the operator only).
-
Restore the snapshot into the isolated database and verify row counts plus the migration ledger (
migratereports current; checksum mismatch fails the drill). -
Replay every deletion ledger row present in the restored snapshot:
go run ./cmd/ops privacy replay --contributor <id> --operator op-7
Repeat for each contributor with ledger rows (list pending rows with
SELECT contributor_id FROM privacy_deletion_ledger WHERE replayed_at IS NULL). Scopes with no matching rows report zero and still stamp: replays converge. -
Cross-check deletion receipts newer than the snapshot: any erasure completed after the snapshot started is absent from a stale backup. Re-run it explicitly:
go run ./cmd/ops privacy erase --contributor <id> --reason "post-restore re-issue (backup <timestamp>)" --operator op-7
A documented deletion request for backed-up persistent data may take until backup expiry to age out of immutable backups; notices must explain this accurately (SECURITY_PRIVACY).
-
Run retention expiry (P07-T05 sweeper) so transient rows older than their caps do not survive through the backup.
-
Rebuild projections that depend on unlinked rows (community consensus jobs drain through the worker; verify current-price reads before opening traffic).
-
Smoke the release endpoints, then reopen traffic.
- Ledger rows carry identifiers plus scope only: enough to find reappeared rows, never payload. The ledger itself ages out with the backup horizon under retention (35 days max: 7 daily + 4 weekly).
- Price facts stay for history with anonymized references (ADR-008); private payload, links, keys, sessions, objects, trust views and export bytes go.
- A failed replay blocks traffic: fix the scope error, re-run
privacy replay(convergent), and only then proceed. Never bypass with manual SQL edits. - Full encrypted off-host restore drills with measured RPO/RTO land in P08-T04; this runbook's deletion-replay path is proven by the P07-T04 restore simulation test.