Status: v2 (S9.T01 — P1/P2/P3 SLAs + PagerDuty + full war-room protocol) Canonical URL: .claude/docs/operations/incident-response-runbook.md Cross-ref: incident-response.md (SEV definitions + decision tree), bus-factor.md, postmortem-template.md, pagerduty-setup.md, runbooks/ (per-scenario)

nSelf uses four severity levels. P1/P2/P3 map to the most common operator response patterns.

• Auth broken for ANY user → P1 minimum.
• Payment/billing broken → P1 minimum.
• Any credential or secret exposed → P0 regardless of user count.
• If uncertain between P1 and P2: pick P1 and downgrade after triage.

nSelf is a single-maintainer ecosystem. Primary on-call is [email protected] (24/7). Backup on-call: per bus-factor.md (USER DECISION PENDING — nomination required for 9 critical services).

Via nself-incident-mgmt plugin UI (preferred when running):

1. Open https://your-nself-instance/incidents or localhost:3833/incidents
2. Click "Acknowledge" on the triggered alert
3. Fill in: IC name, initial hypothesis, war-room channel name
4. Set status to "Investigating"

Via Slack (fallback when UI unavailable):

Post to #incidents within ack SLA:

:ack: P{N} ack — {one-line summary} — IC: {your name} — {HH:MM UTC}
Opening war room: #incident-{YYYY-MM-DD}-{slug}

Via PagerDuty (if wired — see pagerduty-setup.md):

1. Accept the page from the PagerDuty mobile app or email
2. Post ack note in the PagerDuty incident: "Investigating — IC: {name}"
3. PagerDuty ack automatically updates nself-incident-mgmt via webhook

Every action during the incident gets a timestamped entry. Minimum data per entry:

{HH:MM UTC} | {who} | {what} | {outcome/finding}

Examples:

14:23 UTC | Aric | Checked Hasura logs — confirmed query timeout spike | 500 errors since 14:17
14:28 UTC | Aric | Restarted Hasura container | Errors clearing
14:35 UTC | Aric | Confirmed resolution — no new errors for 5 min | Mitigated

Update the timeline in nself-incident-mgmt UI or in the war-room Slack channel pinned message.

Initial post:

[Investigating] {service} — {one-line symptom} — {start-time UTC}

We are investigating reports of {symptom}. Affected users may see {impact}.

Next update: {now + 15 min}.

Identified:

[Identified] Root cause located — fix in progress.

We identified {cause}. Rollout is underway; impact should clear by {ETA UTC}.

Next update: {now + 15 min}.

Resolved:

[Resolved] Service restored at {time UTC}. Duration: {N} min.

{service} is fully restored. Post-mortem within 48h at {link}.

Subject: nSelf service incident — {one-line summary}

Hi {name or "there"},

We had an issue affecting {service} from {start UTC} to {end UTC} ({N} min).

What happened: {plain-English explanation — no jargon}.
Impact on you: {specific impact if known, else "some requests may have failed"}.
What we did: {plain-English mitigation steps}.
What's next: post-mortem within 48h; we'll send it your way.

If you noticed anything we missed, reply here.

— Aric, nSelf

(Tone rule: human, no "we apologize for the inconvenience" — see GCI Outbound Human Correspondence.)

nSelf uses PagerDuty in stub mode — routing is configured but single-escalation for now.

Alert triggered (nself-alert-router → PagerDuty)
  └─ Tier 1: Primary on-call (push notification + SMS) — 5 min
       └─ No ack → Tier 2: Backup on-call (see bus-factor.md) — +5 min
            └─ No ack → Escalation policy: email + phone — +10 min

See pagerduty-setup.md for integration key setup and test flow.

See War Room Protocol (Section 6).

Run in this order:

# Check overall health
nself health

# Check specific service
nself service status <postgres|hasura|auth|nginx|redis>

# Check recent logs (last 100 lines)
nself logs <service> --tail 100

# Check error rate spike
nself monitor --last 15m

• Which nself.org subdomains affected? (curl -I https://{subdomain}.nself.org/health)
• Which plugins returning errors? (Grafana → Plugin Error Rate panel)
• Cloud customers affected? (check np_cloud_tenants activity via Hasura)
• License validation broken? (curl https://ping.nself.org/health)

# What deployed in the last 2 hours?
cd /Volumes/X9/Sites/nself/cli && git log --oneline --since="2 hours ago"

# Any nself update recently?
nself version && nself update --dry-run

Navigate to the matching scenario, then follow it to resolution:

Rollback a bad deploy:

# Vercel — rollback to prior production deployment
vercel rollback --prod

# CLI fix pushed to prod — roll back to prior tag
cd /Volumes/X9/Sites/nself/cli
git tag --sort=-creatordate | head -5  # find prior tag
# then trigger deploy via nself deploy with prior version

Restart a crashed service:

nself service restart <service-name>

Enable graceful degradation (disable a bad plugin):

nself plugin disable <plugin-name>

Route around a failing dependency: See specific runbooks for Hetzner / Cloudflare / Vercel failovers.

• P0: required, within 48 hours.
• P1: required, within 48 hours.
• P2: optional but recommended for recurring issues.
• P3/P4: skip.

Use postmortem-template.md verbatim. Key sections:

## Post-Mortem: {one-line title}

**Date:** {YYYY-MM-DD}
**Severity:** P{N}
**Duration:** {start UTC} → {resolved UTC} ({N} min)
**IC:** {name}
**Authors:** {names}

### Impact
{Quantified user impact: N users affected, N min downtime, N failed requests.}

### Timeline
{Timestamped list — detection through resolution.}

### Root Cause
{Specific, factual. Use 5-Whys: keep asking "why" until you reach a process or system gap.}

### What Went Well
{Be honest — what actually helped?}

### What Went Wrong
{Be honest — what slowed response or caused the gap?}

### Action Items

| Action | Owner | Due | Status |
|---|---|---|---|
| {specific preventive or detective fix} | {name} | {YYYY-MM-DD} | open |

• Blameless means no "X did the wrong thing." Focus on system and process gaps.
• Every action item gets an owner and due date. Track in .claude/tasks/active.md until closed.
• Action items are always one specific, verifiable fix — never vague ("improve monitoring").
• Publish within 48 hours. Send link to affected customers by email.

• P0: always.
• P1: always.
• P2: at IC discretion.

1. Create the Slack channel:

#incident-{YYYY-MM-DD}-{slug}   e.g.  #incident-2026-05-07-auth-down

Channel topic:

P{N} | {one-line summary} | IC: {name} | Zoom: {url} | started {HH:MM UTC}

2. Assign roles:

For single-maintainer: IC = Technical Lead. Scribe and Comms can be the same person or deferred until mitigation is underway.

3. Open Zoom bridge:

Use the standing bridge URL stored in vault:

source ~/.claude/vault.env && echo $INCIDENT_ZOOM_URL

Paste the URL in the channel topic immediately. For P2: text-only in Slack is fine.

4. Pin the root-cause template:

Paste into the channel as a pinned message:

## Incident {YYYY-MM-DD} — {slug}

**P-level:** P{N}
**IC:** {name}
**Started:** {HH:MM UTC}
**Mitigated:** pending
**Resolved:** pending

### Symptoms
- {bullet each user-visible signal with timestamp}

### Affected surfaces
- [ ] {service / endpoint / customer segment}

### Hypotheses (live — update as ruled out or confirmed)
- H1: {hypothesis} — Evidence: {link/paste} — Status: open
- H2: …

### Actions taken
- {HH:MM} {action} → {outcome}

Update format for internal:

[HH:MM UTC] Update #{N}: {one-line current state}. Next: {what we're doing now}.

The IC declares resolution when:

1. The user-visible symptom is confirmed gone (not just "looks better").
2. Root cause is identified and either fixed or safely mitigated.
3. No new errors in the past 10 min (P0) or 5 min (P1).

Say explicitly in the channel:

:white_check_mark: RESOLVED — P{N} resolved at {HH:MM UTC}. Duration: {N} min.
Post-mortem due: {YYYY-MM-DD HH:MM UTC}.

Then: post status page resolution, send customer email if required, and file the post-mortem task in .claude/tasks/active.md.

• nself-incident-mgmt UI — incident ack + timeline
• status.nself.org — public status page
• PagerDuty setup — alert routing + integration key
• postmortem-template.md — blameless format
• on-call-rotation.md — who is primary on-call
• vendor-contacts.md — Cloudflare / Hetzner / Vercel / Stripe escalation
• bus-factor.md — backup admin assignments
• dr-runbook.md — full disaster recovery