Bite 33.6 — Recovery Rehearsal and Operational Runbook - RodrigoMattosoSilveira/EnterpriseRemoteSystems GitHub Wiki

Bite 33.6 — Recovery Rehearsal and Operational Runbook

Manual Verification Checklist Before Promotion and Operational Activation

Branch: 945-bite-336-recovery-rehearsal-and-operational-runbook
Validated HEAD SHA: 7c7c5cc66558b7d8a5c747ceb4bd6c01a604108a

1. Purpose

This suite validates Bite 33.6, including the owner-unavailable recovery hardening.

The operational requirement is:

A qualified designated technician, with the primary system owner completely unavailable, must be able to obtain every recovery prerequisite from documented shared administrative sources, recover an exact protected ERS database restore point, identify and prove the compatible application Git revision/tree, and prove the recovered application can start successfully without information or approval available only from the owner's memory, computer, phone, or personal password vault.

The suite has two phases.

Phase A — Pre-promotion validation

Phase A is performed against an isolated Development deployment and validates all recovery mechanics without touching normal Development, Test, or Production.

Use:

Server root:       /tmp/ers-336-manual-server
Git worktree:      /tmp/ers-336-manual-server/development
Compose project:   ers336-7c7c5cc6
Container prefix:  ers336-7c7c5cc6
S3 prefix:         manual-336-7c7c5cc6-<UTC timestamp>

Phase A must pass before promotion.

Phase B — Post-promotion owner-unavailable activation

The strengthened owner-unavailable proof requires a new Production backup created by this Bite 33.6 code, because older backups do not contain:

"repository": {
  "revision": "<compatible Git revision>",
  "tree_sha": "<compatible Git tree>"
}

Therefore the true clean-host owner-unavailable rehearsal is performed after this code has been promoted to Production and a fresh verified Production backup has been created.

Phase B is performed by a designated technician other than the primary system owner. The primary owner must not provide a password, file, key, hostname, MFA approval, backup filename, account identity, or recovery decision during that rehearsal.

Promotion may occur after Phase A passes so that the new backup format can be created in Production, but Bite 33.6 must not be considered operationally complete/closed until Phase B passes.

2. Global Safety Rules

Do not use or delete the normal local deployment:

ers-dev
ers-dev-backend
ers-dev-frontend
ers-dev-caddy

Do not perform Phase A against:

ers-tst
ers-prd
/opt/EnterpriseRemoteSystems/test
/opt/EnterpriseRemoteSystems/production

Only these Phase A Docker resources belong to this suite:

ers336-7c7c5cc6-backend
ers336-7c7c5cc6-frontend
ers336-7c7c5cc6-caddy

ers336-7c7c5cc6_backend-data
ers336-7c7c5cc6_caddy-data
ers336-7c7c5cc6_caddy-config

The Hetzner bucket has Object Lock / Governance retention. Do not bypass retention merely to remove manual-test objects.

Never copy secret values into:

Git
manual test notes
rehearsal.json
screenshots
terminal transcripts
issue comments
evidence summary files

The committed recovery access manifest contains locators, never secret values.

3. Phase A — Test 00 — Confirm Exact Branch, SHA, Tools, and Docker

Identity / context

  • Operator: primary developer is acceptable for Phase A.
  • ERS application login: none.
  • Working directory: primary ERS repository root.

Procedure

Run:

git branch --show-current
git rev-parse HEAD

Expected exactly:

945-bite-336-recovery-rehearsal-and-operational-runbook
7c7c5cc66558b7d8a5c747ceb4bd6c01a604108a

Record:

export ERS336_HEAD="$(git rev-parse HEAD)"
export ERS336_TREE="$(git rev-parse HEAD^{tree})"

Confirm required local commands:

for command in \
  git \
  docker \
  sqlite3 \
  aws \
  jq \
  shasum \
  python3 \
  make
do
  command -v "$command" >/dev/null 2>&1 \
    && echo "PASS: $command is available" \
    || echo "FAIL: $command is missing"
done

Expected: one PASS: line for every command and no FAIL: line.

Confirm Docker:

docker info >/dev/null 2>&1 \
  && echo "PASS: Docker daemon is reachable" \
  || echo "FAIL: Docker daemon is not reachable"

Expected:

PASS: Docker daemon is reachable

PASS criteria

PASS only if branch/SHA are exact, all tools are present, and Docker is reachable.

Promotion blocker

Stop if branch or SHA differs.

4. Phase A — Test 01 — Verify the Recovery Access Contract Is Explicit

Identity / context

No ERS application login required.

Procedure

Open:

cat ops/recovery-access.production.env

Confirm these exact non-secret values are present:

RECOVERY_ACCESS_FORMAT_VERSION=1
RECOVERY_OWNER_UNAVAILABLE_REQUIRED=true
RECOVERY_SHARED_SECRETS_PROVIDER=LastPass
RECOVERY_SHARED_COLLECTION=ERS-Recovery
RECOVERY_REPOSITORY_SLUG=RodrigoMattosoSilveira/EnterpriseRemoteSystems
[email protected]:RodrigoMattosoSilveira/EnterpriseRemoteSystems.git
RECOVERY_REPOSITORY_SOURCE=github://RodrigoMattosoSilveira/EnterpriseRemoteSystems
RECOVERY_PRODUCTION_ENV_SOURCE=lastpass-shared://ERS-Recovery/Production/.env.production
RECOVERY_S3_CREDENTIAL_SOURCE=lastpass-shared://ERS-Recovery/Production/Hetzner-S3-ers-backup
RECOVERY_HOST_ACCESS_SOURCE=lastpass-shared://ERS-Recovery/Production/Host-SSH
RECOVERY_HETZNER_CONSOLE_SOURCE=lastpass-shared://ERS-Recovery/Infrastructure/Hetzner-Console
RECOVERY_DNS_SOURCE=lastpass-shared://ERS-Recovery/Infrastructure/DNS
RECOVERY_MFA_RECOVERY_SOURCE=technician-controlled://independent-mfa-and-recovery-codes
RECOVERY_REPOSITORY_INSTALL_PATH=/opt/EnterpriseRemoteSystems/production
RECOVERY_PRODUCTION_ENV_INSTALL_PATH=/opt/EnterpriseRemoteSystems/production/.env.production
RECOVERY_S3_CREDENTIAL_INSTALL_PATH=/opt/EnterpriseRemoteSystems/secrets/aws-credentials
RECOVERY_PRODUCTION_DOMAIN=app.enterpriseremotesystems.com
RECOVERY_DNS_DOMAIN=enterpriseremotesystems.com
RECOVERY_S3_ENDPOINT=https://hel1.your-objectstorage.com
RECOVERY_S3_REGION=hel1
RECOVERY_S3_BUCKET=ers
RECOVERY_S3_PROFILE=ers-backup
RECOVERY_SHARED_ACCESS_POLICY=technician-access-independent-of-owner
RECOVERY_MFA_POLICY=must-not-depend-on-owner-device-or-approval

Run:

if grep -Eiq \
  'normal administrative|normal-administrative|personal-vault|TBD|TODO|<[^>]+>' \
  ops/recovery-access.production.env
then
  echo "FAIL: recovery access manifest still contains a vague/unconfigured source"
else
  echo "PASS: recovery access manifest contains explicit non-personal locators"
fi

Expected:

PASS: recovery access manifest contains explicit non-personal locators

Check that obvious secret-value names are not assigned real secret material in the locator manifest:

if grep -Eiq \
  'aws_secret_access_key|JWT_SECRET=|LEDGER_CORRECTION_KEY=|LEDGER_SETTLEMENT_KEY=|OPENAI_API_KEY=' \
  ops/recovery-access.production.env
then
  echo "FAIL: recovery access manifest appears to contain secret material"
else
  echo "PASS: recovery access manifest contains locators only"
fi

Expected:

PASS: recovery access manifest contains locators only

PASS criteria

All recovery sources are explicit and non-personal; no secret values are stored in the manifest.

5. Phase A — Test 02 — Verify Shared Administrative Access Exists Before an Incident

Identity / context

This test must be performed by at least one designated recovery technician other than the primary owner.

The owner may observe this setup test, but the technician must use their own accounts.

Procedure

The designated technician must sign in to their own LastPass account and confirm they can open the shared collection:

ERS-Recovery

The technician must confirm these exact shared items exist and are accessible:

ERS-Recovery / Production / .env.production
ERS-Recovery / Production / Hetzner-S3-ers-backup
ERS-Recovery / Production / Host-SSH
ERS-Recovery / Infrastructure / Hetzner-Console
ERS-Recovery / Infrastructure / DNS

Do not copy the secret contents into this test record.

Record only:

Production/.env.production accessible: PASS / FAIL
Production/Hetzner-S3-ers-backup accessible: PASS / FAIL
Production/Host-SSH accessible: PASS / FAIL
Infrastructure/Hetzner-Console accessible: PASS / FAIL
Infrastructure/DNS accessible: PASS / FAIL

The same technician must confirm they have their own independent MFA/recovery capability for their individual LastPass/GitHub/provider accounts.

Record only:

Independent MFA/recovery capability: PASS / FAIL

The technician must confirm that unlocking LastPass, GitHub, Hetzner, and DNS does not require:

the owner's phone
the owner's MFA approval
the owner's password
the owner's personal vault
a file stored only on the owner's computer

PASS criteria

PASS only if the designated technician independently has all shared access before an incident.

Promotion blocker

If any required item is owner-only, personal-vault-only, inaccessible to the designated technician, or depends on owner MFA approval, do not promote.

6. Phase A — Test 03 — Verify the Persistent Hetzner S3 Profile on the Local Test Machine

Identity / context

This is the Phase A local technical credential only.

Procedure

Confirm:

if [[ -f "$HOME/.aws/credentials" ]] \
  && grep -q '^\[ers-backup\]$' "$HOME/.aws/credentials"
then
  echo "PASS: persistent ers-backup profile exists"
else
  echo "FAIL: persistent ers-backup profile is missing"
fi

Expected:

PASS: persistent ers-backup profile exists

Set:

export ERS336_AWS_CREDENTIALS_FILE="$HOME/.aws/credentials"

Verify Hetzner access:

AWS_SHARED_CREDENTIALS_FILE="$ERS336_AWS_CREDENTIALS_FILE" \
AWS_PROFILE=ers-backup \
AWS_REGION=hel1 \
aws s3api head-bucket \
  --bucket ers \
  --endpoint-url https://hel1.your-objectstorage.com

Then:

echo $?

Expected:

0

7. Phase A — Test 04 — Prove the Disposable Docker Namespace Is Safe

Procedure

Define:

export ERS336_COMPOSE_PROJECT="ers336-7c7c5cc6"
export ERS336_CONTAINER_PREFIX="ers336-7c7c5cc6"

Display normal Development resources for information only:

docker ps -a --format '{{.Names}}' \
  | grep -E '^ers-dev-(backend|frontend|caddy)$' \
  || true

Existing ers-dev-* resources are acceptable and must remain untouched.

Check only for stale Bite 33.6 containers:

ERS336_STALE_CONTAINERS="$(
  docker ps -a --format '{{.Names}}' \
    | grep -E "^${ERS336_CONTAINER_PREFIX}-(backend|frontend|caddy)$" \
    || true
)"

if [[ -z "$ERS336_STALE_CONTAINERS" ]]; then
  echo "PASS: no stale Bite 33.6 manual-test containers exist"
else
  echo "STALE: Bite 33.6 manual-test containers exist:"
  printf '%s\n' "$ERS336_STALE_CONTAINERS"
fi

Check dedicated volumes:

ERS336_STALE_VOLUMES="$(
  docker volume ls --format '{{.Name}}' \
    | grep "^${ERS336_COMPOSE_PROJECT}_" \
    || true
)"

if [[ -z "$ERS336_STALE_VOLUMES" ]]; then
  echo "PASS: no stale Bite 33.6 manual-test volumes exist"
else
  echo "STALE: Bite 33.6 manual-test volumes exist:"
  printf '%s\n' "$ERS336_STALE_VOLUMES"
fi

If stale resources exist, remove only the Bite 33.6 resources:

docker rm -f \
  "${ERS336_CONTAINER_PREFIX}-backend" \
  "${ERS336_CONTAINER_PREFIX}-frontend" \
  "${ERS336_CONTAINER_PREFIX}-caddy" \
  2>/dev/null || true

for volume in \
  "${ERS336_COMPOSE_PROJECT}_backend-data" \
  "${ERS336_COMPOSE_PROJECT}_caddy-data" \
  "${ERS336_COMPOSE_PROJECT}_caddy-config"
do
  docker volume rm "$volume" 2>/dev/null || true
done

Re-run the two checks.

PASS criteria

Final output contains:

PASS: no stale Bite 33.6 manual-test containers exist
PASS: no stale Bite 33.6 manual-test volumes exist

8. Phase A — Test 05 — Create the Exact Git Worktree and Isolated Environment

Procedure

From the primary repository root:

git worktree prune
rm -rf /tmp/ers-336-manual-server
mkdir -p /tmp/ers-336-manual-server

Canonicalize:

export ERS336_SERVER_ROOT="$(
  cd /tmp/ers-336-manual-server && pwd -P
)"

export ERS336_ENV_DIR="$ERS336_SERVER_ROOT/development"
export ERS336_COMPOSE_PROJECT="ers336-7c7c5cc6"
export ERS336_CONTAINER_PREFIX="ers336-7c7c5cc6"
export ERS336_S3_PREFIX="manual-336-7c7c5cc6-$(date -u +%Y%m%dT%H%M%SZ)"
export ERS336_MIGRATION_STATE_FILE="$ERS336_ENV_DIR/recovery/migration-gates/current.state"

Create the detached worktree:

git worktree add \
  --detach \
  "$ERS336_ENV_DIR" \
  "$ERS336_HEAD"

Verify:

test "$(git -C "$ERS336_ENV_DIR" rev-parse HEAD)" = "$ERS336_HEAD" \
  && echo "PASS: disposable worktree HEAD matches validated SHA"

test "$(git -C "$ERS336_ENV_DIR" rev-parse HEAD^{tree})" = "$ERS336_TREE" \
  && echo "PASS: disposable worktree tree SHA matches validated tree"

Expected:

PASS: disposable worktree HEAD matches validated SHA
PASS: disposable worktree tree SHA matches validated tree

Initialize Development environment:

(
  cd "$ERS336_ENV_DIR"
  ./scripts/init-server-env.sh development
)

Define:

set_ers336_env_value() {
  local file="$1"
  local key="$2"
  local value="$3"
  local tmp="${file}.tmp"

  grep -v "^${key}=" "$file" > "$tmp" || true
  printf '%s=%s\n' "$key" "$value" >> "$tmp"
  mv "$tmp" "$file"
}

Set isolated container prefix:

set_ers336_env_value \
  "$ERS336_ENV_DIR/.env.development" \
  CONTAINER_PREFIX \
  "$ERS336_CONTAINER_PREFIX"

Ensure startup does not auto-migrate:

set_ers336_env_value \
  "$ERS336_ENV_DIR/.env.development" \
  APP_AUTO_MIGRATE \
  false

Enable S3 under a unique Development prefix:

set_ers336_env_value \
  "$ERS336_ENV_DIR/.env.development" \
  SERVER_OFFHOST_BACKUP_ENABLED \
  true

set_ers336_env_value \
  "$ERS336_ENV_DIR/.env.development" \
  SERVER_OFFHOST_BACKUP_TRANSPORT \
  s3

set_ers336_env_value \
  "$ERS336_ENV_DIR/.env.development" \
  SERVER_OFFHOST_BACKUP_S3_ENDPOINT \
  https://hel1.your-objectstorage.com

set_ers336_env_value \
  "$ERS336_ENV_DIR/.env.development" \
  SERVER_OFFHOST_BACKUP_S3_REGION \
  hel1

set_ers336_env_value \
  "$ERS336_ENV_DIR/.env.development" \
  SERVER_OFFHOST_BACKUP_S3_BUCKET \
  ers

set_ers336_env_value \
  "$ERS336_ENV_DIR/.env.development" \
  SERVER_OFFHOST_BACKUP_S3_PREFIX \
  "$ERS336_S3_PREFIX"

set_ers336_env_value \
  "$ERS336_ENV_DIR/.env.development" \
  AWS_SHARED_CREDENTIALS_FILE \
  "$ERS336_AWS_CREDENTIALS_FILE"

set_ers336_env_value \
  "$ERS336_ENV_DIR/.env.development" \
  AWS_PROFILE \
  ers-backup

Validate:

make \
  SERVER_ROOT="$ERS336_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS336_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS336_CONTAINER_PREFIX" \
  server-environment-contract-check \
  ENV=development

Expected exit status:

echo $?
0

Persist state:

cat > /tmp/ers-336-manual-session.env <<EOF
export ERS336_HEAD='$ERS336_HEAD'
export ERS336_TREE='$ERS336_TREE'
export ERS336_SERVER_ROOT='$ERS336_SERVER_ROOT'
export ERS336_ENV_DIR='$ERS336_ENV_DIR'
export ERS336_COMPOSE_PROJECT='$ERS336_COMPOSE_PROJECT'
export ERS336_CONTAINER_PREFIX='$ERS336_CONTAINER_PREFIX'
export ERS336_S3_PREFIX='$ERS336_S3_PREFIX'
export ERS336_AWS_CREDENTIALS_FILE='$ERS336_AWS_CREDENTIALS_FILE'
export ERS336_MIGRATION_STATE_FILE='$ERS336_MIGRATION_STATE_FILE'
EOF

9. Phase A — Test 06 — Build and Start the Isolated Deployment

Reload:

source /tmp/ers-336-manual-session.env

Build:

make \
  SERVER_ROOT="$ERS336_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS336_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS336_CONTAINER_PREFIX" \
  server-build \
  ENV=development

Create fresh Development DB:

make \
  SERVER_ROOT="$ERS336_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS336_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS336_CONTAINER_PREFIX" \
  server-replace-development-db \
  ENV=development

Expected output includes:

Fresh Development database is ready.
Database: /app/data/app.db
Integrity check: ok
Foreign key check: clean

Start:

make \
  SERVER_ROOT="$ERS336_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS336_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS336_CONTAINER_PREFIX" \
  server-up \
  ENV=development

Verify:

make \
  SERVER_ROOT="$ERS336_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS336_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS336_CONTAINER_PREFIX" \
  server-backend-health \
  ENV=development

Expected:

ers336-7c7c5cc6-backend health check passed.

Integrity:

docker exec "${ERS336_CONTAINER_PREFIX}-backend" \
  sqlite3 /app/data/app.db 'PRAGMA integrity_check;'

Expected:

ok

Foreign keys:

docker exec "${ERS336_CONTAINER_PREFIX}-backend" \
  sqlite3 /app/data/app.db 'PRAGMA foreign_key_check;'

Expected: no output.

10. Phase A — Test 07 — Create a New Backup with Compatible Git Revision/Tree Evidence

Purpose

This validates the new 33.6 backup contract that makes owner-unavailable recovery deterministic.

Procedure

Create a source marker:

docker exec "${ERS336_CONTAINER_PREFIX}-backend" \
  sqlite3 /app/data/app.db "
    CREATE TABLE bite336_recovery_source_probe (
      id TEXT PRIMARY KEY,
      value TEXT NOT NULL
    );
    INSERT INTO bite336_recovery_source_probe(id, value)
    VALUES ('source-1', 'present-in-selected-backup');
  "

Create verified/off-host backup:

make \
  SERVER_ROOT="$ERS336_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS336_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS336_CONTAINER_PREFIX" \
  server-backup \
  ENV=development

Locate it:

export ERS336_SOURCE_BACKUP="$(
  find "$ERS336_ENV_DIR/backups" \
    -maxdepth 1 \
    -type f \
    -name 'app-*.db' \
    -print \
  | sort \
  | tail -1
)"

export ERS336_BACKUP_NAME="$(basename "$ERS336_SOURCE_BACKUP")"
export ERS336_SOURCE_MANIFEST="$ERS336_SOURCE_BACKUP.manifest.json"
export ERS336_SOURCE_RECEIPT="$ERS336_SOURCE_BACKUP.offhost.json"
export ERS336_SOURCE_SHA256="$(
  shasum -a 256 "$ERS336_SOURCE_BACKUP" | awk '{print $1}'
)"

Confirm all three exist:

test -s "$ERS336_SOURCE_BACKUP" \
  && test -s "$ERS336_SOURCE_MANIFEST" \
  && test -s "$ERS336_SOURCE_RECEIPT" \
  && echo "PASS: verified local + manifest + off-host receipt exist"

Expected:

PASS: verified local + manifest + off-host receipt exist

Read new repository evidence:

export ERS336_BACKUP_REVISION="$(
  jq -er '.repository.revision' "$ERS336_SOURCE_MANIFEST"
)"

export ERS336_BACKUP_TREE="$(
  jq -er '.repository.tree_sha' "$ERS336_SOURCE_MANIFEST"
)"

Print:

printf 'Backup compatible revision: %s\n' "$ERS336_BACKUP_REVISION"
printf 'Backup compatible tree:     %s\n' "$ERS336_BACKUP_TREE"

Assert both are full Git object IDs:

if [[ "$ERS336_BACKUP_REVISION" =~ ^[0-9a-f]{40}$ \
      && "$ERS336_BACKUP_TREE" =~ ^[0-9a-f]{40}$ ]]
then
  echo "PASS: backup contains full compatible revision/tree evidence"
else
  echo "FAIL: backup compatible revision/tree evidence is invalid"
fi

Expected:

PASS: backup contains full compatible revision/tree evidence

Verify they match the code that created this backup:

if [[ "$ERS336_BACKUP_REVISION" == "$ERS336_HEAD" \
      && "$ERS336_BACKUP_TREE" == "$ERS336_TREE" ]]
then
  echo "PASS: backup revision/tree identify the exact running test revision"
else
  echo "FAIL: backup revision/tree do not match the expected code"
fi

Expected:

PASS: backup revision/tree identify the exact running test revision

Independently prove the relationship:

git -C "$ERS336_ENV_DIR" \
  cat-file -e "${ERS336_BACKUP_REVISION}^{commit}" \
  && echo "PASS: compatible backup revision exists in Git"

Then:

if [[ "$(
  git -C "$ERS336_ENV_DIR" rev-parse "${ERS336_BACKUP_REVISION}^{tree}"
)" == "$ERS336_BACKUP_TREE" ]]
then
  echo "PASS: compatible backup tree belongs to the recorded revision"
else
  echo "FAIL: recorded backup tree does not belong to the recorded revision"
fi

Expected: both PASS.

Record exact S3 evidence:

export ERS336_DB_KEY="$(
  jq -r '.remote.backup.key' "$ERS336_SOURCE_RECEIPT"
)"

export ERS336_DB_VERSION_ID="$(
  jq -r '.remote.backup.version_id' "$ERS336_SOURCE_RECEIPT"
)"

export ERS336_MANIFEST_KEY="$(
  jq -r '.remote.manifest.key' "$ERS336_SOURCE_RECEIPT"
)"

export ERS336_MANIFEST_VERSION_ID="$(
  jq -r '.remote.manifest.version_id' "$ERS336_SOURCE_RECEIPT"
)"

Verify exact remote DB version:

AWS_SHARED_CREDENTIALS_FILE="$ERS336_AWS_CREDENTIALS_FILE" \
AWS_PROFILE=ers-backup \
AWS_REGION=hel1 \
aws s3api head-object \
  --bucket ers \
  --key "$ERS336_DB_KEY" \
  --version-id "$ERS336_DB_VERSION_ID" \
  --endpoint-url https://hel1.your-objectstorage.com \
  >/dev/null \
  && echo "PASS: exact database VersionId exists in Hetzner"

Verify exact manifest version:

AWS_SHARED_CREDENTIALS_FILE="$ERS336_AWS_CREDENTIALS_FILE" \
AWS_PROFILE=ers-backup \
AWS_REGION=hel1 \
aws s3api head-object \
  --bucket ers \
  --key "$ERS336_MANIFEST_KEY" \
  --version-id "$ERS336_MANIFEST_VERSION_ID" \
  --endpoint-url https://hel1.your-objectstorage.com \
  >/dev/null \
  && echo "PASS: exact manifest VersionId exists in Hetzner"

Create live state after the backup:

docker exec "${ERS336_CONTAINER_PREFIX}-backend" \
  sqlite3 /app/data/app.db "
    CREATE TABLE bite336_live_after_backup_probe (
      id TEXT PRIMARY KEY,
      value TEXT NOT NULL
    );
    INSERT INTO bite336_live_after_backup_probe(id, value)
    VALUES ('live-1', 'must-remain-in-live-database');
  "

Verify:

docker exec "${ERS336_CONTAINER_PREFIX}-backend" \
  sqlite3 /app/data/app.db \
  "SELECT id || ':' || value FROM bite336_live_after_backup_probe;"

Expected:

live-1:must-remain-in-live-database

Persist:

{
  printf "export ERS336_SOURCE_BACKUP='%s'\n" "$ERS336_SOURCE_BACKUP"
  printf "export ERS336_BACKUP_NAME='%s'\n" "$ERS336_BACKUP_NAME"
  printf "export ERS336_SOURCE_MANIFEST='%s'\n" "$ERS336_SOURCE_MANIFEST"
  printf "export ERS336_SOURCE_RECEIPT='%s'\n" "$ERS336_SOURCE_RECEIPT"
  printf "export ERS336_SOURCE_SHA256='%s'\n" "$ERS336_SOURCE_SHA256"
  printf "export ERS336_BACKUP_REVISION='%s'\n" "$ERS336_BACKUP_REVISION"
  printf "export ERS336_BACKUP_TREE='%s'\n" "$ERS336_BACKUP_TREE"
  printf "export ERS336_DB_KEY='%s'\n" "$ERS336_DB_KEY"
  printf "export ERS336_DB_VERSION_ID='%s'\n" "$ERS336_DB_VERSION_ID"
  printf "export ERS336_MANIFEST_KEY='%s'\n" "$ERS336_MANIFEST_KEY"
  printf "export ERS336_MANIFEST_VERSION_ID='%s'\n" "$ERS336_MANIFEST_VERSION_ID"
} >> /tmp/ers-336-manual-session.env

11. Phase A — Test 08 — Local-Backup Recovery Rehearsal

Reload:

source /tmp/ers-336-manual-session.env

Define report:

export ERS336_LOCAL_REPORT="$ERS336_ENV_DIR/recovery/rehearsals/manual-local-rehearsal.json"
rm -f "$ERS336_LOCAL_REPORT"

Run:

make \
  SERVER_ROOT="$ERS336_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS336_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS336_CONTAINER_PREFIX" \
  server-recovery-rehearsal \
  ENV=development \
  RECOVERY_REHEARSAL_BACKUP_FILE="$ERS336_SOURCE_BACKUP" \
  RECOVERY_REHEARSAL_REPORT_FILE="$ERS336_LOCAL_REPORT"

Expected summary includes:

Recovery rehearsal passed.
Environment: development
Selected restore-point age / RPO:
Recovery rehearsal RTO:
Evidence:

Validate:

if jq -e \
  --arg revision "$ERS336_HEAD" \
  --arg tree "$ERS336_TREE" \
  --arg backup_revision "$ERS336_BACKUP_REVISION" \
  --arg backup_tree "$ERS336_BACKUP_TREE" \
  --arg sha "$ERS336_SOURCE_SHA256" \
  '
    .status == "passed"
    and .environment == "development"
    and .repository.revision == $revision
    and .repository.tree_sha == $tree
    and .source.kind == "local-backup"
    and .source.backup_sha256 == $sha
    and .source.repository_revision == $backup_revision
    and .source.repository_tree_sha == $backup_tree
    and .source.compatible_revision_available == true
    and .recovery.restored_sha256 == $sha
    and .recovery.isolated_backend_health == "passed"
    and .recovery.post_boot_integrity_check == "ok"
    and .recovery.post_boot_foreign_key_check == "clean"
    and .objectives.max_rpo_seconds == 86400
    and .objectives.max_rto_seconds == 14400
    and .objectives.rpo_met == true
    and .objectives.rto_met == true
  ' \
  "$ERS336_LOCAL_REPORT" \
  >/dev/null
then
  echo "PASS: local rehearsal evidence satisfies Bite 33.6"
else
  echo "FAIL: local rehearsal evidence is incomplete"
fi

Expected:

PASS: local rehearsal evidence satisfies Bite 33.6

Confirm live backend remains running:

docker ps --format '{{.Names}}' \
  | grep -qx "${ERS336_CONTAINER_PREFIX}-backend" \
  && echo "PASS: live backend remained running during rehearsal"

Verify post-backup live data survived:

docker exec "${ERS336_CONTAINER_PREFIX}-backend" \
  sqlite3 /app/data/app.db \
  "SELECT id || ':' || value FROM bite336_live_after_backup_probe;"

Expected:

live-1:must-remain-in-live-database

Persist:

printf "export ERS336_LOCAL_REPORT='%s'\n" \
  "$ERS336_LOCAL_REPORT" \
  >> /tmp/ers-336-manual-session.env

12. Phase A — Test 09 — Surviving-Receipt Exact-Version Rehearsal

Define:

source /tmp/ers-336-manual-session.env

export ERS336_RECEIPT_REPORT="$ERS336_ENV_DIR/recovery/rehearsals/manual-receipt-rehearsal.json"
rm -f "$ERS336_RECEIPT_REPORT"

Run:

make \
  SERVER_ROOT="$ERS336_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS336_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS336_CONTAINER_PREFIX" \
  server-recovery-rehearsal \
  ENV=development \
  RECOVERY_REHEARSAL_RECEIPT_FILE="$ERS336_SOURCE_RECEIPT" \
  RECOVERY_REHEARSAL_REPORT_FILE="$ERS336_RECEIPT_REPORT"

Validate exact versions:

if jq -e \
  --arg sha "$ERS336_SOURCE_SHA256" \
  --arg revision "$ERS336_BACKUP_REVISION" \
  --arg tree "$ERS336_BACKUP_TREE" \
  --arg db_key "$ERS336_DB_KEY" \
  --arg db_version "$ERS336_DB_VERSION_ID" \
  --arg manifest_key "$ERS336_MANIFEST_KEY" \
  --arg manifest_version "$ERS336_MANIFEST_VERSION_ID" \
  '
    .status == "passed"
    and .source.kind == "offhost-receipt"
    and .source.backup_sha256 == $sha
    and .source.repository_revision == $revision
    and .source.repository_tree_sha == $tree
    and .source.compatible_revision_available == true
    and .source.remote.backup_key == $db_key
    and .source.remote.backup_version_id == $db_version
    and .source.remote.manifest_key == $manifest_key
    and .source.remote.manifest_version_id == $manifest_version
    and .recovery.restored_sha256 == $sha
  ' \
  "$ERS336_RECEIPT_REPORT" \
  >/dev/null
then
  echo "PASS: receipt rehearsal used exact protected S3 versions and compatible code evidence"
else
  echo "FAIL: receipt rehearsal evidence differs from expected protected versions"
fi

Expected:

PASS: receipt rehearsal used exact protected S3 versions and compatible code evidence

Confirm live marker:

docker exec "${ERS336_CONTAINER_PREFIX}-backend" \
  sqlite3 /app/data/app.db \
  "SELECT id || ':' || value FROM bite336_live_after_backup_probe;"

Expected:

live-1:must-remain-in-live-database

Persist:

printf "export ERS336_RECEIPT_REPORT='%s'\n" \
  "$ERS336_RECEIPT_REPORT" \
  >> /tmp/ers-336-manual-session.env

13. Phase A — Test 10 — Host-Loss Rehearsal with No Local Backup, Manifest, or Receipt

Purpose

Prove the recovery can bootstrap from only the managed app-*.db name.

Procedure

Reload:

source /tmp/ers-336-manual-session.env

Hide all local recovery artifacts:

rm -rf /tmp/ers-336-host-loss-hidden
mkdir -p /tmp/ers-336-host-loss-hidden

mv "$ERS336_SOURCE_BACKUP" /tmp/ers-336-host-loss-hidden/
mv "$ERS336_SOURCE_MANIFEST" /tmp/ers-336-host-loss-hidden/
mv "$ERS336_SOURCE_RECEIPT" /tmp/ers-336-host-loss-hidden/

Assert unavailable:

if [[ ! -e "$ERS336_SOURCE_BACKUP" \
      && ! -e "$ERS336_SOURCE_MANIFEST" \
      && ! -e "$ERS336_SOURCE_RECEIPT" ]]
then
  echo "PASS: all local source artifacts are unavailable"
else
  echo "FAIL: a local source artifact remains available"
fi

Define report:

export ERS336_HOSTLOSS_REPORT="$ERS336_ENV_DIR/recovery/rehearsals/manual-host-loss-rehearsal.json"
rm -f "$ERS336_HOSTLOSS_REPORT"

Run while guaranteeing local test files are restored afterward:

set +e

make \
  SERVER_ROOT="$ERS336_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS336_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS336_CONTAINER_PREFIX" \
  server-recovery-rehearsal \
  ENV=development \
  RECOVERY_REHEARSAL_OFFHOST_BACKUP_NAME="$ERS336_BACKUP_NAME" \
  RECOVERY_REHEARSAL_REPORT_FILE="$ERS336_HOSTLOSS_REPORT"

ERS336_HOSTLOSS_STATUS=$?

mv \
  "/tmp/ers-336-host-loss-hidden/$ERS336_BACKUP_NAME" \
  "$ERS336_SOURCE_BACKUP"

mv \
  "/tmp/ers-336-host-loss-hidden/$ERS336_BACKUP_NAME.manifest.json" \
  "$ERS336_SOURCE_MANIFEST"

mv \
  "/tmp/ers-336-host-loss-hidden/$ERS336_BACKUP_NAME.offhost.json" \
  "$ERS336_SOURCE_RECEIPT"

set -e

Assert:

if [[ "$ERS336_HOSTLOSS_STATUS" -eq 0 ]]; then
  echo "PASS: host-loss rehearsal succeeded using only managed backup name"
else
  echo "FAIL: host-loss rehearsal returned $ERS336_HOSTLOSS_STATUS"
fi

Expected:

PASS: host-loss rehearsal succeeded using only managed backup name

Validate:

if jq -e \
  --arg sha "$ERS336_SOURCE_SHA256" \
  --arg revision "$ERS336_BACKUP_REVISION" \
  --arg tree "$ERS336_BACKUP_TREE" \
  --arg db_version "$ERS336_DB_VERSION_ID" \
  --arg manifest_version "$ERS336_MANIFEST_VERSION_ID" \
  '
    .status == "passed"
    and .source.kind == "offhost-backup-name"
    and (.source.receipt_version_id | length) > 0
    and .source.backup_sha256 == $sha
    and .source.repository_revision == $revision
    and .source.repository_tree_sha == $tree
    and .source.compatible_revision_available == true
    and .source.remote.backup_version_id == $db_version
    and .source.remote.manifest_version_id == $manifest_version
    and .recovery.restored_sha256 == $sha
  ' \
  "$ERS336_HOSTLOSS_REPORT" \
  >/dev/null
then
  echo "PASS: host-loss report proves receipt bootstrap, exact versions, and compatible code"
else
  echo "FAIL: host-loss evidence is incomplete"
fi

Expected:

PASS: host-loss report proves receipt bootstrap, exact versions, and compatible code

Verify live marker survived:

docker exec "${ERS336_CONTAINER_PREFIX}-backend" \
  sqlite3 /app/data/app.db \
  "SELECT id || ':' || value FROM bite336_live_after_backup_probe;"

Expected:

live-1:must-remain-in-live-database

Persist:

printf "export ERS336_HOSTLOSS_REPORT='%s'\n" \
  "$ERS336_HOSTLOSS_REPORT" \
  >> /tmp/ers-336-manual-session.env

14. Phase A — Test 11 — Safety Guards

Test 11A — Multiple recovery sources must be refused

source /tmp/ers-336-manual-session.env

set +e

make \
  SERVER_ROOT="$ERS336_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS336_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS336_CONTAINER_PREFIX" \
  server-recovery-rehearsal \
  ENV=development \
  RECOVERY_REHEARSAL_BACKUP_FILE="$ERS336_SOURCE_BACKUP" \
  RECOVERY_REHEARSAL_RECEIPT_FILE="$ERS336_SOURCE_RECEIPT" \
  > /tmp/ers-336-multiple-sources.out 2>&1

ERS336_MULTI_STATUS=$?

set -e

cat /tmp/ers-336-multiple-sources.out

Expected output contains:

Exactly one recovery rehearsal source is required:

Assert:

[[ "$ERS336_MULTI_STATUS" -ne 0 ]] \
  && echo "PASS: multiple source definitions refused"

Test 11B — Owner-unavailable target must refuse Development

set +e

make \
  SERVER_ROOT="$ERS336_SERVER_ROOT" \
  server-owner-unavailable-recovery-rehearsal \
  ENV=development \
  RECOVERY_REHEARSAL_OFFHOST_BACKUP_NAME="$ERS336_BACKUP_NAME" \
  RECOVERY_TECHNICIAN_ID="phase-a-test" \
  RECOVERY_SHARED_ACCESS_CONFIRM=OWNER-UNAVAILABLE-RECOVERY-ACCESS-CONFIRMED \
  > /tmp/ers-336-owner-mode-dev.out 2>&1

ERS336_OWNER_DEV_STATUS=$?

set -e

cat /tmp/ers-336-owner-mode-dev.out

Expected includes:

Owner-unavailable recovery rehearsal must use ENV=production.

Assert:

[[ "$ERS336_OWNER_DEV_STATUS" -ne 0 ]] \
  && echo "PASS: owner-unavailable mode refuses non-Production environment"

Test 11C — Owner-unavailable mode requires technician identity and explicit shared-access confirmation

Run without technician:

set +e

make \
  server-owner-unavailable-recovery-rehearsal \
  ENV=production \
  RECOVERY_REHEARSAL_OFFHOST_BACKUP_NAME="$ERS336_BACKUP_NAME" \
  RECOVERY_SHARED_ACCESS_CONFIRM=OWNER-UNAVAILABLE-RECOVERY-ACCESS-CONFIRMED \
  > /tmp/ers-336-owner-no-tech.out 2>&1

ERS336_NO_TECH_STATUS=$?

set -e

Expected output includes:

RECOVERY_TECHNICIAN_ID is required for owner-unavailable recovery evidence.

This command must fail before any recovery operation.

15. Phase A — Test 12 — Deliberately Miss RPO While Recovery Mechanics Still Pass

Ensure backup age exceeds one second:

sleep 2

Define:

export ERS336_OBJECTIVE_REPORT="$ERS336_ENV_DIR/recovery/rehearsals/manual-objective-failure.json"
rm -f "$ERS336_OBJECTIVE_REPORT"

Run:

set +e

make \
  SERVER_ROOT="$ERS336_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS336_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS336_CONTAINER_PREFIX" \
  server-recovery-rehearsal \
  ENV=development \
  RECOVERY_REHEARSAL_BACKUP_FILE="$ERS336_SOURCE_BACKUP" \
  RECOVERY_REHEARSAL_MAX_RPO_SECONDS=1 \
  RECOVERY_REHEARSAL_MAX_RTO_SECONDS=14400 \
  RECOVERY_REHEARSAL_REPORT_FILE="$ERS336_OBJECTIVE_REPORT" \
  > /tmp/ers-336-objective.out 2>&1

ERS336_OBJECTIVE_STATUS=$?

set -e

cat /tmp/ers-336-objective.out

Expected includes:

Recovery rehearsal objective_failed.
Recovery mechanics verified, but one or more RPO/RTO objectives were not met.

Assert report:

if [[ "$ERS336_OBJECTIVE_STATUS" -ne 0 ]] \
  && jq -e \
    '
      .status == "objective_failed"
      and .recovery.isolated_backend_health == "passed"
      and .recovery.post_boot_integrity_check == "ok"
      and .recovery.post_boot_foreign_key_check == "clean"
      and .objectives.max_rpo_seconds == 1
      and .objectives.rpo_met == false
      and .objectives.rto_met == true
    ' \
    "$ERS336_OBJECTIVE_REPORT" \
    >/dev/null
then
  echo "PASS: objective failure is distinguished from recovery-mechanics failure"
else
  echo "FAIL: objective-failure contract is incorrect"
fi

Expected:

PASS: objective failure is distinguished from recovery-mechanics failure

Persist:

printf "export ERS336_OBJECTIVE_REPORT='%s'\n" \
  "$ERS336_OBJECTIVE_REPORT" \
  >> /tmp/ers-336-manual-session.env

16. Phase A — Test 13 — Migration Gate Records Compatible-Code Evidence and Blocks Rehearsal While Armed

Define:

source /tmp/ers-336-manual-session.env

export ERS336_PREVIOUS_REVISION="$(
  git -C "$ERS336_ENV_DIR" rev-parse HEAD^
)"

export ERS336_PREVIOUS_TREE="$(
  git -C "$ERS336_ENV_DIR" rev-parse "${ERS336_PREVIOUS_REVISION}^{tree}"
)"

export ERS336_TARGET_REVISION="$(
  git -C "$ERS336_ENV_DIR" rev-parse HEAD
)"

export ERS336_TARGET_TREE="$(
  git -C "$ERS336_ENV_DIR" rev-parse HEAD^{tree}
)"

Run gate:

make \
  SERVER_ROOT="$ERS336_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS336_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS336_CONTAINER_PREFIX" \
  MIGRATION_PREVIOUS_REVISION="$ERS336_PREVIOUS_REVISION" \
  MIGRATION_PREVIOUS_TREE_SHA="$ERS336_PREVIOUS_TREE" \
  MIGRATION_TARGET_REVISION="$ERS336_TARGET_REVISION" \
  MIGRATION_TARGET_TREE_SHA="$ERS336_TARGET_TREE" \
  server-migration-recovery-gate \
  ENV=development

Expected includes:

Migration/deployment recovery gate ARMED.

Verify:

if grep -qx 'status=armed' "$ERS336_MIGRATION_STATE_FILE" \
  && grep -qx "previous_revision=$ERS336_PREVIOUS_REVISION" "$ERS336_MIGRATION_STATE_FILE" \
  && grep -qx "previous_tree_sha=$ERS336_PREVIOUS_TREE" "$ERS336_MIGRATION_STATE_FILE" \
  && grep -qx "target_revision=$ERS336_TARGET_REVISION" "$ERS336_MIGRATION_STATE_FILE" \
  && grep -qx "target_tree_sha=$ERS336_TARGET_TREE" "$ERS336_MIGRATION_STATE_FILE"
then
  echo "PASS: migration gate records exact previous/target revision-tree evidence"
else
  echo "FAIL: migration gate revision evidence is incomplete"
fi

Expected:

PASS: migration gate records exact previous/target revision-tree evidence

Try rehearsal while armed:

set +e

make \
  SERVER_ROOT="$ERS336_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS336_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS336_CONTAINER_PREFIX" \
  server-recovery-rehearsal \
  ENV=development \
  RECOVERY_REHEARSAL_BACKUP_FILE="$ERS336_SOURCE_BACKUP" \
  > /tmp/ers-336-armed-rehearsal.out 2>&1

ERS336_ARMED_STATUS=$?

set -e

cat /tmp/ers-336-armed-rehearsal.out

Expected includes:

Refusing recovery rehearsal while a migration/deployment recovery gate remains armed:

Assert non-zero.

Roll back:

make \
  SERVER_ROOT="$ERS336_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS336_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS336_CONTAINER_PREFIX" \
  server-migration-recovery-rollback \
  ENV=development

Expected:

Migration/deployment recovery gate rollback completed.

Confirm compatible-code evidence remains:

if grep -qx 'status=rolled_back' "$ERS336_MIGRATION_STATE_FILE" \
  && grep -qx "previous_revision=$ERS336_PREVIOUS_REVISION" "$ERS336_MIGRATION_STATE_FILE" \
  && grep -qx "previous_tree_sha=$ERS336_PREVIOUS_TREE" "$ERS336_MIGRATION_STATE_FILE"
then
  echo "PASS: rollback state retains compatible previous-code evidence"
else
  echo "FAIL: rollback state lost compatible previous-code evidence"
fi

Explicitly restart:

make \
  SERVER_ROOT="$ERS336_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS336_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS336_CONTAINER_PREFIX" \
  server-up \
  ENV=development

make \
  SERVER_ROOT="$ERS336_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS336_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS336_CONTAINER_PREFIX" \
  server-backend-health \
  ENV=development

17. Phase A — Test 14 — Read and Validate the Operational Runbook

Set:

export ERS336_RUNBOOK="$ERS336_ENV_DIR/docs/database-recovery-runbook.md"
export ERS336_ACCESS_GUIDE="$ERS336_ENV_DIR/docs/owner-unavailable-recovery-access.md"

Confirm both exist:

test -s "$ERS336_RUNBOOK" \
  && test -s "$ERS336_ACCESS_GUIDE" \
  && echo "PASS: runbook and owner-unavailable access guide exist"

Expected:

PASS: runbook and owner-unavailable access guide exist

Confirm the runbook no longer contains the vague phrase that triggered this hardening:

if grep -Fq 'normal administrative sources' "$ERS336_RUNBOOK"; then
  echo "FAIL: runbook still uses vague 'normal administrative sources'"
else
  echo "PASS: vague administrative-source wording has been removed"
fi

Expected:

PASS: vague administrative-source wording has been removed

Confirm owner-unavailable contract:

grep -Fq 'Owner-unavailable recovery contract' "$ERS336_RUNBOOK" \
  && grep -Fq 'ops/recovery-access.production.env' "$ERS336_RUNBOOK" \
  && grep -Fq 'lastpass-shared://ERS-Recovery/Production/.env.production' "$ERS336_RUNBOOK" \
  && grep -Fq 'lastpass-shared://ERS-Recovery/Production/Hetzner-S3-ers-backup' "$ERS336_RUNBOOK" \
  && grep -Fq 'server-owner-unavailable-recovery-preflight' "$ERS336_RUNBOOK" \
  && grep -Fq 'server-owner-unavailable-recovery-rehearsal' "$ERS336_RUNBOOK" \
  && grep -Fq 'OWNER-UNAVAILABLE-RECOVERY-ACCESS-CONFIRMED' "$ERS336_RUNBOOK" \
  && echo "PASS: runbook contains explicit owner-unavailable recovery sources and commands"

Confirm compatible revision recovery:

grep -Fq '.repository.revision' "$ERS336_RUNBOOK" \
  && grep -Fq '.repository.tree_sha' "$ERS336_RUNBOOK" \
  && grep -Fq 'git cat-file -e "${BACKUP_REVISION}^{commit}"' "$ERS336_RUNBOOK" \
  && echo "PASS: runbook requires deterministic database-compatible Git evidence"

Confirm GitHub Actions secrets are not treated as a recovery source:

grep -Fq \
  'GitHub Actions secrets are not an authoritative recovery source' \
  "$ERS336_RUNBOOK" \
  && echo "PASS: runbook does not depend on unreadable GitHub Actions secrets"

Confirm exact Production restore token:

grep -Fq 'RESTORE_CONFIRM=RESTORE-PRODUCTION' "$ERS336_RUNBOOK" \
  && echo "PASS: runbook documents exact Production restore confirmation"

Confirm owner contact fails the rehearsal:

grep -Fq \
  'If the technician has to contact the system owner, the owner-unavailable rehearsal fails.' \
  "$ERS336_RUNBOOK" \
  && echo "PASS: runbook explicitly makes owner dependency a rehearsal failure"

Human review

Read the complete runbook:

less "$ERS336_RUNBOOK"

A qualified technician must be able to locate exact procedures for:

1. Routine non-destructive recovery rehearsal
2. Database corruption / host survives
3. Complete host loss
4. Retrieval of canonical .env.production
5. Retrieval/installation of S3 credentials
6. Repository-compatible revision/tree verification
7. Production restore confirmation
8. Migration rollback
9. Previous-compatible-code recovery
10. Live migration failure
11. Restore failure after destructive recovery begins
12. Stop-and-escalate conditions
13. RPO/RTO evidence
14. Owner-unavailable quarterly rehearsal

Record FAIL if any procedure still requires undocumented owner knowledge.

18. Phase A — Test 15 — Run the Full Automated Recovery Chain

From the primary repository root:

make production-environment-guardrails-check

Expected Bite 33.1 PASS.

Run:

make backup-creation-retention-verification-check

Expected Bite 33.2 PASS.

Run:

make off-host-backup-protection-check

Expected Bite 33.3 PASS.

Run:

make restore-tooling-recovery-verification-check

Expected Bite 33.4 PASS.

Run:

make migration-deployment-recovery-gate-check

Expected:

Bite 33.5 migration/deployment recovery gate verified.

Run:

make recovery-rehearsal-operational-runbook-check

Expected:

Bite 33.6 recovery rehearsal and operational runbook verified.

Run:

make owner-unavailable-recovery-readiness-check

Expected:

Bite 33.6 owner-unavailable recovery readiness verified.

Finally:

make local-check

Expected: complete LOCAL gate passes.

CI must also pass for the exact SHA being promoted.

19. Phase A — Test 16 — Retain Evidence and Clean Up Only Bite 33.6 Resources

Reload:

source /tmp/ers-336-manual-session.env

Create:

export ERS336_EVIDENCE_DIR="$HOME/Downloads/ers-bite-336-manual-evidence-7c7c5cc6"

rm -rf "$ERS336_EVIDENCE_DIR"
mkdir -p "$ERS336_EVIDENCE_DIR"

Copy non-secret reports:

cp "$ERS336_LOCAL_REPORT" \
  "$ERS336_EVIDENCE_DIR/local-rehearsal.json"

cp "$ERS336_RECEIPT_REPORT" \
  "$ERS336_EVIDENCE_DIR/receipt-rehearsal.json"

cp "$ERS336_HOSTLOSS_REPORT" \
  "$ERS336_EVIDENCE_DIR/host-loss-rehearsal.json"

cp "$ERS336_OBJECTIVE_REPORT" \
  "$ERS336_EVIDENCE_DIR/objective-failure.json"

cp "$ERS336_RUNBOOK" \
  "$ERS336_EVIDENCE_DIR/database-recovery-runbook.md"

cp "$ERS336_ACCESS_GUIDE" \
  "$ERS336_EVIDENCE_DIR/owner-unavailable-recovery-access.md"

cp "$ERS336_ENV_DIR/ops/recovery-access.production.env" \
  "$ERS336_EVIDENCE_DIR/recovery-access.production.env"

Check retained evidence:

if grep -R -Eiq \
  'aws_secret_access_key|PRIVATE KEY|JWT_SECRET=|LEDGER_CORRECTION_KEY=|LEDGER_SETTLEMENT_KEY=|OPENAI_API_KEY=' \
  "$ERS336_EVIDENCE_DIR"
then
  echo "FAIL: retained evidence appears to contain secret material"
else
  echo "PASS: retained evidence contains no obvious secret material"
fi

Expected:

PASS: retained evidence contains no obvious secret material

Remove only dedicated Docker resources:

make \
  SERVER_ROOT="$ERS336_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS336_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS336_CONTAINER_PREFIX" \
  server-down-volumes \
  ENV=development

Verify:

ERS336_REMAINING_CONTAINERS="$(
  docker ps -a --format '{{.Names}}' \
    | grep -E "^${ERS336_CONTAINER_PREFIX}-(backend|frontend|caddy)$" \
    || true
)"

[[ -z "$ERS336_REMAINING_CONTAINERS" ]] \
  && echo "PASS: Bite 33.6 containers removed"

Verify volumes:

ERS336_REMAINING_VOLUMES="$(
  docker volume ls --format '{{.Name}}' \
    | grep "^${ERS336_COMPOSE_PROJECT}_" \
    || true
)"

[[ -z "$ERS336_REMAINING_VOLUMES" ]] \
  && echo "PASS: Bite 33.6 volumes removed"

Remove worktree:

git worktree remove --force "$ERS336_ENV_DIR"
git worktree prune
rm -rf "$ERS336_SERVER_ROOT"
rm -rf /tmp/ers-336-host-loss-hidden
rm -f /tmp/ers-336-manual-session.env

Do not remove:

$HOME/.aws/credentials

Do not remove normal ers-dev-* resources.

20. Phase A Promotion Gate

Promotion is allowed only if all of the following are true:

  • Exact branch/SHA validated.
  • Shared recovery access manifest contains explicit non-personal locators.
  • Designated technician can access all required ERS-Recovery LastPass items.
  • Designated technician has independent MFA/recovery capability.
  • Technician access does not depend on owner device/approval.
  • New verified Development backup contains full repository revision/tree evidence.
  • Recorded backup revision exists in Git and its tree matches.
  • Exact S3 DB and manifest VersionIds exist.
  • Local-backup rehearsal passes.
  • Receipt-based exact-version rehearsal passes.
  • Host-loss backup-name rehearsal passes with local artifacts unavailable.
  • Rehearsal leaves live Development DB/service intact.
  • RPO/RTO defaults are 24 hours / 4 hours.
  • Deliberate RPO miss returns non-zero with objective_failed.
  • Multiple recovery sources are refused.
  • Owner-unavailable target refuses non-Production use.
  • Technician identity is mandatory.
  • Migration gate records previous/target revision/tree.
  • Recovery rehearsal is refused while migration gate is armed.
  • Rollback retains previous compatible-code evidence.
  • Runbook contains explicit shared administrative sources.
  • Runbook contains deterministic backup-compatible Git recovery.
  • Runbook explicitly says contacting the owner fails owner-unavailable rehearsal.
  • Bite 33.1–33.6 regressions pass.
  • Owner-unavailable readiness regression passes.
  • make local-check passes.
  • CI passes for the promoted SHA.

Passing Phase A authorizes promotion. It does not yet prove Production owner-unavailable operational readiness.

21. Phase B — Mandatory Post-Promotion Operational Activation

Important

Phase B is mandatory before Bite 33.6 is declared operationally complete.

It must be performed after the Bite 33.6 code is running in Production and a new Production backup has been created with compatible Git revision/tree metadata.

The primary system owner must be unavailable during the designated-technician portion.

22. Phase B — Test 17 — Create the First Owner-Unavailable-Eligible Production Backup

Identity / context

This step may be performed as part of the normal Production deployment by the authorized Production operator.

Do not change application data for this test.

Procedure

On the Production host, confirm the deployed repository contains the new backup metadata contract:

cd /opt/EnterpriseRemoteSystems/production

git rev-parse HEAD
git rev-parse HEAD^{tree}

grep -n 'BACKUP_REPOSITORY_REVISION' \
  scripts/server-sqlite-backup.sh

Create a fresh verified Production backup:

make server-backup ENV=production

Record only the resulting managed filename:

app-YYYYMMDDTHHMMSSZ.db

Locate its manifest:

LATEST_PROD_BACKUP="$(
  find /opt/EnterpriseRemoteSystems/production/backups \
    -maxdepth 1 \
    -type f \
    -name 'app-*.db' \
    -print \
  | sort \
  | tail -1
)"

LATEST_PROD_MANIFEST="$LATEST_PROD_BACKUP.manifest.json"

Confirm repository evidence:

jq '{
  created_at,
  repository,
  backup: {
    filename: .backup.filename,
    sha256: .backup.sha256
  }
}' "$LATEST_PROD_MANIFEST"

Required:

repository.revision = 40 lowercase hex characters
repository.tree_sha = 40 lowercase hex characters

Verify:

BACKUP_REVISION="$(jq -er '.repository.revision' "$LATEST_PROD_MANIFEST")"
BACKUP_TREE="$(jq -er '.repository.tree_sha' "$LATEST_PROD_MANIFEST")"

git cat-file -e "${BACKUP_REVISION}^{commit}"

test "$(git rev-parse "${BACKUP_REVISION}^{tree}")" = "$BACKUP_TREE" \
  && echo "PASS: first Production owner-unavailable backup has valid compatible-code evidence"

Expected:

PASS: first Production owner-unavailable backup has valid compatible-code evidence

Confirm off-host receipt exists:

test -s "$LATEST_PROD_BACKUP.offhost.json" \
  && echo "PASS: Production backup has protected off-host receipt evidence"

Do not send the backup filename privately to the technician as the only way they can identify it. The technician must be able to discover an eligible managed restore point from documented shared Hetzner access.

23. Phase B — Test 18 — Owner Becomes Unavailable

From this point until Test 21 is complete:

the primary system owner must not provide assistance.

The technician may use only:

the ERS runbook in GitHub
their own GitHub access
the shared ERS-Recovery LastPass collection
their own independent MFA/recovery material
their pre-authorized Hetzner project access
their pre-authorized DNS administrative access
a clean recovery host/workstation

The technician must record FAIL immediately if they need to ask the owner for:

a password
MFA approval
a key
a file
a hostname
a repository identity
a backup filename
a DNS provider/account identity
a Hetzner project/account identity
a recovery command
a recovery decision

24. Phase B — Test 19 — Designated Technician Reconstructs Recovery Prerequisites on a Clean Host

Identity / context

  • Operator: designated technician other than primary owner.
  • Host: clean disposable recovery host or equivalent clean replacement-host context.
  • The host must not contain /opt/EnterpriseRemoteSystems from an earlier rehearsal.
  • Do not point Production DNS to this rehearsal host.

Prerequisites on the clean host

The technician must have:

git
Docker Engine / Docker Compose
make
jq
curl

Verify:

for command in git docker make jq curl; do
  command -v "$command" >/dev/null 2>&1 \
    && echo "PASS: $command available" \
    || echo "FAIL: $command missing"
done

docker info >/dev/null 2>&1 \
  && echo "PASS: Docker daemon reachable" \
  || echo "FAIL: Docker daemon unavailable"

Clone shared GitHub repository

Using the technician's own GitHub access:

sudo mkdir -p /opt/EnterpriseRemoteSystems
sudo chown "$USER":"$(id -gn)" /opt/EnterpriseRemoteSystems

git clone \
  [email protected]:RodrigoMattosoSilveira/EnterpriseRemoteSystems.git \
  /opt/EnterpriseRemoteSystems/production

cd /opt/EnterpriseRemoteSystems/production
git fetch origin --prune

Checkout the currently deployed Bite 33.6-capable Production revision according to the shared Production deployment record/repository state. Record:

git rev-parse HEAD
git rev-parse HEAD^{tree}

Retrieve canonical Production environment from shared LastPass

Using:

ERS-Recovery / Production / .env.production

install the exact shared canonical file at:

/opt/EnterpriseRemoteSystems/production/.env.production

Then:

chmod 600 /opt/EnterpriseRemoteSystems/production/.env.production

Do not use:

./scripts/init-server-env.sh production

as a substitute for the canonical Production file.

Retrieve S3 credentials from shared LastPass

Using:

ERS-Recovery / Production / Hetzner-S3-ers-backup

install the credential material at:

/opt/EnterpriseRemoteSystems/secrets/aws-credentials

with profile:

[ers-backup]

Create directory/mode:

mkdir -p /opt/EnterpriseRemoteSystems/secrets
chmod 700 /opt/EnterpriseRemoteSystems/secrets
chmod 600 /opt/EnterpriseRemoteSystems/secrets/aws-credentials

Never paste the secret into the shell command line or test notes.

Prove other administrative access

The technician must independently open/access:

ERS-Recovery / Production / Host-SSH
ERS-Recovery / Infrastructure / Hetzner-Console
ERS-Recovery / Infrastructure / DNS

The technician must successfully sign in to the corresponding Hetzner/DNS administrative services using their own account/MFA where supported.

No DNS change is required.

Run the owner-unavailable preflight

From:

cd /opt/EnterpriseRemoteSystems/production

run:

make server-owner-unavailable-recovery-preflight ENV=production

Expected output begins:

Owner-unavailable recovery access preflight passed.

Expected output also identifies:

Repository source: github://RodrigoMattosoSilveira/EnterpriseRemoteSystems
Production environment source: lastpass-shared://ERS-Recovery/Production/.env.production
S3 credential source: lastpass-shared://ERS-Recovery/Production/Hetzner-S3-ers-backup
Host access source: lastpass-shared://ERS-Recovery/Production/Host-SSH
Hetzner console source: lastpass-shared://ERS-Recovery/Infrastructure/Hetzner-Console
DNS source: lastpass-shared://ERS-Recovery/Infrastructure/DNS
MFA recovery source: technician-controlled://independent-mfa-and-recovery-codes
S3 target: https://hel1.your-objectstorage.com / hel1 / ers

PASS criteria

PASS only if the technician reconstructs all prerequisites and the preflight passes without owner assistance.

25. Phase B — Test 20 — Technician Independently Identifies an Eligible Production Restore Point

Purpose

The technician must not depend on the owner knowing the backup filename.

Procedure

Read the active Production S3 prefix from the canonical recovered config:

cd /opt/EnterpriseRemoteSystems/production

PROD_S3_PREFIX="$(
  grep '^SERVER_OFFHOST_BACKUP_S3_PREFIX=' .env.production \
  | cut -d= -f2-
)"

printf 'Production S3 prefix: %s\n' "$PROD_S3_PREFIX"

Using the documented shared Hetzner access, inspect:

Endpoint: https://hel1.your-objectstorage.com
Region:   hel1
Bucket:   ers
Prefix:   <value from .env.production>/production/

Managed backups are grouped as:

<prefix>/production/app-<timestamp>.db/

Select the newest appropriate backup created after Bite 33.6 backup revision/tree evidence was deployed.

Record only:

export RECOVERY_BACKUP_NAME='app-YYYYMMDDTHHMMSSZ.db'

The technician must determine this from shared provider access/evidence, not by asking the owner.

PASS criteria

PASS only if the technician independently identifies a new eligible managed backup.

26. Phase B — Test 21 — Owner-Unavailable Clean-Host Recovery Rehearsal

Purpose

This is the decisive acceptance test.

It starts without a local database backup and without a local .offhost.json receipt.

Procedure

From:

cd /opt/EnterpriseRemoteSystems/production

Build the current recovery-capable Production backend image:

make server-prod-build

Define technician identity. Use an auditable non-secret identity such as the technician's company username/email label; do not use a password/token.

Example:

export RECOVERY_TECHNICIAN_ID='technician-name-or-company-id'

Run:

make server-owner-unavailable-recovery-rehearsal \
  ENV=production \
  RECOVERY_REHEARSAL_OFFHOST_BACKUP_NAME="$RECOVERY_BACKUP_NAME" \
  RECOVERY_TECHNICIAN_ID="$RECOVERY_TECHNICIAN_ID" \
  RECOVERY_SHARED_ACCESS_CONFIRM=OWNER-UNAVAILABLE-RECOVERY-ACCESS-CONFIRMED

Expected preflight:

Owner-unavailable recovery access preflight passed.

Expected final summary includes:

Recovery rehearsal passed.
Environment: production
Source: app-...db (offhost-backup-name)
Selected restore-point age / RPO: ... true
Recovery rehearsal RTO: ... true
Owner-unavailable technician: ...
Compatible backup revision: <40-char SHA> (tree <40-char SHA>)
Shared recovery access manifest: ... (SHA-256 ...)

The command must exit 0.

Locate the newest rehearsal report:

OWNER_REPORT="$(
  find /opt/EnterpriseRemoteSystems/production/recovery/rehearsals \
    -type f \
    -name 'rehearsal.json' \
    -print \
  | sort \
  | tail -1
)"

printf 'Owner-unavailable report: %s\n' "$OWNER_REPORT"

Display only structured non-secret evidence:

jq '{
  status,
  environment,
  repository,
  owner_unavailable,
  source: {
    kind: .source.kind,
    backup_file: .source.backup_file,
    backup_created_at: .source.backup_created_at,
    backup_sha256: .source.backup_sha256,
    repository_revision: .source.repository_revision,
    repository_tree_sha: .source.repository_tree_sha,
    compatible_revision_available: .source.compatible_revision_available,
    receipt_version_id: .source.receipt_version_id,
    remote: .source.remote
  },
  recovery,
  objectives
}' "$OWNER_REPORT"

Assert:

jq -e \
  '
    .status == "passed"
    and .environment == "production"
    and .owner_unavailable.mode == true
    and .owner_unavailable.shared_access_confirmed == true
    and (.owner_unavailable.technician_id | length) > 0
    and (.owner_unavailable.access_manifest_sha256 | length) == 64
    and .owner_unavailable.sources.repository
        == "github://RodrigoMattosoSilveira/EnterpriseRemoteSystems"
    and .owner_unavailable.sources.production_environment
        == "lastpass-shared://ERS-Recovery/Production/.env.production"
    and .owner_unavailable.sources.s3_credentials
        == "lastpass-shared://ERS-Recovery/Production/Hetzner-S3-ers-backup"
    and .owner_unavailable.sources.host_access
        == "lastpass-shared://ERS-Recovery/Production/Host-SSH"
    and .owner_unavailable.sources.hetzner_console
        == "lastpass-shared://ERS-Recovery/Infrastructure/Hetzner-Console"
    and .owner_unavailable.sources.dns
        == "lastpass-shared://ERS-Recovery/Infrastructure/DNS"
    and .owner_unavailable.sources.mfa_recovery
        == "technician-controlled://independent-mfa-and-recovery-codes"
    and .source.kind == "offhost-backup-name"
    and (.source.receipt_version_id | length) > 0
    and (.source.remote.backup_version_id | length) > 0
    and (.source.remote.manifest_version_id | length) > 0
    and (.source.repository_revision | test("^[0-9a-f]{40}$"))
    and (.source.repository_tree_sha | test("^[0-9a-f]{40}$"))
    and .source.compatible_revision_available == true
    and .recovery.restored_sha256 == .source.backup_sha256
    and .recovery.isolated_backend_health == "passed"
    and .recovery.post_boot_integrity_check == "ok"
    and .recovery.post_boot_foreign_key_check == "clean"
    and .objectives.max_rpo_seconds == 86400
    and .objectives.max_rto_seconds == 14400
    and .objectives.rpo_met == true
    and .objectives.rto_met == true
  ' \
  "$OWNER_REPORT" \
  >/dev/null \
  && echo "PASS: owner-unavailable rehearsal evidence satisfies the full recovery contract"

Expected:

PASS: owner-unavailable rehearsal evidence satisfies the full recovery contract

Independently verify compatible revision/tree:

BACKUP_REVISION="$(
  jq -er '.source.repository_revision' "$OWNER_REPORT"
)"

BACKUP_TREE="$(
  jq -er '.source.repository_tree_sha' "$OWNER_REPORT"
)"

git fetch origin --prune
git cat-file -e "${BACKUP_REVISION}^{commit}"

test "$(git rev-parse "${BACKUP_REVISION}^{tree}")" = "$BACKUP_TREE" \
  && echo "PASS: recovered backup identifies a resolvable compatible application revision/tree"

Expected:

PASS: recovered backup identifies a resolvable compatible application revision/tree

Check report for obvious secrets:

if grep -Eiq \
  'aws_secret_access_key|PRIVATE KEY|JWT_SECRET=|LEDGER_CORRECTION_KEY=|LEDGER_SETTLEMENT_KEY=|OPENAI_API_KEY=' \
  "$OWNER_REPORT"
then
  echo "FAIL: owner-unavailable rehearsal evidence appears to contain secret material"
else
  echo "PASS: owner-unavailable rehearsal evidence contains no obvious secret material"
fi

Expected:

PASS: owner-unavailable rehearsal evidence contains no obvious secret material

PASS criteria

The technician independently obtains prerequisites, independently locates the backup, bootstraps the remote protected receipt, retrieves the exact DB and manifest versions, verifies the database, proves compatible Git evidence, migrates a disposable copy, starts an isolated Production backend against the recovered data, passes integrity/FKs, and meets RPO/RTO.

The primary owner provides zero assistance.

27. Phase B — Test 22 — Owner-Unavailable Human Acceptance

After Test 21, the designated technician records:

Did I contact the primary owner?                         NO
Did I need owner MFA approval?                          NO
Did I use the owner's personal LastPass vault?          NO
Did I use a file stored only on the owner's computer?   NO
Did I ask the owner which backup to use?                NO
Did I ask the owner for Git/Hetzner/DNS account info?   NO
Did I ask the owner which code revision was compatible? NO
Could I resolve the compatible code from backup evidence? YES
Did the isolated recovered backend become healthy?      YES
Did SQLite integrity pass?                              YES
Were foreign keys clean?                                YES
Was RPO <= 24h?                                         YES
Was RTO <= 4h?                                          YES

If any required NO becomes YES, or any required YES becomes NO, the owner-unavailable use case has failed.

28. Phase B Operational Acceptance Gate

Bite 33.6 is operationally complete only when:

  • Bite 33.6 code is running in Production.
  • Fresh verified Production backup was created after deployment.
  • Production backup manifest contains compatible repository revision/tree.
  • Production backup was successfully protected off-host.
  • Designated technician is someone other than primary owner.
  • Technician uses their own GitHub/provider accounts and MFA.
  • Technician independently accesses the shared ERS-Recovery collection.
  • Technician retrieves canonical .env.production from shared LastPass.
  • Technician retrieves S3 credentials from shared LastPass.
  • Technician independently has host/replacement-host SSH material.
  • Technician independently has Hetzner administrative access.
  • Technician independently has DNS administrative access.
  • Clean-host owner-unavailable preflight passes.
  • Technician independently identifies an eligible managed backup.
  • Rehearsal starts from backup name only; no local receipt is supplied.
  • Protected remote receipt is bootstrapped successfully.
  • Exact database VersionId is recorded.
  • Exact manifest VersionId is recorded.
  • Compatible repository revision is full and resolvable from shared GitHub.
  • Compatible tree matches the recorded revision.
  • Scratch restored SHA equals source backup SHA.
  • Isolated recovered backend becomes healthy.
  • Integrity check is ok.
  • Foreign-key check is clean.
  • RPO <= 24 hours.
  • RTO <= 4 hours.
  • Rehearsal evidence contains no secrets.
  • Technician did not contact or depend on the primary owner.

29. Promotion / Closure Rules

Promotion to Production

Promotion is allowed after:

Phase A = PASS
LOCAL = PASS
CI = PASS

This is necessary because the enhanced Production backup metadata must first be deployed before an eligible Production restore point can exist.

Bite 33.6 operational closure

Do not close Bite 33.6 as operationally complete until:

Phase B = PASS

If Phase B cannot be performed successfully by another technician without contacting the owner, keep Bite 33.6 open and treat the failure as a recovery readiness defect.

30. Promotion / Operational-Closure Blocking Defects

Block promotion or operational closure, as applicable, if:

  • recovery sources remain vague or owner-personal;
  • designated technician cannot access shared LastPass items;
  • designated technician requires owner MFA approval;
  • backup manifest lacks repository revision/tree;
  • backup repository revision/tree cannot be resolved from Git;
  • host-loss rehearsal depends on a local receipt;
  • S3 exact VersionIds are absent;
  • ETag is treated as SHA-256;
  • scratch restored SHA differs from backup SHA;
  • rehearsal touches or replaces the live database;
  • isolated recovered backend cannot become healthy;
  • integrity/FK verification fails;
  • missed RPO/RTO returns success;
  • owner-unavailable preflight accepts a personal/vague recovery source;
  • migration rollback loses previous compatible-code evidence;
  • runbook tells the operator to guess a compatible revision;
  • canonical .env.production is regenerated instead of retrieved from the shared authoritative source;
  • any required recovery artifact exists only on the owner's machine;
  • any retained evidence contains secrets;
  • technician must contact the owner during Phase B;
  • any Bite 33.1–33.6 regression, LOCAL, or CI check fails.

31. Evidence to Retain

Retain non-secret evidence only.

Phase A

Branch
SHA
Tree SHA
Development S3 prefix
Selected backup filename
Backup SHA-256
Backup compatible Git revision
Backup compatible Git tree
DB S3 VersionId
Manifest S3 VersionId
Local rehearsal report
Receipt rehearsal report
Host-loss rehearsal report
Objective-failure report
Migration-gate previous/target revision-tree evidence
Automated checks
LOCAL result
CI result

Phase B

Production backup managed filename
Production backup created_at
Production backup SHA-256
Production backup compatible Git revision
Production backup compatible Git tree
Designated technician identity
Owner-unavailable rehearsal report
Remote receipt VersionId
DB VersionId
Manifest VersionId
Actual RPO seconds
Actual RTO seconds
Technician owner-contact attestation

Do not retain:

AWS secret access keys
LastPass secret values
JWT secrets
SSH private keys
passwords
ledger keys
OpenAI/API keys
MFA recovery codes

32. Glossary Impact

none

⚠️ **GitHub.com Fallback** ⚠️