Bite 33.4 — Restore Tooling and Recovery Verification - RodrigoMattosoSilveira/EnterpriseRemoteSystems GitHub Wiki

Bite 33.4 — Restore Tooling and Recovery Verification

Manual Verification Checklist Before Promotion

Branch: 941-bite-334-restore-tooling-and-recovery-verification
Validated HEAD SHA: b5e2bb928cc3b95af60db82a1811de0b4cc5139f

1. Purpose

This suite manually verifies the complete Bite 33.4 recovery contract without touching Test or Production.

The tests use a disposable deployed Development environment created under:

/tmp/ers-334-manual-server/development

and the real Hetzner Object Storage bucket already established for Bite 33.3:

Region:   hel1
Endpoint: https://hel1.your-objectstorage.com
Bucket:   ers
Profile:  ers-backup

The suite proves:

a verified Bite 33.2 local backup can be staged as a recovery candidate;
a verified Bite 33.3 S3 backup can be materialized from a local receipt;
S3 recovery also works when the local receipt is unavailable and only the managed backup filename survives;
recovery uses the exact database and manifest S3 VersionIds recorded by Bite 33.3;
a candidate for one environment is rejected when verified as another environment;
restore targets outside the selected environment's recovery/ tree are rejected;
an incorrect restore confirmation token is rejected before the live database is touched;
an actual deployed restore replaces the Development database with the exact verified earlier backup;
a deliberate database probe created after the backup disappears after restoration;
the database being replaced is captured in both a fresh pre-restore Bite 33.2/33.3 backup and a preserved pre-restore database copy;
the restore report accurately records source, previous-target, restored-target, integrity, foreign-key, migration, and record-count evidence;
a failure after the workflow has entered its destructive phase leaves the backend stopped;
a failure before replacement does not silently change the database;
the backend recovers healthy after a successful restore.

No ERS application login is required. These are operator/recovery tests and are intentionally terminal-driven.

2. Safety Rules

These tests must not use:

Test;
Production;
/opt/EnterpriseRemoteSystems/test;
/opt/EnterpriseRemoteSystems/production;
the normal Development Docker namespace ers-dev.

This suite uses its own dedicated disposable Docker identity:

Compose project:  ers334-b5e2bb92
Container prefix: ers334-b5e2bb92

Containers:
  ers334-b5e2bb92-backend
  ers334-b5e2bb92-frontend
  ers334-b5e2bb92-caddy

Volumes:
  ers334-b5e2bb92_backend-data
  ers334-b5e2bb92_caddy-data
  ers334-b5e2bb92_caddy-config

An existing normal Development deployment such as:

ers-dev-backend
ers-dev-frontend
ers-dev-caddy

is not a failure and must be left untouched.

Test 01 checks only the dedicated Bite 33.4 manual-test namespace. If stale ers334-b5e2bb92-* resources remain from an earlier interrupted execution of this same manual suite, Test 01 provides explicit cleanup commands for those manual-test resources only.

The S3 objects created under this suite use a unique prefix:

manual-334-<UTC timestamp>

Because the Hetzner bucket has Object Lock / Governance retention, those objects may remain until retention expires. Do not use Governance bypass simply to clean up this manual test.

3. Test 00 — Confirm Exact Branch, SHA, and Required Tools

Identity

No ERS login required.

Terminal

Run from the ERS repository root.

Procedure

Run:

git branch --show-current
git rev-parse HEAD

Expected exactly:

941-bite-334-restore-tooling-and-recovery-verification
b5e2bb928cc3b95af60db82a1811de0b4cc5139f

Confirm the required commands:

for command in git docker sqlite3 aws jq shasum python3 make tar; do
  command -v "$command" >/dev/null 2>&1 \
    && echo "PASS: $command is available" \
    || echo "FAIL: $command is missing"
done

Expected: one PASS: line for every command and no FAIL: line.

Confirm Docker is reachable:

docker info >/dev/null \
  && echo "PASS: Docker daemon is reachable" \
  || echo "FAIL: Docker daemon is not reachable"

Expected:

PASS: Docker daemon is reachable

PASS Criteria

PASS only if:

branch is exactly 941-bite-334-restore-tooling-and-recovery-verification;
HEAD is exactly b5e2bb928cc3b95af60db82a1811de0b4cc5139f;
all required commands are present;
Docker is reachable.

Promotion Blocker

Do not continue if the branch/SHA differs or Docker is unavailable.

4. Test 01 — Prove the Dedicated Bite 33.4 Docker Namespace Is Safe to Use

Identity

No ERS login required.

Purpose

Keep this recovery rehearsal completely isolated from any existing normal Development deployment.

The normal ers-dev-* containers may already be running. Do not stop, delete, rename, or reuse them for this test.

Procedure

Define the dedicated manual-test Docker identity:

export ERS334_COMPOSE_PROJECT="ers334-b5e2bb92"
export ERS334_CONTAINER_PREFIX="ers334-b5e2bb92"

For information only, show whether the normal Development deployment exists:

docker ps -a --format '{{.Names}}' \
  | grep -E '^ers-dev-(backend|frontend|caddy)$' \
  || true

If names such as these appear:

ers-dev-backend
ers-dev-frontend
ers-dev-caddy

that is acceptable. They are outside this suite's dedicated namespace.

Check only for stale Bite 33.4 manual-test containers:

ERS334_EXISTING_CONTAINERS="$(
  docker ps -a --format '{{.Names}}' \
    | grep -E '^ers334-b5e2bb92-(backend|frontend|caddy)$' \
    || true
)"

if [ -n "$ERS334_EXISTING_CONTAINERS" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--n-"$ERS334_EXISTING_CONTAINERS"-); then
  echo "STALE: Bite 33.4 manual-test containers are present:"
  printf '%s\n' "$ERS334_EXISTING_CONTAINERS"
else
  echo "PASS: no stale Bite 33.4 manual-test containers are present"
fi

Check only for stale Bite 33.4 manual-test volumes:

ERS334_EXISTING_VOLUMES="$(
  docker volume ls --format '{{.Name}}' \
    | grep '^ers334-b5e2bb92_' \
    || true
)"

if [ -n "$ERS334_EXISTING_VOLUMES" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--n-"$ERS334_EXISTING_VOLUMES"-); then
  echo "STALE: Bite 33.4 manual-test Docker volumes are present:"
  printf '%s\n' "$ERS334_EXISTING_VOLUMES"
else
  echo "PASS: no stale Bite 33.4 manual-test Docker volumes are present"
fi

If neither command reported STALE:, continue to Test 02.

If either command reported STALE:, remove only this suite's dedicated manual-test resources:

docker rm -f \
  ers334-b5e2bb92-backend \
  ers334-b5e2bb92-frontend \
  ers334-b5e2bb92-caddy \
  2>/dev/null || true

Then remove only this suite's dedicated volumes:

for volume in \
  ers334-b5e2bb92_backend-data \
  ers334-b5e2bb92_caddy-data \
  ers334-b5e2bb92_caddy-config
do
  docker volume rm "$volume" 2>/dev/null || true
done

Re-run the two namespace checks above.

Expected final results:

PASS: no stale Bite 33.4 manual-test containers are present
PASS: no stale Bite 33.4 manual-test Docker volumes are present

Confirm that any normal Development containers that existed before cleanup still exist:

docker ps -a --format '{{.Names}}' \
  | grep -E '^ers-dev-(backend|frontend|caddy)$' \
  || true

The output may still show the existing normal Development deployment. That is expected and proves the dedicated cleanup did not touch it.

PASS Criteria

PASS only when no ers334-b5e2bb92-* containers or ers334-b5e2bb92_* volumes remain before the disposable rehearsal begins.

Existing ers-dev-* containers are allowed and are not part of this test.

Promotion Blocker

Do not continue if the dedicated manual-test namespace cannot be made clean without touching non-manual-test Docker resources.

5. Test 02 — Locate and Verify the Existing Hetzner S3 Credential

Identity

No ERS login required.

Purpose

Reuse the already-proven ers-backup S3 credential without placing its secret in the ERS repository or this manual-test evidence.

Procedure

Run this exact credential-file discovery block:

unset ERS334_SOURCE_AWS_CREDENTIALS_FILE

if [[ -n "${AWS_SHARED_CREDENTIALS_FILE:-}" \
      && -f "$AWS_SHARED_CREDENTIALS_FILE" \
      && "$(grep -c '^\[ers-backup\]$' "$AWS_SHARED_CREDENTIALS_FILE")" -ge 1 ]]; then
  export ERS334_SOURCE_AWS_CREDENTIALS_FILE="$AWS_SHARED_CREDENTIALS_FILE"
elif [[ -f "$HOME/.aws/credentials" \
        && "$(grep -c '^\[ers-backup\]$' "$HOME/.aws/credentials")" -ge 1 ]]; then
  export ERS334_SOURCE_AWS_CREDENTIALS_FILE="$HOME/.aws/credentials"
elif [[ -f /tmp/ers-333-s3-manual/aws/credentials \
        && "$(grep -c '^\[ers-backup\]$' /tmp/ers-333-s3-manual/aws/credentials)" -ge 1 ]]; then
  export ERS334_SOURCE_AWS_CREDENTIALS_FILE="/tmp/ers-333-s3-manual/aws/credentials"
else
  echo "FAIL: no credentials file containing [ers-backup] was found"
fi

Verify:

if [ -n "${ERS334_SOURCE_AWS_CREDENTIALS_FILE:-}" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--n-"${ERS334_SOURCE_AWS_CREDENTIALS_FILE:-}"-); then
  printf 'Using credentials file: %s\n' "$ERS334_SOURCE_AWS_CREDENTIALS_FILE"
  echo "PASS: ers-backup credential profile located"
else
  echo "FAIL: ers-backup credential profile was not located"
fi

Expected:

PASS: ers-backup credential profile located

Create a dedicated temporary copy outside the disposable server tree:

rm -rf /tmp/ers-334-manual-secrets
mkdir -p /tmp/ers-334-manual-secrets
chmod 700 /tmp/ers-334-manual-secrets

cp \
  "$ERS334_SOURCE_AWS_CREDENTIALS_FILE" \
  /tmp/ers-334-manual-secrets/aws-credentials

chmod 600 /tmp/ers-334-manual-secrets/aws-credentials

export ERS334_AWS_CREDENTIALS_FILE="/tmp/ers-334-manual-secrets/aws-credentials"

Verify real Hetzner access:

AWS_SHARED_CREDENTIALS_FILE="$ERS334_AWS_CREDENTIALS_FILE" \
AWS_PROFILE=ers-backup \
AWS_REGION=hel1 \
aws s3api head-bucket \
  --bucket ers \
  --endpoint-url https://hel1.your-objectstorage.com

Immediately run:

echo $?

Expected:

PASS Criteria

PASS only if:

the ers-backup profile is found;
the temporary credentials file has mode 600;
head-bucket exits 0.

Promotion Blocker

Do not continue if the S3 credential cannot access bucket ers.

6. Test 03 — Build the Disposable Development Server Tree

Identity

No ERS login required.

Procedure

Define the disposable environment:

export ERS334_SERVER_ROOT="/tmp/ers-334-manual-server"
export ERS334_ENV_DIR="$ERS334_SERVER_ROOT/development"
export ERS334_S3_PREFIX="manual-334-$(date -u +%Y%m%dT%H%M%SZ)"
export ERS334_COMPOSE_PROJECT="ers334-b5e2bb92"
export ERS334_CONTAINER_PREFIX="ers334-b5e2bb92"

Start from a clean filesystem location:

rm -rf "$ERS334_SERVER_ROOT"
mkdir -p "$ERS334_ENV_DIR"

Export the exact current Git HEAD into the disposable server directory:

git archive HEAD | tar -x -C "$ERS334_ENV_DIR"

Confirm the exported source carries the expected 33.4 files:

test -f "$ERS334_ENV_DIR/backend/cmd/ers-restore/main.go" \
  && test -f "$ERS334_ENV_DIR/scripts/server-sqlite-restore.sh" \
  && echo "PASS: Bite 33.4 source exported into disposable server tree"

Expected:

PASS: Bite 33.4 source exported into disposable server tree

Initialize the Development server environment:

(
  cd "$ERS334_ENV_DIR"
  ./scripts/init-server-env.sh development
)

Define a small helper that updates one environment variable without creating duplicate active definitions:

set_ers334_env_value() {
  local file="$1"
  local key="$2"
  local value="$3"
  local tmp="${file}.tmp"

  grep -v "^${key}=" "$file" > "$tmp" || true
  printf '%s=%s\n' "$key" "$value" >> "$tmp"
  mv "$tmp" "$file"
}

Override the normal Development container prefix so Docker Compose cannot collide with an existing ers-dev deployment:

set_ers334_env_value   "$ERS334_ENV_DIR/.env.development"   CONTAINER_PREFIX   "$ERS334_CONTAINER_PREFIX"

Configure the disposable Development deployment to use the real Hetzner S3 bucket under the unique manual prefix:

set_ers334_env_value \
  "$ERS334_ENV_DIR/.env.development" \
  SERVER_OFFHOST_BACKUP_ENABLED \
  true

set_ers334_env_value \
  "$ERS334_ENV_DIR/.env.development" \
  SERVER_OFFHOST_BACKUP_TRANSPORT \
  s3

set_ers334_env_value \
  "$ERS334_ENV_DIR/.env.development" \
  SERVER_OFFHOST_BACKUP_S3_ENDPOINT \
  https://hel1.your-objectstorage.com

set_ers334_env_value \
  "$ERS334_ENV_DIR/.env.development" \
  SERVER_OFFHOST_BACKUP_S3_REGION \
  hel1

set_ers334_env_value \
  "$ERS334_ENV_DIR/.env.development" \
  SERVER_OFFHOST_BACKUP_S3_BUCKET \
  ers

set_ers334_env_value \
  "$ERS334_ENV_DIR/.env.development" \
  SERVER_OFFHOST_BACKUP_S3_PREFIX \
  "$ERS334_S3_PREFIX"

set_ers334_env_value \
  "$ERS334_ENV_DIR/.env.development" \
  AWS_SHARED_CREDENTIALS_FILE \
  "$ERS334_AWS_CREDENTIALS_FILE"

set_ers334_env_value \
  "$ERS334_ENV_DIR/.env.development" \
  AWS_PROFILE \
  ers-backup

Display only the non-secret S3 configuration:

grep -E \
  '^(APP_ENV|SERVER_OFFHOST_BACKUP_ENABLED|SERVER_OFFHOST_BACKUP_TRANSPORT|SERVER_OFFHOST_BACKUP_S3_ENDPOINT|SERVER_OFFHOST_BACKUP_S3_REGION|SERVER_OFFHOST_BACKUP_S3_BUCKET|SERVER_OFFHOST_BACKUP_S3_PREFIX|AWS_SHARED_CREDENTIALS_FILE|AWS_PROFILE)=' \
  "$ERS334_ENV_DIR/.env.development"

Expected values include:

APP_ENV=development
SERVER_OFFHOST_BACKUP_ENABLED=true
SERVER_OFFHOST_BACKUP_TRANSPORT=s3
SERVER_OFFHOST_BACKUP_S3_ENDPOINT=https://hel1.your-objectstorage.com
SERVER_OFFHOST_BACKUP_S3_REGION=hel1
SERVER_OFFHOST_BACKUP_S3_BUCKET=ers
AWS_PROFILE=ers-backup

Validate the server environment contract:

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-environment-contract-check \
  ENV=development

Expected exit code:

echo $?

Persist the non-secret session state:

cat > /tmp/ers-334-manual-session.env <<EOF
export ERS334_SERVER_ROOT='$ERS334_SERVER_ROOT'
export ERS334_ENV_DIR='$ERS334_ENV_DIR'
export ERS334_S3_PREFIX='$ERS334_S3_PREFIX'
export ERS334_AWS_CREDENTIALS_FILE='$ERS334_AWS_CREDENTIALS_FILE'
export ERS334_COMPOSE_PROJECT='$ERS334_COMPOSE_PROJECT'
export ERS334_CONTAINER_PREFIX='$ERS334_CONTAINER_PREFIX'
EOF

PASS Criteria

PASS only if:

the current HEAD was exported;
the Development environment was initialized;
S3 is enabled for the disposable deployment;
the Development environment contract passes.

7. Test 04 — Build, Create a Fresh Development Database, and Start the Disposable Deployment

Identity

No ERS login required.

Procedure

Reload session values if necessary:

source /tmp/ers-334-manual-session.env

Build the deployed images:

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-build \
  ENV=development

Expected: successful backend/frontend image build and exit 0.

Create a fresh disposable Development database from the full migration chain:

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-replace-development-db \
  ENV=development

Expected output includes:

Fresh Development database is ready.
Integrity check: ok
Foreign key check: clean

Start the disposable deployment:

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-up \
  ENV=development

Verify the backend:

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-backend-health \
  ENV=development

Expected:

ers334-b5e2bb92-backend health check passed.

Independently verify SQLite:

docker exec ers334-b5e2bb92-backend \
  sqlite3 /app/data/app.db 'PRAGMA integrity_check;'

Expected:

ok

Run:

docker exec ers334-b5e2bb92-backend \
  sqlite3 /app/data/app.db 'PRAGMA foreign_key_check;'

Expected: no output.

PASS Criteria

PASS only if:

the disposable Development database is built;
backend health passes;
integrity_check returns ok;
foreign_key_check returns no rows.

8. Test 05 — Create the Known-Good Source Backup and Off-Host Evidence

Identity

No ERS login required.

Purpose

This backup represents the database state to which the later restore must return.

Procedure

Load session values:

source /tmp/ers-334-manual-session.env

Create a verified backup:

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-backup \
  ENV=development

Because S3 is enabled, this command must also complete off-host replication and verification.

Locate the newly created managed backup:

export ERS334_SOURCE_BACKUP="$(
  find "$ERS334_ENV_DIR/backups" \
    -maxdepth 1 \
    -type f \
    -name 'app-*.db' \
    -print \
    | sort \
    | tail -1
)"

Confirm:

test -s "$ERS334_SOURCE_BACKUP" \
  && echo "PASS: source backup exists"

test -s "$ERS334_SOURCE_BACKUP.manifest.json" \
  && echo "PASS: source manifest exists"

test -s "$ERS334_SOURCE_BACKUP.offhost.json" \
  && echo "PASS: source off-host receipt exists"

Expected:

PASS: source backup exists
PASS: source manifest exists
PASS: source off-host receipt exists

Record the managed filename and SHA-256:

export ERS334_BACKUP_NAME="$(basename "$ERS334_SOURCE_BACKUP")"

export ERS334_SOURCE_SHA256="$(
  shasum -a 256 "$ERS334_SOURCE_BACKUP" | awk '{print $1}'
)"

printf 'Source backup: %s\n' "$ERS334_SOURCE_BACKUP"
printf 'Source SHA-256: %s\n' "$ERS334_SOURCE_SHA256"

Verify the local pair through the deployed operator interface:

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-backup-verify \
  ENV=development \
  BACKUP_FILE="$ERS334_SOURCE_BACKUP"

Expected output includes:

Backup verification passed

Inspect the S3 receipt:

jq -e '
  .status == "verified"
  and .environment == "development"
  and .transport == "s3"
  and .remote.bucket == "ers"
  and .remote.backup.version_id != ""
  and .remote.manifest.version_id != ""
' "$ERS334_SOURCE_BACKUP.offhost.json"

Expected exit code:

echo $?

Append the source evidence to the session file:

{
  printf "export ERS334_SOURCE_BACKUP='%s'\n" "$ERS334_SOURCE_BACKUP"
  printf "export ERS334_BACKUP_NAME='%s'\n" "$ERS334_BACKUP_NAME"
  printf "export ERS334_SOURCE_SHA256='%s'\n" "$ERS334_SOURCE_SHA256"
} >> /tmp/ers-334-manual-session.env

PASS Criteria

PASS only if local and S3 evidence both verify and exact S3 VersionIds are present.

9. Test 06 — Stage and Verify a Local Recovery Candidate

Identity

No ERS login required.

Procedure

Load state:

source /tmp/ers-334-manual-session.env

Stage the local Bite 33.2 pair:

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-restore-prepare-local \
  ENV=development \
  BACKUP_FILE="$ERS334_SOURCE_BACKUP"

Expected output includes:

Local recovery candidate staged and verified.
Recovery candidate directory:

Locate the staged backup:

export ERS334_LOCAL_CANDIDATE="$(
  find "$ERS334_ENV_DIR/recovery" \
    -type f \
    -name "$ERS334_BACKUP_NAME" \
    -print \
    | grep '/local-' \
    | sort \
    | tail -1
)"

Confirm:

test -s "$ERS334_LOCAL_CANDIDATE" \
  && test -s "$ERS334_LOCAL_CANDIDATE.manifest.json" \
  && echo "PASS: local recovery candidate pair exists"

Expected:

PASS: local recovery candidate pair exists

Verify its bytes match the source backup:

test "$(
  shasum -a 256 "$ERS334_LOCAL_CANDIDATE" | awk '{print $1}'
)" = "$ERS334_SOURCE_SHA256" \
  && echo "PASS: local recovery candidate is byte-for-byte identical to source backup"

Expected:

PASS: local recovery candidate is byte-for-byte identical to source backup

Run the operator verification target:

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-restore-verify \
  ENV=development \
  RECOVERY_BACKUP_FILE="$ERS334_LOCAL_CANDIDATE"

Expected output includes:

Restore candidate verification passed:
Environment: development

Persist:

printf "export ERS334_LOCAL_CANDIDATE='%s'\n" \
  "$ERS334_LOCAL_CANDIDATE" \
  >> /tmp/ers-334-manual-session.env

Negative check — environment mismatch

Run:

set +e

ERS_BACKUP_ENV_DIR="$ERS334_ENV_DIR" \
ERS_BACKUP_ENV_FILE=".env.development" \
ERS_BACKUP_COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
./scripts/run-backup-go-tool.sh ers-restore verify \
  --backup "$ERS334_LOCAL_CANDIDATE" \
  --manifest "$ERS334_LOCAL_CANDIDATE.manifest.json" \
  --expected-environment test \
  > /tmp/ers-334-environment-mismatch.out 2>&1

ERS334_ENV_MISMATCH_STATUS=$?

set -e

Display:

cat /tmp/ers-334-environment-mismatch.out
printf 'Exit status: %s\n' "$ERS334_ENV_MISMATCH_STATUS"

Expected output includes:

backup environment mismatch: expected test, manifest declares development

Assert:

test "$ERS334_ENV_MISMATCH_STATUS" -ne 0 \
  && echo "PASS: recovery candidate cannot cross environment identity"

Expected:

PASS: recovery candidate cannot cross environment identity

Negative check — candidate outside recovery tree

Run:

set +e

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-restore-verify \
  ENV=development \
  RECOVERY_BACKUP_FILE="$ERS334_SOURCE_BACKUP" \
  > /tmp/ers-334-outside-recovery.out 2>&1

ERS334_OUTSIDE_RECOVERY_STATUS=$?

set -e

Display:

cat /tmp/ers-334-outside-recovery.out

Expected output includes:

Refusing recovery candidate outside

Assert:

test "$ERS334_OUTSIDE_RECOVERY_STATUS" -ne 0 \
  && echo "PASS: restore verification refuses files outside the recovery tree"

Expected:

PASS: restore verification refuses files outside the recovery tree

10. Test 07 — Materialize the Exact S3 Recovery Candidate Using the Local Receipt

Identity

No ERS login required.

Procedure

Load state:

source /tmp/ers-334-manual-session.env

Wait two seconds so the timestamped materialization directory cannot collide with the earlier local staging directory:

sleep 2

Materialize from the local off-host receipt:

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-restore-materialize-offhost \
  ENV=development \
  RECEIPT_FILE="$ERS334_SOURCE_BACKUP.offhost.json"

Expected output includes:

Off-host recovery candidate materialized and verified.
Backup:
Manifest:
Receipt:
Recovery candidate directory:

Locate the newest off-host materialized database:

export ERS334_OFFHOST_RECEIPT_CANDIDATE="$(
  find "$ERS334_ENV_DIR/recovery" \
    -type f \
    -name "$ERS334_BACKUP_NAME" \
    -print \
    | grep '/offhost-' \
    | sort \
    | tail -1
)"

Verify:

test -s "$ERS334_OFFHOST_RECEIPT_CANDIDATE" \
  && test -s "$ERS334_OFFHOST_RECEIPT_CANDIDATE.manifest.json" \
  && test -s "$ERS334_OFFHOST_RECEIPT_CANDIDATE.offhost.json" \
  && echo "PASS: exact S3 recovery evidence materialized"

Expected:

PASS: exact S3 recovery evidence materialized

Compare SHA-256:

test "$(
  shasum -a 256 "$ERS334_OFFHOST_RECEIPT_CANDIDATE" | awk '{print $1}'
)" = "$ERS334_SOURCE_SHA256" \
  && echo "PASS: S3 materialized database exactly matches source backup"

Expected:

PASS: S3 materialized database exactly matches source backup

Re-run restore verification:

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-restore-verify \
  ENV=development \
  RECOVERY_BACKUP_FILE="$ERS334_OFFHOST_RECEIPT_CANDIDATE"

Expected:

Restore candidate verification passed:

PASS Criteria

The exact-version S3 recovery candidate must match the original source backup and pass the complete restore verification contract.

11. Test 08 — Recover from Hetzner When the Local Off-Host Receipt Is Unavailable

Identity

No ERS login required.

Purpose

Simulate loss of the application host's local receipt while retaining only the managed backup filename visible in Hetzner Object Storage.

Procedure

Load state:

source /tmp/ers-334-manual-session.env

Define a temporary hidden-receipt path:

export ERS334_HIDDEN_RECEIPT="/tmp/ers-334-hidden-$ERS334_BACKUP_NAME.offhost.json"
rm -f "$ERS334_HIDDEN_RECEIPT"

Move the local receipt away:

mv \
  "$ERS334_SOURCE_BACKUP.offhost.json" \
  "$ERS334_HIDDEN_RECEIPT"

Prove the local receipt is unavailable:

test ! -e "$ERS334_SOURCE_BACKUP.offhost.json" \
  && echo "PASS: local off-host receipt is unavailable"

Expected:

PASS: local off-host receipt is unavailable

Wait two seconds to guarantee a distinct materialization directory:

sleep 2

Run recovery by managed backup filename and capture the command's exit status without losing the hidden local receipt if the command fails:

set +e

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-restore-materialize-offhost \
  ENV=development \
  OFFHOST_BACKUP_NAME="$ERS334_BACKUP_NAME"

ERS334_BOOTSTRAP_STATUS=$?

mv \
  "$ERS334_HIDDEN_RECEIPT" \
  "$ERS334_SOURCE_BACKUP.offhost.json"

set -e

Assert success:

test "$ERS334_BOOTSTRAP_STATUS" -eq 0 \
  && echo "PASS: S3 recovery bootstrapped using only the managed backup filename"

Expected:

PASS: S3 recovery bootstrapped using only the managed backup filename

Confirm the original local receipt has been restored after the test:

test -s "$ERS334_SOURCE_BACKUP.offhost.json" \
  && echo "PASS: original local receipt restored after host-loss simulation"

Expected:

PASS: original local receipt restored after host-loss simulation

Locate the newest off-host candidate:

export ERS334_BOOTSTRAP_CANDIDATE="$(
  find "$ERS334_ENV_DIR/recovery" \
    -type f \
    -name "$ERS334_BACKUP_NAME" \
    -print \
    | grep '/offhost-' \
    | sort \
    | tail -1
)"

Verify all three recovered artifacts:

test -s "$ERS334_BOOTSTRAP_CANDIDATE" \
  && test -s "$ERS334_BOOTSTRAP_CANDIDATE.manifest.json" \
  && test -s "$ERS334_BOOTSTRAP_CANDIDATE.offhost.json" \
  && echo "PASS: S3 receipt bootstrap materialized DB, manifest, and receipt"

Expected:

PASS: S3 receipt bootstrap materialized DB, manifest, and receipt

Compare SHA-256:

test "$(
  shasum -a 256 "$ERS334_BOOTSTRAP_CANDIDATE" | awk '{print $1}'
)" = "$ERS334_SOURCE_SHA256" \
  && echo "PASS: host-loss materialized database matches original source SHA-256"

Expected:

PASS: host-loss materialized database matches original source SHA-256

Run full restore verification:

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-restore-verify \
  ENV=development \
  RECOVERY_BACKUP_FILE="$ERS334_BOOTSTRAP_CANDIDATE"

Expected:

Restore candidate verification passed:

PASS Criteria

PASS only if recovery succeeds without the local receipt and the materialized database still matches the original source SHA-256.

Promotion Blocker

If loss of the local receipt prevents S3 recovery, Bite 33.4 must not be promoted.

12. Test 09 — Prove Restore Safety Guards Do Not Touch the Live Database

Identity

No ERS login required.

Procedure

Load state:

source /tmp/ers-334-manual-session.env

Record the current live database SHA-256:

export ERS334_LIVE_SHA_BEFORE_GUARDS="$(
  docker exec ers334-b5e2bb92-backend \
    sha256sum /app/data/app.db \
    | awk '{print $1}'
)"

printf 'Live DB SHA before negative tests: %s\n' \
  "$ERS334_LIVE_SHA_BEFORE_GUARDS"

Negative check A — wrong confirmation token

Run:

set +e

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-restore-apply \
  ENV=development \
  RECOVERY_BACKUP_FILE="$ERS334_LOCAL_CANDIDATE" \
  RESTORE_CONFIRM=RESTORE-PRODUCTION \
  > /tmp/ers-334-wrong-confirm.out 2>&1

ERS334_WRONG_CONFIRM_STATUS=$?

set -e

Display:

cat /tmp/ers-334-wrong-confirm.out

Expected output includes:

Restore confirmation mismatch: expected RESTORE-DEVELOPMENT

Assert:

test "$ERS334_WRONG_CONFIRM_STATUS" -ne 0 \
  && echo "PASS: wrong restore confirmation token rejected"

Expected:

PASS: wrong restore confirmation token rejected

Negative check B — source backup outside recovery tree

Run:

set +e

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-restore-apply \
  ENV=development \
  RECOVERY_BACKUP_FILE="$ERS334_SOURCE_BACKUP" \
  RESTORE_CONFIRM=RESTORE-DEVELOPMENT \
  > /tmp/ers-334-outside-apply.out 2>&1

ERS334_OUTSIDE_APPLY_STATUS=$?

set -e

Display:

cat /tmp/ers-334-outside-apply.out

Expected output includes:

Refusing recovery candidate outside

Assert:

test "$ERS334_OUTSIDE_APPLY_STATUS" -ne 0 \
  && echo "PASS: restore apply refuses a source outside the recovery tree"

Expected:

PASS: restore apply refuses a source outside the recovery tree

Confirm backend is still running:

docker ps --format '{{.Names}}' \
  | grep -qx 'ers334-b5e2bb92-backend' \
  && echo "PASS: backend remained running after pre-destructive refusals"

Expected:

PASS: backend remained running after pre-destructive refusals

Recompute the live SHA:

ERS334_LIVE_SHA_AFTER_GUARDS="$(
  docker exec ers334-b5e2bb92-backend \
    sha256sum /app/data/app.db \
    | awk '{print $1}'
)"

Compare:

test "$ERS334_LIVE_SHA_AFTER_GUARDS" = "$ERS334_LIVE_SHA_BEFORE_GUARDS" \
  && echo "PASS: safety-guard failures did not modify the live database"

Expected:

PASS: safety-guard failures did not modify the live database

PASS Criteria

Both unsafe requests fail non-zero, the backend remains running, and the live DB SHA-256 is unchanged.

13. Test 10 — Create a Deliberate Post-Backup Database Probe

Identity

No ERS login required.

Purpose

The source backup from Test 05 must not contain this probe. The live database will contain it immediately before restoration. Successful restoration to the source backup must therefore remove it.

Procedure

Load state:

source /tmp/ers-334-manual-session.env

Confirm the source backup does not contain the probe table:

ERS334_SOURCE_PROBE_TABLE_COUNT="$(
  sqlite3 "$ERS334_SOURCE_BACKUP" \
    "SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name='bite334_manual_restore_probe';"
)"

printf 'Probe tables in source backup: %s\n' \
  "$ERS334_SOURCE_PROBE_TABLE_COUNT"

Expected:

Probe tables in source backup: 0

Create the probe in the live disposable Development database:

docker exec ers334-b5e2bb92-backend \
  sqlite3 /app/data/app.db "
    CREATE TABLE bite334_manual_restore_probe (
      id TEXT PRIMARY KEY,
      value TEXT NOT NULL
    );
    INSERT INTO bite334_manual_restore_probe(id, value)
    VALUES ('probe-1', 'created-after-source-backup');
  "

Confirm:

docker exec ers334-b5e2bb92-backend \
  sqlite3 /app/data/app.db \
  "SELECT id || ':' || value FROM bite334_manual_restore_probe;"

Expected exactly:

probe-1:created-after-source-backup

Confirm health remains good:

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-backend-health \
  ENV=development

Record the live database SHA after adding the probe:

export ERS334_LIVE_SHA_WITH_PROBE="$(
  docker exec ers334-b5e2bb92-backend \
    sha256sum /app/data/app.db \
    | awk '{print $1}'
)"

printf 'Live DB SHA with probe: %s\n' \
  "$ERS334_LIVE_SHA_WITH_PROBE"

Confirm it is no longer the same byte stream as the source backup:

test "$ERS334_LIVE_SHA_WITH_PROBE" != "$ERS334_SOURCE_SHA256" \
  && echo "PASS: live database differs from the earlier source backup"

Expected:

PASS: live database differs from the earlier source backup

Record the current number of managed local backups:

export ERS334_BACKUP_COUNT_BEFORE_APPLY="$(
  find "$ERS334_ENV_DIR/backups" \
    -maxdepth 1 \
    -type f \
    -name 'app-*.db' \
    | wc -l \
    | tr -d ' '
)"

printf 'Managed backups before restore apply: %s\n' \
  "$ERS334_BACKUP_COUNT_BEFORE_APPLY"

printf "export ERS334_BACKUP_COUNT_BEFORE_APPLY='%s'\n" \
  "$ERS334_BACKUP_COUNT_BEFORE_APPLY" \
  >> /tmp/ers-334-manual-session.env

PASS Criteria

PASS only if the probe exists in the live DB, does not exist in the source backup, and the live DB SHA differs from the source backup SHA.

14. Test 11 — Apply the Verified Recovery Candidate to the Disposable Deployment

Identity

No ERS login required.

Warning

This is the deliberate destructive restore test, but it operates only on the disposable Development Docker volume created by this suite.

Procedure

Load state:

source /tmp/ers-334-manual-session.env

Run the restore:

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-restore-apply \
  ENV=development \
  RECOVERY_BACKUP_FILE="$ERS334_LOCAL_CANDIDATE" \
  RESTORE_CONFIRM=RESTORE-DEVELOPMENT

Expected output includes all of these logical events:

Restore candidate verification passed:
Creating a fresh verified pre-restore backup of the running development database.
Stopping ers334-b5e2bb92-backend before SQLite replacement.
Restore applied and verified: /app/data/app.db
Restore completed and post-restart database verification passed.
Recovery candidate:
Pre-restore database copy:
Restore verification report:

Immediately run:

echo $?

Expected:

Confirm backend health:

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-backend-health \
  ENV=development

Expected:

ers334-b5e2bb92-backend health check passed.

PASS Criteria

PASS only if the restore exits 0, the backend restarts, /healthz passes, and post-restart SQLite verification passes.

15. Test 12 — Prove the Exact Earlier Database Was Restored

Identity

No ERS login required.

Procedure

Load:

source /tmp/ers-334-manual-session.env

Confirm the post-backup probe table is gone:

ERS334_LIVE_PROBE_TABLE_COUNT="$(
  docker exec ers334-b5e2bb92-backend \
    sqlite3 /app/data/app.db \
    "SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name='bite334_manual_restore_probe';"
)"

printf 'Probe tables after restore: %s\n' \
  "$ERS334_LIVE_PROBE_TABLE_COUNT"

Expected:

Probe tables after restore: 0

Compute the restored live database SHA:

export ERS334_RESTORED_LIVE_SHA="$(
  docker exec ers334-b5e2bb92-backend \
    sha256sum /app/data/app.db \
    | awk '{print $1}'
)"

Compare it with the source backup:

printf 'Source SHA:        %s\n' "$ERS334_SOURCE_SHA256"
printf 'Restored live SHA: %s\n' "$ERS334_RESTORED_LIVE_SHA"

test "$ERS334_RESTORED_LIVE_SHA" = "$ERS334_SOURCE_SHA256" \
  && echo "PASS: restored live database exactly matches selected source backup"

printf "export ERS334_RESTORED_LIVE_SHA='%s'\n" \
  "$ERS334_RESTORED_LIVE_SHA" \
  >> /tmp/ers-334-manual-session.env

Expected:

PASS: restored live database exactly matches selected source backup

Run SQLite checks again:

docker exec ers334-b5e2bb92-backend \
  sqlite3 /app/data/app.db 'PRAGMA integrity_check;'

Expected:

ok

Run:

docker exec ers334-b5e2bb92-backend \
  sqlite3 /app/data/app.db 'PRAGMA foreign_key_check;'

Expected: no output.

PASS Criteria

The probe is absent, restored live SHA equals the original selected backup SHA, integrity is ok, and foreign-key check is empty.

16. Test 13 — Verify the Fresh Pre-Restore Backup and Preserved Pre-Restore Evidence

Identity

No ERS login required.

Purpose

Prove that the database state being replaced was protected before restoration.

Critical evidence rule: never open the original preserved recovery/pre-restore/app-before-*.db directly with ordinary sqlite3. SQLite may perform WAL/journal recovery or checkpoint work and change the preserved .db bytes. The restore report's SHA-256 must continue to describe the exact preserved evidence.

Procedure

Load:

source /tmp/ers-334-manual-session.env

Count managed backups after restore:

ERS334_BACKUP_COUNT_AFTER_APPLY="$(
  find "$ERS334_ENV_DIR/backups" \
    -maxdepth 1 \
    -type f \
    -name 'app-*.db' \
    | wc -l \
    | tr -d ' '
)"

printf 'Managed backups before apply: %s\n' "$ERS334_BACKUP_COUNT_BEFORE_APPLY"
printf 'Managed backups after apply:  %s\n' "$ERS334_BACKUP_COUNT_AFTER_APPLY"

Assert one new managed backup was created:

test "$ERS334_BACKUP_COUNT_AFTER_APPLY" -eq "$((ERS334_BACKUP_COUNT_BEFORE_APPLY + 1))" \
  && echo "PASS: restore workflow created one fresh pre-restore managed backup" \
  || echo "FAIL: restore workflow did not create exactly one fresh pre-restore managed backup"

Expected:

PASS: restore workflow created one fresh pre-restore managed backup

Locate the newest managed backup:

export ERS334_PRERESTORE_BACKUP="$(
  find "$ERS334_ENV_DIR/backups" \
    -maxdepth 1 \
    -type f \
    -name 'app-*.db' \
    -print \
    | sort \
    | tail -1
)"

printf 'Fresh pre-restore backup: %s\n' \
  "$ERS334_PRERESTORE_BACKUP"

printf "export ERS334_PRERESTORE_BACKUP='%s'\n" \
  "$ERS334_PRERESTORE_BACKUP" \
  >> /tmp/ers-334-manual-session.env

Confirm it is distinct from the selected restore source:

test "$ERS334_PRERESTORE_BACKUP" != "$ERS334_SOURCE_BACKUP" \
  && echo "PASS: fresh pre-restore backup is distinct from selected restore source" \
  || echo "FAIL: fresh pre-restore backup is not distinct from selected restore source"

Expected:

PASS: fresh pre-restore backup is distinct from selected restore source

Inspect the managed backup, not the raw preserved target, for the deliberate probe:

sqlite3 \
  "file:$ERS334_PRERESTORE_BACKUP?mode=ro&immutable=1" \
  "SELECT id || ':' || value FROM bite334_manual_restore_probe;"

Expected:

probe-1:created-after-source-backup

Confirm its manifest and off-host receipt exist:

test -s "$ERS334_PRERESTORE_BACKUP.manifest.json" \
  && echo "PASS: pre-restore backup manifest exists" \
  || echo "FAIL: pre-restore backup manifest is missing"

test -s "$ERS334_PRERESTORE_BACKUP.offhost.json" \
  && echo "PASS: pre-restore off-host receipt exists" \
  || echo "FAIL: pre-restore off-host receipt is missing"

Expected:

PASS: pre-restore backup manifest exists
PASS: pre-restore off-host receipt exists

Verify the off-host receipt captured exact S3 versions:

if jq -e '
  .status == "verified"
  and .transport == "s3"
  and .remote.backup.version_id != ""
  and .remote.manifest.version_id != ""
' "$ERS334_PRERESTORE_BACKUP.offhost.json" >/dev/null
then
  echo "PASS: pre-restore off-host receipt records exact S3 versions"
else
  echo "FAIL: pre-restore off-host receipt does not record exact S3 versions"
fi

Expected:

PASS: pre-restore off-host receipt records exact S3 versions

Locate the verified restore report whose source SHA matches the selected source backup:

unset ERS334_RESTORE_REPORT

for candidate in "$ERS334_ENV_DIR"/recovery/reports/restore-*.json; do
  [ -f "$candidate" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--f-"$candidate"-) || continue

  if jq -e \
    --arg source_sha "$ERS334_SOURCE_SHA256" \
    '
      .status == "verified"
      and .environment == "development"
      and .source.backup_sha256 == $source_sha
    ' \
    "$candidate" \
    >/dev/null
  then
    export ERS334_RESTORE_REPORT="$candidate"
  fi
done

if [ -n "${ERS334_RESTORE_REPORT:-}" && -s "$ERS334_RESTORE_REPORT" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--n-"${ERS334_RESTORE_REPORT:-}"-&&--s-"$ERS334_RESTORE_REPORT"-); then
  printf 'Restore report: %s\n' "$ERS334_RESTORE_REPORT"
  echo "PASS: matching verified restore report located"
else
  echo "FAIL: matching verified restore report was not found"
fi

Read the exact raw pre-restore copy path from that report:

export ERS334_PREVIOUS_COPY="$(
  jq -r '.previous_target.copy // empty' "$ERS334_RESTORE_REPORT"
)"

if [ -n "$ERS334_PREVIOUS_COPY" && -s "$ERS334_PREVIOUS_COPY" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--n-"$ERS334_PREVIOUS_COPY"-&&--s-"$ERS334_PREVIOUS_COPY"-); then
  printf 'Report-linked raw pre-restore DB: %s\n' "$ERS334_PREVIOUS_COPY"
  echo "PASS: report-linked raw pre-restore database exists"
else
  echo "FAIL: report-linked raw pre-restore database is missing"
fi

Before opening that evidence with any SQLite command, compute its SHA:

export ERS334_PREVIOUS_COPY_SHA="$(
  shasum -a 256 "$ERS334_PREVIOUS_COPY" | awk '{print $1}'
)"

export ERS334_REPORT_PREVIOUS_SHA="$(
  jq -r '.previous_target.sha256 // empty' "$ERS334_RESTORE_REPORT"
)"

printf 'Report previous SHA: %s\n' "$ERS334_REPORT_PREVIOUS_SHA"
printf 'File previous SHA:   %s\n' "$ERS334_PREVIOUS_COPY_SHA"

if [ "$ERS334_PREVIOUS_COPY_SHA" == "$ERS334_REPORT_PREVIOUS_SHA" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$ERS334_PREVIOUS_COPY_SHA"-==-"$ERS334_REPORT_PREVIOUS_SHA"-); then
  echo "PASS: untouched raw pre-restore database SHA-256 matches report evidence"
else
  echo "FAIL: untouched raw pre-restore database SHA-256 does not match report evidence"
fi

Expected:

PASS: untouched raw pre-restore database SHA-256 matches report evidence

Persist the authoritative evidence paths:

{
  printf "export ERS334_RESTORE_REPORT='%s'\n" "$ERS334_RESTORE_REPORT"
  printf "export ERS334_PREVIOUS_COPY='%s'\n" "$ERS334_PREVIOUS_COPY"
} >> /tmp/ers-334-manual-session.env

Inspect the raw pre-restore logical state without mutating the evidence

Create a disposable inspection directory:

export ERS334_PREVIOUS_INSPECTION_DIR="/tmp/ers-334-previous-inspection"

rm -rf "$ERS334_PREVIOUS_INSPECTION_DIR"
mkdir -p "$ERS334_PREVIOUS_INSPECTION_DIR"

export ERS334_INSPECTION_DB="$ERS334_PREVIOUS_INSPECTION_DIR/$(basename "$ERS334_PREVIOUS_COPY")"

cp "$ERS334_PREVIOUS_COPY" "$ERS334_INSPECTION_DB"

for suffix in -wal -shm -journal; do
  if [ -f "$ERS334_PREVIOUS_COPY$suffix" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--f-"$ERS334_PREVIOUS_COPY$suffix"-); then
    cp \
      "$ERS334_PREVIOUS_COPY$suffix" \
      "$ERS334_INSPECTION_DB$suffix"
  fi
done

Open only the disposable clone:

sqlite3 "$ERS334_INSPECTION_DB" \
  "SELECT id || ':' || value FROM bite334_manual_restore_probe;"

Expected:

probe-1:created-after-source-backup

Remove the disposable inspection copy:

rm -rf "$ERS334_PREVIOUS_INSPECTION_DIR"

Recalculate the SHA of the untouched original evidence:

ERS334_PREVIOUS_COPY_SHA_AFTER_INSPECTION="$(
  shasum -a 256 "$ERS334_PREVIOUS_COPY" | awk '{print $1}'
)"

if [ "$ERS334_PREVIOUS_COPY_SHA_AFTER_INSPECTION" == "$ERS334_REPORT_PREVIOUS_SHA" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$ERS334_PREVIOUS_COPY_SHA_AFTER_INSPECTION"-==-"$ERS334_REPORT_PREVIOUS_SHA"-); then
  echo "PASS: raw pre-restore evidence remained unchanged during logical inspection"
else
  echo "FAIL: raw pre-restore evidence changed during logical inspection"
fi

Expected:

PASS: raw pre-restore evidence remained unchanged during logical inspection

PASS Criteria

PASS only if:

exactly one fresh managed pre-restore backup was created;
that managed backup contains the state that was replaced;
its manifest and exact-version S3 receipt exist;
the restore report's exact raw pre-restore file exists;
the untouched raw file's SHA-256 matches the report;
logical inspection is performed only on a disposable clone;
the original raw evidence SHA remains unchanged after inspection.

17. Test 14 — Inspect the Structured Restore Verification Report

Identity

No ERS login required.

Procedure

Load persisted state:

source /tmp/ers-334-manual-session.env

Confirm the report selected in Test 13 still exists:

if [ -n "${ERS334_RESTORE_REPORT:-}" && -s "$ERS334_RESTORE_REPORT" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--n-"${ERS334_RESTORE_REPORT:-}"-&&--s-"$ERS334_RESTORE_REPORT"-); then
  printf 'Restore report: %s\n' "$ERS334_RESTORE_REPORT"
  echo "PASS: restore verification report located"
else
  echo "FAIL: restore verification report was not found"
fi

Expected:

PASS: restore verification report located

Validate JSON:

if jq -e . "$ERS334_RESTORE_REPORT" >/dev/null; then
  echo "PASS: restore report is valid JSON"
else
  echo "FAIL: restore report is not valid JSON"
fi

Expected:

PASS: restore report is valid JSON

Re-read the exact preserved pre-restore copy path from that report:

export ERS334_PREVIOUS_COPY="$(
  jq -r '.previous_target.copy // empty' "$ERS334_RESTORE_REPORT"
)"

if [ -n "$ERS334_PREVIOUS_COPY" && -s "$ERS334_PREVIOUS_COPY" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--n-"$ERS334_PREVIOUS_COPY"-&&--s-"$ERS334_PREVIOUS_COPY"-); then
  printf 'Report-linked pre-restore DB: %s\n' "$ERS334_PREVIOUS_COPY"
  echo "PASS: report-linked pre-restore database copy exists"
else
  echo "FAIL: report-linked pre-restore database copy is missing"
fi

Expected:

PASS: report-linked pre-restore database copy exists

Calculate its current SHA-256 without opening it with SQLite:

export ERS334_PREVIOUS_COPY_SHA="$(
  shasum -a 256 "$ERS334_PREVIOUS_COPY" | awk '{print $1}'
)"

Assert the complete report evidence:

if jq -e \
  --arg source_sha "$ERS334_SOURCE_SHA256" \
  --arg previous_sha "$ERS334_PREVIOUS_COPY_SHA" \
  '
    .status == "verified"
    and .environment == "development"
    and .source.backup_sha256 == $source_sha
    and .previous_target.existed == true
    and .previous_target.sha256 == $previous_sha
    and .restored_target.path == "/app/data/app.db"
    and .restored_target.sha256 == $source_sha
    and .restored_target.integrity_check == "ok"
    and .restored_target.foreign_key_check == "ok"
    and .restored_target.schema_migration_count > 0
    and (.restored_target.latest_schema_migration | length) > 0
  ' \
  "$ERS334_RESTORE_REPORT" \
  >/dev/null
then
  echo "PASS: restore report core recovery evidence is correct"
else
  echo "FAIL: restore report core recovery evidence is incorrect"
fi

Expected:

PASS: restore report core recovery evidence is correct

Check the restored-target SHA explicitly:

ERS334_REPORT_RESTORED_SHA="$(
  jq -r '.restored_target.sha256' "$ERS334_RESTORE_REPORT"
)"

if [ "$ERS334_REPORT_RESTORED_SHA" == "$ERS334_SOURCE_SHA256" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$ERS334_REPORT_RESTORED_SHA"-==-"$ERS334_SOURCE_SHA256"-); then
  echo "PASS: report restored-target SHA-256 matches selected source backup"
else
  echo "FAIL: report restored-target SHA-256 does not match selected source backup"
fi

Expected:

PASS: report restored-target SHA-256 matches selected source backup

Check for forbidden secret material:

if grep -Eiq \
  'aws_secret_access_key|PRIVATE KEY|JWT_SECRET|LEDGER_CORRECTION_KEY|LEDGER_SETTLEMENT_KEY|OPENAI_API_KEY' \
  "$ERS334_RESTORE_REPORT"; then
  echo "FAIL: restore report appears to contain secret material"
else
  echo "PASS: restore report contains no obvious secret material"
fi

Expected:

PASS: restore report contains no obvious secret material

Re-verify the selected recovery candidate:

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-restore-verify \
  ENV=development \
  RECOVERY_BACKUP_FILE="$ERS334_LOCAL_CANDIDATE"

Expected output includes:

Restore candidate verification passed:

Immediately run:

echo $?

Expected:

PASS Criteria

Test 14 passes only if every explicit assertion above prints PASS: and none prints FAIL:.

18. Test 15 — Prove Fail-Closed Behavior After Entering the Destructive Recovery Phase

Identity

No ERS login required.

Purpose

Force a deterministic failure after the backend has been stopped but before the database is replaced by giving the Go restore tool a report path that already exists.

On macOS, /tmp normally resolves to /private/tmp. The deployed Go runner mounts the canonical environment directory into its one-off container. Therefore this test must use the canonical physical environment path for the forced report file. Using /tmp/... while the mounted environment is /private/tmp/... would create two different paths inside the container and would fail to induce the intended refusal.

Expected safe result:

restore command fails non-zero;
live database is unchanged;
backend remains stopped;
operator must explicitly restart it.

Procedure

Load persisted state:

source /tmp/ers-334-manual-session.env

Canonicalize the disposable environment path:

export ERS334_CANONICAL_ENV_DIR="$(
  cd "$ERS334_ENV_DIR" && pwd -P
)"

printf 'Logical env path:   %s\n' "$ERS334_ENV_DIR"
printf 'Canonical env path: %s\n' "$ERS334_CANONICAL_ENV_DIR"

On macOS the output normally resembles:

Logical env path:   /tmp/ers-334-manual-server/development
Canonical env path: /private/tmp/ers-334-manual-server/development

Confirm the backend is currently running before beginning the failure test:

if docker ps --format '{{.Names}}' \
  | grep -qx "${ERS334_CONTAINER_PREFIX}-backend"; then
  echo "PASS: backend is running before forced-failure test"
else
  echo "FAIL: backend is not running before forced-failure test"
fi

Expected:

PASS: backend is running before forced-failure test

Record the current live database SHA-256. This value must be captured now, even if an earlier attempt at Test 15 accidentally completed a successful restore:

export ERS334_LIVE_SHA_BEFORE_FORCED_FAILURE="$(
  docker exec "${ERS334_CONTAINER_PREFIX}-backend" \
    sha256sum /app/data/app.db \
    | awk '{print $1}'
)"

printf 'Live DB SHA before corrected forced-failure test: %s\n' \
  "$ERS334_LIVE_SHA_BEFORE_FORCED_FAILURE"

Create an intentionally occupied report path under the canonical mounted environment directory:

export ERS334_FORCED_REPORT="$ERS334_CANONICAL_ENV_DIR/recovery/reports/forced-existing-report.json"

rm -f "$ERS334_FORCED_REPORT"

printf '{"purpose":"force Bite 33.4 report overwrite refusal"}\n' \
  > "$ERS334_FORCED_REPORT"

Confirm the file exists at the exact path that will be passed to the deployed Go tool:

if [ -s "$ERS334_FORCED_REPORT" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--s-"$ERS334_FORCED_REPORT"-); then
  printf 'Forced existing report: %s\n' "$ERS334_FORCED_REPORT"
  echo "PASS: forced report exists on the canonical mounted path"
else
  echo "FAIL: forced report was not created"
fi

Expected:

PASS: forced report exists on the canonical mounted path

Run the server restore orchestration directly. Use the canonical environment directory consistently:

set +e

RESTORE_ENVIRONMENT=development \
RESTORE_ENV_DIR="$ERS334_CANONICAL_ENV_DIR" \
RESTORE_ENV_FILE=".env.development" \
RESTORE_COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
RESTORE_CONTAINER="${ERS334_CONTAINER_PREFIX}-backend" \
RESTORE_BACKUP_FILE="$ERS334_LOCAL_CANDIDATE" \
RESTORE_CONFIRM="RESTORE-DEVELOPMENT" \
RESTORE_REPORT_FILE="$ERS334_FORCED_REPORT" \
./scripts/server-sqlite-restore.sh \
  > /tmp/ers-334-forced-failure.out 2>&1

ERS334_FORCED_FAILURE_STATUS=$?

set -e

Display the complete result:

cat /tmp/ers-334-forced-failure.out
printf 'Exit status: %s\n' "$ERS334_FORCED_FAILURE_STATUS"

Expected output includes:

Stopping ers334-b5e2bb92-backend before SQLite replacement.
Restore error: refusing to overwrite existing restore report:
Restore failed after entering the destructive recovery phase.

The exact Go error prefix may differ slightly, but the output must include:

refusing to overwrite existing restore report

and:

Restore failed after entering the destructive recovery phase.

Assert the non-zero status:

if [ "$ERS334_FORCED_FAILURE_STATUS" -ne 0 ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$ERS334_FORCED_FAILURE_STATUS"--ne-0-); then
  echo "PASS: deliberately induced destructive-phase failure returned non-zero"
else
  echo "FAIL: deliberately induced destructive-phase failure unexpectedly returned zero"
fi

Expected:

PASS: deliberately induced destructive-phase failure returned non-zero

Confirm the backend is stopped:

if docker ps --format '{{.Names}}' \
  | grep -qx "${ERS334_CONTAINER_PREFIX}-backend"; then
  echo "FAIL: backend is still running after destructive-phase restore failure"
else
  echo "PASS: backend was left stopped after destructive-phase restore failure"
fi

Expected:

PASS: backend was left stopped after destructive-phase restore failure

While the normal backend is stopped, calculate the database SHA through a one-off container sharing the same Development volume:

export ERS334_LIVE_SHA_AFTER_FORCED_FAILURE="$(
  cd "$ERS334_CANONICAL_ENV_DIR" \
  && env \
      -u APP_ENV \
      -u APP_AUTO_MIGRATE \
      -u AUTHZ_ACTOR_HEADER_MODE \
      -u DEV_SEED_ADMIN \
      AUTHZ_BOOTSTRAP_ENABLED=false \
      docker compose \
        -p "$ERS334_COMPOSE_PROJECT" \
        --env-file .env.development \
        -f docker-compose.server.yml \
        run --rm --no-deps \
        --entrypoint sha256sum \
        backend \
        /app/data/app.db \
      | awk '{print $1}'
)"

Compare with the SHA captured immediately before this corrected forced-failure attempt:

printf 'DB SHA before forced failure: %s\n' \
  "$ERS334_LIVE_SHA_BEFORE_FORCED_FAILURE"

printf 'DB SHA after forced failure:  %s\n' \
  "$ERS334_LIVE_SHA_AFTER_FORCED_FAILURE"

if [ "$ERS334_LIVE_SHA_AFTER_FORCED_FAILURE" == "$ERS334_LIVE_SHA_BEFORE_FORCED_FAILURE" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$ERS334_LIVE_SHA_AFTER_FORCED_FAILURE"-==-"$ERS334_LIVE_SHA_BEFORE_FORCED_FAILURE"-); then
  echo "PASS: forced failure did not change the live database"
else
  echo "FAIL: live database changed during forced failure"
fi

Expected:

PASS: forced failure did not change the live database

Confirm that the deliberately occupied host report was not overwritten:

if [ "$(cat "$ERS334_FORCED_REPORT")" == '{"purpose":"force Bite 33.4 report overwrite refusal"}' ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$(cat-"$ERS334_FORCED_REPORT")"-==-'{"purpose":"force-Bite-33.4-report-overwrite-refusal"}'-); then
  echo "PASS: existing report file was not overwritten"
else
  echo "FAIL: existing report file was modified"
fi

Expected:

PASS: existing report file was not overwritten

Remove only the intentionally occupied report file:

rm -f "$ERS334_FORCED_REPORT"

Explicitly restart the disposable Development deployment:

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-up \
  ENV=development

Verify backend health:

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-backend-health \
  ENV=development

Expected output includes:

health check passed

After the explicit restart, do not require the SQLite file SHA-256 to remain identical to the stopped-state SHA.

ERS opens SQLite using journal_mode(WAL) and performs normal application startup maintenance. Those operations can legitimately change the bytes of /app/data/app.db after the backend is restarted even though the logical database remains valid. The byte-for-byte "no modification" proof was already performed while the backend was stopped, before this restart.

For evidence only, record the post-restart SHA:

export ERS334_LIVE_SHA_AFTER_RECOVERY_RESTART="$(
  docker exec "${ERS334_CONTAINER_PREFIX}-backend" \
    sha256sum /app/data/app.db \
    | awk '{print $1}'
)"

printf 'DB SHA before forced failure: %s\n' \
  "$ERS334_LIVE_SHA_BEFORE_FORCED_FAILURE"

printf 'DB SHA after explicit restart: %s\n' \
  "$ERS334_LIVE_SHA_AFTER_RECOVERY_RESTART"

echo "INFO: post-restart SHA equality is not a PASS criterion because normal ERS startup can change SQLite file bytes"

Expected final informational line:

INFO: post-restart SHA equality is not a PASS criterion because normal ERS startup can change SQLite file bytes

Now run the same post-restart database checks used by the restore orchestration itself.

Integrity check:

ERS334_POST_RESTART_INTEGRITY="$(
  docker exec "${ERS334_CONTAINER_PREFIX}-backend" \
    sqlite3 /app/data/app.db 'PRAGMA integrity_check;'
)"

if [ "$ERS334_POST_RESTART_INTEGRITY" == "ok" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$ERS334_POST_RESTART_INTEGRITY"-==-"ok"-); then
  echo "PASS: post-restart database integrity_check is ok"
else
  printf 'FAIL: post-restart integrity_check returned: %s\n' \
    "$ERS334_POST_RESTART_INTEGRITY"
fi

Expected:

PASS: post-restart database integrity_check is ok

Foreign-key check:

ERS334_POST_RESTART_FOREIGN_KEYS="$(
  docker exec "${ERS334_CONTAINER_PREFIX}-backend" \
    sqlite3 /app/data/app.db 'PRAGMA foreign_key_check;'
)"

if [ -z "$ERS334_POST_RESTART_FOREIGN_KEYS" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--z-"$ERS334_POST_RESTART_FOREIGN_KEYS"-); then
  echo "PASS: post-restart database foreign_key_check is clean"
else
  echo "FAIL: post-restart database foreign_key_check returned violations:"
  printf '%s\n' "$ERS334_POST_RESTART_FOREIGN_KEYS"
fi

Expected:

PASS: post-restart database foreign_key_check is clean

Confirm the expected migration chain is still present:

ERS334_POST_RESTART_MIGRATION_COUNT="$(
  docker exec "${ERS334_CONTAINER_PREFIX}-backend" \
    sqlite3 /app/data/app.db \
    'SELECT COUNT(*) FROM schema_migrations;'
)"

printf 'Post-restart schema migration count: %s\n' \
  "$ERS334_POST_RESTART_MIGRATION_COUNT"

if [ "$ERS334_POST_RESTART_MIGRATION_COUNT" -gt 0 ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$ERS334_POST_RESTART_MIGRATION_COUNT"--gt-0-); then
  echo "PASS: post-restart schema migration evidence remains present"
else
  echo "FAIL: post-restart schema migration evidence is missing"
fi

Expected:

PASS: post-restart schema migration evidence remains present

PASS Criteria

PASS only if:

the forced report exists on the canonical mounted environment path;
the induced restore fails non-zero;
output explicitly reports refusal to overwrite the existing report;
backend remains stopped;
while the backend is stopped, live DB SHA-256 exactly matches the SHA captured immediately before the forced failure;
the occupied report itself is not overwritten;
backend starts only after the explicit operator restart;
health recovers after the explicit restart;
post-restart PRAGMA integrity_check returns ok;
post-restart PRAGMA foreign_key_check returns no rows;
post-restart schema-migration evidence remains present.

A post-restart file-level SHA change is not a failure because normal ERS SQLite startup may legitimately alter database bytes.

Promotion Blocker

Do not promote if:

the canonical existing report is overwritten;
the restore exits 0;
the backend remains running after the induced destructive-phase failure;
the database SHA changes while the backend is still stopped;
the backend cannot recover after explicit restart;
post-restart integrity fails;
post-restart foreign-key violations exist;
or schema-migration evidence is missing after restart.

19. Test 16 — Re-Run the Automated Recovery Gates

Identity

No ERS login required.

Procedure

From the repository root, run:

make backup-creation-retention-verification-check

Expected:

Bite 33.2 backup creation, retention, and verification verified.

Run:

make off-host-backup-protection-check

Expected:

Bite 33.3 off-host backup protection verified.

Run:

make restore-tooling-recovery-verification-check

Expected:

Bite 33.4 restore tooling and recovery verification verified with Go Production tooling.

Run:

make production-environment-guardrails-check

Expected: complete Bite 33.1 guardrail regression passes.

Finally:

make local-check

Expected: complete LOCAL gate passes.

CI has already passed for this SHA; retain that result in the promotion evidence.

20. Test 17 — Retain Evidence and Clean Up the Disposable Environment

Identity

No ERS login required.

Procedure

Load:

source /tmp/ers-334-manual-session.env

Re-establish any derived evidence variables if Terminal was restarted after the earlier tests:

if [ -z "${ERS334_PRERESTORE_BACKUP:-}" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/|-!--s-"$ERS334_PRERESTORE_BACKUP"-); then
  export ERS334_PRERESTORE_BACKUP="$(
    find "$ERS334_ENV_DIR/backups" \
      -maxdepth 1 \
      -type f \
      -name 'app-*.db' \
      -print \
      | sort \
      | tail -1
  )"
fi

if [ -z "${ERS334_RESTORE_REPORT:-}" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/|-!--s-"$ERS334_RESTORE_REPORT"-); then
  export ERS334_RESTORE_REPORT="$(
    find "$ERS334_ENV_DIR/recovery/reports" \
      -maxdepth 1 \
      -type f \
      -name 'restore-*.json' \
      -print \
      | sort \
      | tail -1
  )"
fi

if [ -z "${ERS334_RESTORED_LIVE_SHA:-}" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--z-"${ERS334_RESTORED_LIVE_SHA:-}"-); then
  export ERS334_RESTORED_LIVE_SHA="$(
    docker exec "${ERS334_CONTAINER_PREFIX}-backend" \
      sha256sum /app/data/app.db \
      | awk '{print $1}'
  )"
fi

Confirm the required evidence variables are usable:

test -s "$ERS334_PRERESTORE_BACKUP" \
  && echo "PASS: pre-restore managed backup located"

test -s "$ERS334_RESTORE_REPORT" \
  && echo "PASS: restore report located"

test -n "$ERS334_RESTORED_LIVE_SHA" \
  && echo "PASS: restored live SHA-256 available"

Expected:

PASS: pre-restore managed backup located
PASS: restore report located
PASS: restored live SHA-256 available

Create a non-secret evidence directory in Downloads:

export ERS334_EVIDENCE_DIR="$HOME/Downloads/ers-bite-334-manual-evidence-b5e2bb92"

rm -rf "$ERS334_EVIDENCE_DIR"
mkdir -p "$ERS334_EVIDENCE_DIR"

Copy the restore report:

cp "$ERS334_RESTORE_REPORT" "$ERS334_EVIDENCE_DIR/"

Write a summary that contains no credentials:

cat > "$ERS334_EVIDENCE_DIR/summary.txt" <<EOF
Branch=941-bite-334-restore-tooling-and-recovery-verification
SHA=b5e2bb928cc3b95af60db82a1811de0b4cc5139f
S3Region=hel1
S3Endpoint=https://hel1.your-objectstorage.com
S3Bucket=ers
S3Prefix=$ERS334_S3_PREFIX
SourceBackup=$ERS334_BACKUP_NAME
SourceBackupSHA256=$ERS334_SOURCE_SHA256
RestoredLiveSHA256=$ERS334_RESTORED_LIVE_SHA
PreRestoreBackup=$(basename "$ERS334_PRERESTORE_BACKUP")
RestoreReport=$(basename "$ERS334_RESTORE_REPORT")
EOF

Inspect:

cat "$ERS334_EVIDENCE_DIR/summary.txt"

Confirm no obvious secrets exist in retained evidence:

if grep -R -Eiq \
  'aws_secret_access_key|PRIVATE KEY|JWT_SECRET|LEDGER_CORRECTION_KEY|LEDGER_SETTLEMENT_KEY|OPENAI_API_KEY' \
  "$ERS334_EVIDENCE_DIR"; then
  echo "FAIL: retained evidence appears to contain secret material"
else
  echo "PASS: retained evidence contains no obvious secret material"
fi

Expected:

PASS: retained evidence contains no obvious secret material

Stop and delete only the disposable Development containers and volumes:

make \
  SERVER_ROOT="$ERS334_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS334_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS334_CONTAINER_PREFIX" \
  server-down-volumes \
  ENV=development

Verify that no disposable Bite 33.4 containers remain.

Important: do not search for ers-dev-* here. The normal Development deployment is outside this manual suite and may legitimately still exist.

ERS334_REMAINING_CONTAINERS="$(
  docker ps -a --format '{{.Names}}' \
    | grep -E "^${ERS334_CONTAINER_PREFIX}-(backend|frontend|caddy)$" \
    || true
)"

if [ -z "$ERS334_REMAINING_CONTAINERS" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--z-"$ERS334_REMAINING_CONTAINERS"-); then
  echo "PASS: disposable Bite 33.4 containers removed"
else
  echo "FAIL: disposable Bite 33.4 containers still exist:"
  printf '%s\n' "$ERS334_REMAINING_CONTAINERS"
fi

Expected:

PASS: disposable Bite 33.4 containers removed

Verify that no disposable Bite 33.4 Docker volumes remain:

ERS334_REMAINING_VOLUMES="$(
  docker volume ls --format '{{.Name}}' \
    | grep "^${ERS334_COMPOSE_PROJECT}_" \
    || true
)"

if [ -z "$ERS334_REMAINING_VOLUMES" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--z-"$ERS334_REMAINING_VOLUMES"-); then
  echo "PASS: disposable Bite 33.4 Docker volumes removed"
else
  echo "FAIL: disposable Bite 33.4 Docker volumes still exist:"
  printf '%s\n' "$ERS334_REMAINING_VOLUMES"
fi

Expected:

PASS: disposable Bite 33.4 Docker volumes removed

For information only, show whether the normal Development deployment still exists:

docker ps -a --format '{{.Names}}' \
  | grep -E '^ers-dev-(backend|frontend|caddy)$' \
  || true

If your normal Development deployment existed before this suite, seeing names such as these is expected:

ers-dev-backend
ers-dev-frontend
ers-dev-caddy

Those containers are not part of Bite 33.4 manual-test cleanup and must not be treated as a failure.

Remove the local disposable server tree and temporary credential copy:

rm -rf "$ERS334_SERVER_ROOT"
rm -rf /tmp/ers-334-manual-secrets
rm -f /tmp/ers-334-manual-session.env

Do not attempt to bypass Hetzner Object Lock to remove the S3 objects under:

$ERS334_S3_PREFIX

Those protected manual-test objects may remain until their retention period expires.

PASS Criteria

Disposable ers334-b5e2bb92 resources and the temporary copied credential are removed; any normal ers-dev deployment remains untouched; non-secret evidence is retained in Downloads.

21. Promotion Gate

Bite 33.4 is ready for promotion only when all of the following are true:

Branch is 941-bite-334-restore-tooling-and-recovery-verification.
HEAD is b5e2bb928cc3b95af60db82a1811de0b4cc5139f.
Any existing normal ers-dev deployment remained untouched; the suite used only ers334-b5e2bb92.
Disposable Development deployment starts healthy.
Initial Development DB passes SQLite integrity and foreign-key checks.
Source Bite 33.2 backup verifies.
Source Bite 33.3 S3 receipt exists and records exact database/manifest VersionIds.
Local recovery candidate stages and verifies.
Environment mismatch is rejected.
Recovery candidate outside the selected recovery/ tree is rejected.
Exact S3 candidate materializes successfully using the local receipt.
S3 recovery succeeds using only the managed backup filename while the local receipt is unavailable.
Host-loss materialized database SHA equals the original source backup SHA.
Wrong confirmation token is rejected before DB modification.
Apply from outside the recovery tree is rejected before DB modification.
Pre-destructive refusal tests leave backend running and DB SHA unchanged.
Deliberate post-backup probe is present in the live DB before restore.
Deliberate probe is absent from the selected source backup.
Actual restore with RESTORE-DEVELOPMENT exits 0.
Backend health recovers after successful restore.
Probe is absent after restore.
Restored live DB SHA equals selected source backup SHA.
Restored live DB integrity check is ok.
Restored live DB foreign-key check is empty.
Restore workflow creates exactly one fresh pre-restore managed backup.
Fresh pre-restore backup contains the probe state that was replaced.
Fresh pre-restore backup has manifest and verified S3 receipt.
Preserved raw pre-restore DB copy contains the probe state that was replaced.
Restore report status is verified.
Restore report source SHA equals selected source backup SHA.
Restore report previous-target evidence matches the preserved DB copy.
Restore report restored-target SHA equals selected source SHA.
Restore report records successful integrity/foreign-key/migration evidence.
Restore report contains no obvious secret material.
Deliberately induced destructive-phase failure exits non-zero.
Backend remains stopped after that failure.
Forced failure does not modify the DB while the backend is stopped.
Backend recovers only after explicit operator restart.
make backup-creation-retention-verification-check passes.
make off-host-backup-protection-check passes.
make restore-tooling-recovery-verification-check passes.
make production-environment-guardrails-check passes.
make local-check passes.
CI passes.

22. Promotion-Blocking Defects

Do not promote Bite 33.4 if any of the following occurs:

a recovery candidate is accepted without full Bite 33.2 verification;
an environment-mismatched backup is accepted;
a candidate outside the selected environment's recovery/ tree can be applied;
a wrong or missing confirmation token allows restore progress;
a rejected pre-destructive request changes the live DB or stops the backend;
recovery from Hetzner depends on a surviving local .offhost.json receipt;
S3 materialization retrieves latest/current DB or manifest objects instead of the exact recorded VersionIds;
an S3 materialized database differs from the source SHA-256;
the restore does not create a fresh verified pre-restore backup while the backend is running;
Production-style off-host protection is skipped when configured;
the backend remains running while SQLite replacement is occurring;
the restored live DB differs from the chosen source evidence;
post-restart integrity or foreign-key verification fails;
the pre-restore raw database is not preserved;
the restore report is missing, inaccurate, or contains secrets;
a failure after entering the destructive phase leaves the backend running;
a failure before actual replacement changes the database;
the backend cannot recover health after an otherwise successful restore;
any 33.1, 33.2, 33.3, 33.4, LOCAL, or CI regression fails.

23. Evidence to Retain

Retain:

Branch:
SHA:

Disposable environment:
Dedicated Compose project: ers334-b5e2bb92
Dedicated container prefix: ers334-b5e2bb92

Hetzner region:
Hetzner endpoint:
Hetzner bucket:
Manual S3 prefix:

Source backup filename:
Source backup SHA-256:
Source receipt database VersionId:
Source receipt manifest VersionId:

Local recovery candidate:
Off-host receipt-based recovery candidate:
Off-host backup-name bootstrap candidate:

Live DB SHA before restore:
Live DB SHA after restore:

Fresh pre-restore managed backup:
Fresh pre-restore backup receipt present:
Preserved pre-restore DB copy:
Restore report:

Forced-failure exit status:
Backend stopped after forced failure:
DB unchanged after forced failure:
Explicit restart health:

backup-creation-retention-verification-check:
off-host-backup-protection-check:
restore-tooling-recovery-verification-check:
production-environment-guardrails-check:
local-check:
CI:

Do not retain:

S3 secret key;
AWS credentials file;
application passwords;
JWT secrets;
ledger keys;
API keys.

24. Glossary Impact

none