Bite 33.5 — Migration Deployment Recovery Gate - RodrigoMattosoSilveira/EnterpriseRemoteSystems GitHub Wiki

Bite 33.5 — Migration / Deployment Recovery Gate

Manual Verification Checklist Before Promotion

Branch: 943-bite-335-migration-deployment-recovery-gate
Validated HEAD SHA: 806f2ecab3db30612004e98de4dbb70c1ff436f2

1. Purpose

This suite manually verifies Bite 33.5 without touching the normal ERS Development deployment, Test, or Production.

The suite creates an isolated deployed Development environment under a temporary server root and a dedicated Docker namespace:

Server root:       /tmp/ers-335-manual-server
Compose project:   ers335-806f2eca
Container prefix:  ers335-806f2eca

On macOS, /tmp normally resolves physically to /private/tmp. This suite canonicalizes the temporary server path at the beginning and uses that canonical path consistently in all later recovery evidence.

The suite also uses the real Hetzner Object Storage bucket previously established for ERS backups, under a unique manual-test prefix:

Region:   hel1
Endpoint: https://hel1.your-objectstorage.com
Bucket:   ers
Prefix:   manual-335-<UTC timestamp>
Profile:  ers-backup

The suite proves:

  • deployed backend startup honors APP_AUTO_MIGRATE=false;
  • first deployment can build and verify a database without a rollback source;
  • the migration gate remains armed until application verification succeeds;
  • a second gate is refused while an earlier gate remains armed;
  • gate commit is refused while the backend is stopped;
  • successful health/schema verification commits an armed gate;
  • an existing database receives a verified pre-migration backup before live migration;
  • the pre-migration backup is replicated to Hetzner when S3 protection is enabled;
  • the S3 receipt records exact database and manifest VersionIds;
  • the verified pre-migration backup is staged as a Bite 33.4 rollback candidate;
  • repository migrations are rehearsed against an isolated database copy before the live database is taken offline;
  • the complete application stack is stopped before live SQLite migration;
  • a post-deployment rollback restores the exact pre-migration database and leaves the stack stopped;
  • pre-migration logical state is preserved by rollback;
  • post-migration/post-deployment logical state is removed by rollback;
  • a deliberately invalid expected migration fails during rehearsal before the running application is stopped;
  • a successful existing-database migration/deployment flow can start, pass health/schema checks, and commit;
  • rollback is refused after a gate has been committed;
  • the automated 33.1–33.5 recovery gates still pass.

No ERS application login is required. These are deployment/recovery operator tests and are intentionally terminal-driven.

2. Safety Rules

These tests must not use:

/opt/EnterpriseRemoteSystems/test
/opt/EnterpriseRemoteSystems/production
ers-tst
ers-prd

They must also not reuse the normal Development namespace:

ers-dev
ers-dev-backend
ers-dev-frontend
ers-dev-caddy
ers-dev_backend-data

Existing normal ers-dev-* resources are allowed and must be left untouched.

The dedicated manual-test resources are:

ers335-806f2eca-backend
ers335-806f2eca-frontend
ers335-806f2eca-caddy

ers335-806f2eca_backend-data
ers335-806f2eca_caddy-data
ers335-806f2eca_caddy-config

Only those dedicated resources may be removed by this suite.

Because the Hetzner bucket uses Object Lock / Governance retention, manual-test objects created under the unique manual-335-* prefix may remain until their retention period expires. Do not bypass Governance retention merely to clean up this rehearsal.

3. Test 00 — Confirm Branch, SHA, Docker, and Required Tools

Identity / context

  • OS identity: your normal macOS development user.
  • ERS identity: none; do not sign in to ERS.
  • Working directory: ERS repository root.

Procedure

Run:

git branch --show-current
git rev-parse HEAD

Expected exactly:

943-bite-335-migration-deployment-recovery-gate
806f2ecab3db30612004e98de4dbb70c1ff436f2

Confirm required tools:

for command in \
  git \
  docker \
  sqlite3 \
  aws \
  jq \
  shasum \
  python3 \
  make \
  tar
do
  command -v "$command" >/dev/null 2>&1 \
    && echo "PASS: $command is available" \
    || echo "FAIL: $command is missing"
done

Expected: one PASS: line for every command and no FAIL: line.

Confirm Docker:

docker info >/dev/null 2>&1 \
  && echo "PASS: Docker daemon is reachable" \
  || echo "FAIL: Docker daemon is not reachable"

Expected:

PASS: Docker daemon is reachable

PASS criteria

PASS only if:

  • branch matches exactly;
  • SHA matches exactly;
  • every required command is present;
  • Docker is reachable.

Promotion blocker

Do not continue if branch/SHA differs or Docker is unavailable.

4. Test 01 — Prove the Dedicated Docker Namespace Is Safe

Identity / context

No ERS login required.

Procedure

Define the dedicated manual-test Docker identity:

export ERS335_COMPOSE_PROJECT="ers335-806f2eca"
export ERS335_CONTAINER_PREFIX="ers335-806f2eca"

For information only, display any normal Development deployment:

docker ps -a --format '{{.Names}}' \
  | grep -E '^ers-dev-(backend|frontend|caddy)$' \
  || true

If ers-dev-* names appear, that is acceptable. Do not remove them.

Check only for stale Bite 33.5 manual-test containers:

ERS335_STALE_CONTAINERS="$(
  docker ps -a --format '{{.Names}}' \
    | grep -E "^${ERS335_CONTAINER_PREFIX}-(backend|frontend|caddy)$" \
    || true
)"

if [ -z "$ERS335_STALE_CONTAINERS" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--z-"$ERS335_STALE_CONTAINERS"-); then
  echo "PASS: no stale Bite 33.5 manual-test containers exist"
else
  echo "STALE: Bite 33.5 manual-test containers exist:"
  printf '%s\n' "$ERS335_STALE_CONTAINERS"
fi

Check only for stale Bite 33.5 volumes:

ERS335_STALE_VOLUMES="$(
  docker volume ls --format '{{.Name}}' \
    | grep "^${ERS335_COMPOSE_PROJECT}_" \
    || true
)"

if [ -z "$ERS335_STALE_VOLUMES" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--z-"$ERS335_STALE_VOLUMES"-); then
  echo "PASS: no stale Bite 33.5 manual-test volumes exist"
else
  echo "STALE: Bite 33.5 manual-test volumes exist:"
  printf '%s\n' "$ERS335_STALE_VOLUMES"
fi

If either check reports STALE:, remove only the dedicated Bite 33.5 resources:

docker rm -f \
  "${ERS335_CONTAINER_PREFIX}-backend" \
  "${ERS335_CONTAINER_PREFIX}-frontend" \
  "${ERS335_CONTAINER_PREFIX}-caddy" \
  2>/dev/null || true

Then:

for volume in \
  "${ERS335_COMPOSE_PROJECT}_backend-data" \
  "${ERS335_COMPOSE_PROJECT}_caddy-data" \
  "${ERS335_COMPOSE_PROJECT}_caddy-config"
do
  docker volume rm "$volume" 2>/dev/null || true
done

Re-run the two dedicated namespace checks.

PASS criteria

Final output must include:

PASS: no stale Bite 33.5 manual-test containers exist
PASS: no stale Bite 33.5 manual-test volumes exist

Existing ers-dev-* resources are not a failure.

5. Test 02 — Locate and Verify the Existing Hetzner S3 Credential

Identity / context

No ERS login required.

Purpose

The existing-database migration gate must prove that a pre-migration backup leaves the application host when S3 protection is enabled.

Procedure

Locate a credentials file containing the existing ers-backup profile:

unset ERS335_SOURCE_AWS_CREDENTIALS_FILE

if [[ -n "${AWS_SHARED_CREDENTIALS_FILE:-}" \
      && -f "$AWS_SHARED_CREDENTIALS_FILE" \
      && "$(grep -c '^\[ers-backup\]$' "$AWS_SHARED_CREDENTIALS_FILE")" -ge 1 ]]; then
  export ERS335_SOURCE_AWS_CREDENTIALS_FILE="$AWS_SHARED_CREDENTIALS_FILE"
elif [[ -f "$HOME/.aws/credentials" \
        && "$(grep -c '^\[ers-backup\]$' "$HOME/.aws/credentials")" -ge 1 ]]; then
  export ERS335_SOURCE_AWS_CREDENTIALS_FILE="$HOME/.aws/credentials"
elif [[ -f /tmp/ers-333-s3-manual/aws/credentials \
        && "$(grep -c '^\[ers-backup\]$' /tmp/ers-333-s3-manual/aws/credentials)" -ge 1 ]]; then
  export ERS335_SOURCE_AWS_CREDENTIALS_FILE="/tmp/ers-333-s3-manual/aws/credentials"
else
  echo "FAIL: no credentials file containing [ers-backup] was found"
fi

Assert:

if [ -n "${ERS335_SOURCE_AWS_CREDENTIALS_FILE:-}" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--n-"${ERS335_SOURCE_AWS_CREDENTIALS_FILE:-}"-); then
  printf 'Source credentials file: %s\n' \
    "$ERS335_SOURCE_AWS_CREDENTIALS_FILE"
  echo "PASS: ers-backup credential profile located"
else
  echo "FAIL: ers-backup credential profile was not located"
fi

Expected:

PASS: ers-backup credential profile located

Create a dedicated temporary secret directory outside the disposable ERS server tree:

rm -rf /tmp/ers-335-manual-secrets
mkdir -p /tmp/ers-335-manual-secrets
chmod 700 /tmp/ers-335-manual-secrets

export ERS335_SECRET_DIR="$(
  cd /tmp/ers-335-manual-secrets && pwd -P
)"

cp \
  "$ERS335_SOURCE_AWS_CREDENTIALS_FILE" \
  "$ERS335_SECRET_DIR/aws-credentials"

chmod 600 "$ERS335_SECRET_DIR/aws-credentials"

export ERS335_AWS_CREDENTIALS_FILE="$ERS335_SECRET_DIR/aws-credentials"

Check permissions:

ls -l "$ERS335_AWS_CREDENTIALS_FILE"

Verify real Hetzner access:

AWS_SHARED_CREDENTIALS_FILE="$ERS335_AWS_CREDENTIALS_FILE" \
AWS_PROFILE=ers-backup \
AWS_REGION=hel1 \
aws s3api head-bucket \
  --bucket ers \
  --endpoint-url https://hel1.your-objectstorage.com

Immediately:

echo $?

Expected:

0

PASS criteria

PASS only if:

  • ers-backup credentials are found;
  • the temporary copy exists with restrictive permissions;
  • Hetzner head-bucket exits 0.

6. Test 03 — Build the Isolated Server Tree and Persist Session State

Identity / context

No ERS login required.

Procedure

Create a clean disposable server tree:

rm -rf /tmp/ers-335-manual-server
mkdir -p /tmp/ers-335-manual-server/development

Canonicalize the macOS temporary path immediately:

export ERS335_SERVER_ROOT="$(
  cd /tmp/ers-335-manual-server && pwd -P
)"

export ERS335_ENV_DIR="$ERS335_SERVER_ROOT/development"
export ERS335_COMPOSE_PROJECT="ers335-806f2eca"
export ERS335_CONTAINER_PREFIX="ers335-806f2eca"
export ERS335_STATE_FILE="$ERS335_ENV_DIR/recovery/migration-gates/current.state"
export ERS335_S3_PREFIX="manual-335-$(date -u +%Y%m%dT%H%M%SZ)"

Display the physical server path:

printf 'Canonical server root: %s\n' "$ERS335_SERVER_ROOT"
printf 'Environment dir:       %s\n' "$ERS335_ENV_DIR"

On macOS the root will normally begin with:

/private/tmp/

Export the exact current HEAD into the isolated environment:

git archive HEAD | tar -x -C "$ERS335_ENV_DIR"

Confirm Bite 33.5 files exist:

test -f "$ERS335_ENV_DIR/scripts/server-migration-recovery-gate.sh" \
  && test -f "$ERS335_ENV_DIR/scripts/server-migration-recovery-rollback.sh" \
  && test -f "$ERS335_ENV_DIR/scripts/server-migration-recovery-commit.sh" \
  && test -f "$ERS335_ENV_DIR/backend/apply-sql-migrations.sh" \
  && echo "PASS: Bite 33.5 source exported into isolated server tree"

Expected:

PASS: Bite 33.5 source exported into isolated server tree

Initialize a Development server environment:

(
  cd "$ERS335_ENV_DIR"
  ./scripts/init-server-env.sh development
)

Define a helper to update the isolated environment without duplicate active definitions:

set_ers335_env_value() {
  local file="$1"
  local key="$2"
  local value="$3"
  local tmp="${file}.tmp"

  grep -v "^${key}=" "$file" > "$tmp" || true
  printf '%s=%s\n' "$key" "$value" >> "$tmp"
  mv "$tmp" "$file"
}

Force the dedicated container prefix:

set_ers335_env_value \
  "$ERS335_ENV_DIR/.env.development" \
  CONTAINER_PREFIX \
  "$ERS335_CONTAINER_PREFIX"

Confirm deployed startup must not auto-migrate:

set_ers335_env_value \
  "$ERS335_ENV_DIR/.env.development" \
  APP_AUTO_MIGRATE \
  false

Enable real S3 protection under the unique manual prefix:

set_ers335_env_value \
  "$ERS335_ENV_DIR/.env.development" \
  SERVER_OFFHOST_BACKUP_ENABLED \
  true

set_ers335_env_value \
  "$ERS335_ENV_DIR/.env.development" \
  SERVER_OFFHOST_BACKUP_TRANSPORT \
  s3

set_ers335_env_value \
  "$ERS335_ENV_DIR/.env.development" \
  SERVER_OFFHOST_BACKUP_S3_ENDPOINT \
  https://hel1.your-objectstorage.com

set_ers335_env_value \
  "$ERS335_ENV_DIR/.env.development" \
  SERVER_OFFHOST_BACKUP_S3_REGION \
  hel1

set_ers335_env_value \
  "$ERS335_ENV_DIR/.env.development" \
  SERVER_OFFHOST_BACKUP_S3_BUCKET \
  ers

set_ers335_env_value \
  "$ERS335_ENV_DIR/.env.development" \
  SERVER_OFFHOST_BACKUP_S3_PREFIX \
  "$ERS335_S3_PREFIX"

set_ers335_env_value \
  "$ERS335_ENV_DIR/.env.development" \
  AWS_SHARED_CREDENTIALS_FILE \
  "$ERS335_AWS_CREDENTIALS_FILE"

set_ers335_env_value \
  "$ERS335_ENV_DIR/.env.development" \
  AWS_PROFILE \
  ers-backup

Display only non-secret configuration:

grep -E \
  '^(APP_ENV|APP_AUTO_MIGRATE|CONTAINER_PREFIX|SERVER_OFFHOST_BACKUP_ENABLED|SERVER_OFFHOST_BACKUP_TRANSPORT|SERVER_OFFHOST_BACKUP_S3_ENDPOINT|SERVER_OFFHOST_BACKUP_S3_REGION|SERVER_OFFHOST_BACKUP_S3_BUCKET|SERVER_OFFHOST_BACKUP_S3_PREFIX|AWS_SHARED_CREDENTIALS_FILE|AWS_PROFILE)=' \
  "$ERS335_ENV_DIR/.env.development"

Expected values include:

APP_ENV=development
APP_AUTO_MIGRATE=false
CONTAINER_PREFIX=ers335-806f2eca
SERVER_OFFHOST_BACKUP_ENABLED=true
SERVER_OFFHOST_BACKUP_TRANSPORT=s3
SERVER_OFFHOST_BACKUP_S3_ENDPOINT=https://hel1.your-objectstorage.com
SERVER_OFFHOST_BACKUP_S3_REGION=hel1
SERVER_OFFHOST_BACKUP_S3_BUCKET=ers
AWS_PROFILE=ers-backup

Validate the environment contract:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-environment-contract-check \
  ENV=development

Expected exit:

echo $?

0

Persist the non-secret manual-test state:

cat > /tmp/ers-335-manual-session.env <<EOF
export ERS335_SERVER_ROOT='$ERS335_SERVER_ROOT'
export ERS335_ENV_DIR='$ERS335_ENV_DIR'
export ERS335_COMPOSE_PROJECT='$ERS335_COMPOSE_PROJECT'
export ERS335_CONTAINER_PREFIX='$ERS335_CONTAINER_PREFIX'
export ERS335_STATE_FILE='$ERS335_STATE_FILE'
export ERS335_S3_PREFIX='$ERS335_S3_PREFIX'
export ERS335_AWS_CREDENTIALS_FILE='$ERS335_AWS_CREDENTIALS_FILE'
EOF

PASS criteria

PASS only if the environment contract succeeds and the persisted session file exists.

7. Test 04 — Build Images Without Creating a Database Volume

Identity / context

No ERS login required.

Procedure

Reload session state:

source /tmp/ers-335-manual-session.env

Confirm no dedicated database volume exists yet:

if docker volume inspect \
  "${ERS335_COMPOSE_PROJECT}_backend-data" \
  >/dev/null 2>&1
then
  echo "FAIL: dedicated database volume already exists before first-deployment gate"
else
  echo "PASS: no dedicated database volume exists before first-deployment gate"
fi

Expected:

PASS: no dedicated database volume exists before first-deployment gate

Build the isolated deployed images:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-build \
  ENV=development

Expected exit:

echo $?

0

Reconfirm image build did not create the database volume:

if docker volume inspect \
  "${ERS335_COMPOSE_PROJECT}_backend-data" \
  >/dev/null 2>&1
then
  echo "FAIL: image build unexpectedly created the database volume"
else
  echo "PASS: image build did not create the database volume"
fi

Expected:

PASS: image build did not create the database volume

8. Test 05 — First-Deployment Migration Gate

Identity / context

No ERS login required.

Purpose

Exercise the 33.5 first-deployment path where there is no prior SQLite database and therefore no rollback source.

Procedure

Load state:

source /tmp/ers-335-manual-session.env

Run the migration gate:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-migration-recovery-gate \
  ENV=development

Expected output includes:

No existing SQLite volume is present; preparing first-deployment migration gate without a rollback source.
Rehearsing repository migrations against an isolated copy before live database mutation.
Migration rehearsal passed:
Stopping the deployed application stack before live migration.
Migration/deployment recovery gate ARMED.
Verified rollback candidate: none (first deployment)
The application stack remains stopped until post-migration deployment checks succeed.

Immediately:

echo $?

Expected:

0

Confirm state exists:

test -s "$ERS335_STATE_FILE" \
  && echo "PASS: migration gate state exists" \
  || echo "FAIL: migration gate state is missing"

Expected:

PASS: migration gate state exists

Display it:

cat "$ERS335_STATE_FILE"

Assert the first-deployment state:

if grep -qx 'status=armed' "$ERS335_STATE_FILE" \
  && grep -qx 'environment=development' "$ERS335_STATE_FILE" \
  && grep -qx 'first_deployment=1' "$ERS335_STATE_FILE" \
  && grep -qx 'backup_file=' "$ERS335_STATE_FILE" \
  && grep -qx 'recovery_backup=' "$ERS335_STATE_FILE" \
  && grep -qx 'expected_final_migration=000077_journey_bonus_award_approval.up.sql' "$ERS335_STATE_FILE"
then
  echo "PASS: first-deployment gate is armed with no rollback source"
else
  echo "FAIL: first-deployment gate state is incorrect"
fi

Expected:

PASS: first-deployment gate is armed with no rollback source

Confirm the database volume now exists:

docker volume inspect \
  "${ERS335_COMPOSE_PROJECT}_backend-data" \
  >/dev/null \
  && echo "PASS: first-deployment database volume exists"

Expected:

PASS: first-deployment database volume exists

Confirm the final migration exists in the offline live volume without starting the backend:

ERS335_FINAL_MIGRATION_COUNT="$(
  cd "$ERS335_ENV_DIR" \
  && env \
      -u APP_ENV \
      -u APP_AUTO_MIGRATE \
      -u AUTHZ_ACTOR_HEADER_MODE \
      -u DEV_SEED_ADMIN \
      AUTHZ_BOOTSTRAP_ENABLED=false \
      docker compose \
        -p "$ERS335_COMPOSE_PROJECT" \
        --env-file .env.development \
        -f docker-compose.server.yml \
        run --rm --no-deps \
        --entrypoint sqlite3 \
        backend \
        -readonly \
        /app/data/app.db \
        "SELECT COUNT(*) FROM schema_migrations WHERE filename='000077_journey_bonus_award_approval.up.sql';" \
  | tail -1
)"

if [ "$ERS335_FINAL_MIGRATION_COUNT" == "1" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$ERS335_FINAL_MIGRATION_COUNT"-==-"1"-); then
  echo "PASS: first-deployment database contains final migration 000077"
else
  printf 'FAIL: final migration count is %s\n' \
    "$ERS335_FINAL_MIGRATION_COUNT"
fi

Expected:

PASS: first-deployment database contains final migration 000077

Confirm the application backend is not running:

if docker ps --format '{{.Names}}' \
  | grep -qx "${ERS335_CONTAINER_PREFIX}-backend"
then
  echo "FAIL: backend is running while first-deployment gate is only armed"
else
  echo "PASS: backend remains stopped while migration gate is armed"
fi

Expected:

PASS: backend remains stopped while migration gate is armed

PASS criteria

All explicit checks must print PASS.

9. Test 06 — Refuse a Second Migration Gate While One Is Armed

Identity / context

No ERS login required.

Procedure

Run the same gate again:

source /tmp/ers-335-manual-session.env

set +e

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-migration-recovery-gate \
  ENV=development \
  > /tmp/ers-335-second-armed-gate.out 2>&1

ERS335_SECOND_GATE_STATUS=$?

set -e

Display:

cat /tmp/ers-335-second-armed-gate.out
printf 'Exit status: %s\n' "$ERS335_SECOND_GATE_STATUS"

Expected output includes:

Refusing to create a new migration gate while an earlier gate remains armed:
Commit or roll back the existing gate first.

Assert:

if [ "$ERS335_SECOND_GATE_STATUS" -ne 0 ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$ERS335_SECOND_GATE_STATUS"--ne-0-); then
  echo "PASS: second migration gate refused while earlier gate is armed"
else
  echo "FAIL: second migration gate unexpectedly succeeded"
fi

Expected:

PASS: second migration gate refused while earlier gate is armed

Confirm state remains armed:

grep -qx 'status=armed' "$ERS335_STATE_FILE" \
  && echo "PASS: original gate remains armed"

Expected:

PASS: original gate remains armed

10. Test 07 — Refuse Gate Commit While Backend Is Stopped

Identity / context

No ERS login required.

Procedure

Run:

source /tmp/ers-335-manual-session.env

set +e

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-migration-recovery-commit \
  ENV=development \
  > /tmp/ers-335-commit-while-stopped.out 2>&1

ERS335_STOPPED_COMMIT_STATUS=$?

set -e

Display:

cat /tmp/ers-335-commit-while-stopped.out
printf 'Exit status: %s\n' "$ERS335_STOPPED_COMMIT_STATUS"

Expected output includes:

Cannot commit migration gate: backend is not running:

Assert:

if [ "$ERS335_STOPPED_COMMIT_STATUS" -ne 0 ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$ERS335_STOPPED_COMMIT_STATUS"--ne-0-); then
  echo "PASS: commit refused while backend is stopped"
else
  echo "FAIL: commit unexpectedly succeeded while backend was stopped"
fi

Expected:

PASS: commit refused while backend is stopped

State must still be armed:

grep -qx 'status=armed' "$ERS335_STATE_FILE" \
  && echo "PASS: failed commit left gate armed"

Expected:

PASS: failed commit left gate armed

11. Test 08 — Start the First Deployment, Verify No Implicit Migration, and Commit

Identity / context

No ERS login required.

Procedure

Start the isolated application stack:

source /tmp/ers-335-manual-session.env

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-up \
  ENV=development

Verify backend health:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-backend-health \
  ENV=development

Expected output includes:

ers335-806f2eca-backend health check passed.

Verify the complete migrated database contract:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-migrated-db-verify \
  ENV=development

Expected output begins with:

Migrated database verification passed.

and includes:

Final migration: 000077_journey_bonus_award_approval.up.sql
Integrity check: ok
Foreign key check: clean

Verify Caddy/internal environment health:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-env-caddy-health \
  ENV=development

Expected:

ers335-806f2eca-caddy internal health check passed.

Inspect backend startup logs for the explicit no-auto-migrate contract:

docker logs "${ERS335_CONTAINER_PREFIX}-backend" 2>&1 \
  | grep -F \
    'APP_AUTO_MIGRATE=false: refusing implicit schema mutation; verifying deployed database before API startup.'

Expected exactly one or more matching lines containing:

APP_AUTO_MIGRATE=false: refusing implicit schema mutation; verifying deployed database before API startup.

Confirm the backend did not announce the auto-migrate path:

if docker logs "${ERS335_CONTAINER_PREFIX}-backend" 2>&1 \
  | grep -Fq 'APP_AUTO_MIGRATE=true: applying SQL migrations before API startup.'
then
  echo "FAIL: backend used implicit auto-migration path"
else
  echo "PASS: backend startup did not use implicit auto-migration"
fi

Expected:

PASS: backend startup did not use implicit auto-migration

Commit the gate:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-migration-recovery-commit \
  ENV=development

Expected output:

Migration/deployment recovery gate COMMITTED.
Backend health, SQLite integrity/foreign keys, and migration history are verified.

Confirm state:

grep -qx 'status=committed' "$ERS335_STATE_FILE" \
  && grep -q '^committed_at=' "$ERS335_STATE_FILE" \
  && echo "PASS: first-deployment migration gate committed"

Expected:

PASS: first-deployment migration gate committed

12. Test 09 — Existing-Database Gate Creates Verified Local and S3 Recovery Evidence

Identity / context

No ERS login required.

Purpose

Exercise the path Production will use: an existing running database must be backed up, replicated when off-host protection is enabled, staged, rehearsed, then migrated offline.

Procedure

Load session state:

source /tmp/ers-335-manual-session.env

Create a harmless logical marker that must exist in the pre-migration backup:

docker exec "${ERS335_CONTAINER_PREFIX}-backend" \
  sqlite3 /app/data/app.db "
    CREATE TABLE IF NOT EXISTS bite335_manual_pre_gate_probe (
      id TEXT PRIMARY KEY,
      value TEXT NOT NULL
    );
    DELETE FROM bite335_manual_pre_gate_probe;
    INSERT INTO bite335_manual_pre_gate_probe(id, value)
    VALUES ('pre-gate-1', 'must-survive-rollback');
  "

Verify:

docker exec "${ERS335_CONTAINER_PREFIX}-backend" \
  sqlite3 /app/data/app.db \
  "SELECT id || ':' || value FROM bite335_manual_pre_gate_probe;"

Expected:

pre-gate-1:must-survive-rollback

Run the migration gate while the backend is running:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-migration-recovery-gate \
  ENV=development

Expected output includes:

Creating verified online pre-migration backup before taking the application offline.
Off-host backup verification passed:
Rehearsing repository migrations against an isolated copy before live database mutation.
Migration rehearsal passed:
Stopping the deployed application stack before live migration.
Migration/deployment recovery gate ARMED.
Verified rollback candidate:
The application stack remains stopped until post-migration deployment checks succeed.

Confirm state is armed and not first-deployment:

if grep -qx 'status=armed' "$ERS335_STATE_FILE" \
  && grep -qx 'first_deployment=0' "$ERS335_STATE_FILE"
then
  echo "PASS: existing-database migration gate is armed"
else
  echo "FAIL: existing-database migration gate state is incorrect"
fi

Expected:

PASS: existing-database migration gate is armed

Load exact evidence paths from the state file:

export ERS335_ROLLBACK_GATE_DIR="$(
  grep '^gate_dir=' "$ERS335_STATE_FILE" | tail -1 | cut -d= -f2-
)"

export ERS335_BACKUP_FILE="$(
  grep '^backup_file=' "$ERS335_STATE_FILE" | tail -1 | cut -d= -f2-
)"

export ERS335_RECOVERY_BACKUP="$(
  grep '^recovery_backup=' "$ERS335_STATE_FILE" | tail -1 | cut -d= -f2-
)"

export ERS335_SOURCE_SHA256="$(
  grep '^source_sha256=' "$ERS335_STATE_FILE" | tail -1 | cut -d= -f2-
)"

export ERS335_PROBE_SHA256="$(
  grep '^probe_sha256=' "$ERS335_STATE_FILE" | tail -1 | cut -d= -f2-
)"

Persist them:

{
  printf "export ERS335_ROLLBACK_GATE_DIR='%s'\n" "$ERS335_ROLLBACK_GATE_DIR"
  printf "export ERS335_BACKUP_FILE='%s'\n" "$ERS335_BACKUP_FILE"
  printf "export ERS335_RECOVERY_BACKUP='%s'\n" "$ERS335_RECOVERY_BACKUP"
  printf "export ERS335_SOURCE_SHA256='%s'\n" "$ERS335_SOURCE_SHA256"
  printf "export ERS335_PROBE_SHA256='%s'\n" "$ERS335_PROBE_SHA256"
} >> /tmp/ers-335-manual-session.env

Confirm all required local evidence exists:

test -s "$ERS335_BACKUP_FILE" \
  && echo "PASS: verified pre-migration backup exists" \
  || echo "FAIL: pre-migration backup is missing"

test -s "$ERS335_BACKUP_FILE.manifest.json" \
  && echo "PASS: pre-migration backup manifest exists" \
  || echo "FAIL: pre-migration manifest is missing"

test -s "$ERS335_BACKUP_FILE.offhost.json" \
  && echo "PASS: pre-migration S3 receipt exists" \
  || echo "FAIL: pre-migration S3 receipt is missing"

test -s "$ERS335_RECOVERY_BACKUP" \
  && test -s "$ERS335_RECOVERY_BACKUP.manifest.json" \
  && echo "PASS: verified rollback candidate exists" \
  || echo "FAIL: rollback candidate is incomplete"

test -s "$ERS335_ROLLBACK_GATE_DIR/migration-probe.db" \
  && echo "PASS: isolated migration rehearsal database exists" \
  || echo "FAIL: migration rehearsal database is missing"

Expected: five PASS: lines.

Compare the backup SHA with gate state:

ERS335_BACKUP_SHA_ACTUAL="$(
  shasum -a 256 "$ERS335_BACKUP_FILE" | awk '{print $1}'
)"

if [ "$ERS335_BACKUP_SHA_ACTUAL" == "$ERS335_SOURCE_SHA256" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$ERS335_BACKUP_SHA_ACTUAL"-==-"$ERS335_SOURCE_SHA256"-); then
  echo "PASS: gate source SHA-256 matches pre-migration backup"
else
  echo "FAIL: gate source SHA-256 does not match pre-migration backup"
fi

Expected:

PASS: gate source SHA-256 matches pre-migration backup

Compare staged rollback candidate SHA:

ERS335_RECOVERY_SHA_ACTUAL="$(
  shasum -a 256 "$ERS335_RECOVERY_BACKUP" | awk '{print $1}'
)"

if [ "$ERS335_RECOVERY_SHA_ACTUAL" == "$ERS335_SOURCE_SHA256" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$ERS335_RECOVERY_SHA_ACTUAL"-==-"$ERS335_SOURCE_SHA256"-); then
  echo "PASS: staged rollback candidate matches pre-migration source exactly"
else
  echo "FAIL: staged rollback candidate differs from pre-migration source"
fi

Expected:

PASS: staged rollback candidate matches pre-migration source exactly

Compare the migration-probe SHA with state:

ERS335_PROBE_SHA_ACTUAL="$(
  shasum -a 256 "$ERS335_ROLLBACK_GATE_DIR/migration-probe.db" \
    | awk '{print $1}'
)"

if [ "$ERS335_PROBE_SHA_ACTUAL" == "$ERS335_PROBE_SHA256" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$ERS335_PROBE_SHA_ACTUAL"-==-"$ERS335_PROBE_SHA256"-); then
  echo "PASS: migration rehearsal SHA-256 matches gate evidence"
else
  echo "FAIL: migration rehearsal SHA-256 does not match gate evidence"
fi

Expected:

PASS: migration rehearsal SHA-256 matches gate evidence

Validate the S3 receipt contract:

if jq -e \
  --arg prefix "$ERS335_S3_PREFIX" \
  '
    .status == "verified"
    and .environment == "development"
    and .transport == "s3"
    and .remote.endpoint == "https://hel1.your-objectstorage.com"
    and .remote.region == "hel1"
    and .remote.bucket == "ers"
    and .remote.prefix == $prefix
    and .remote.backup.version_id != ""
    and .remote.manifest.version_id != ""
    and .remote.backup.sha256 != ""
    and .remote.manifest.sha256 != ""
  ' \
  "$ERS335_BACKUP_FILE.offhost.json" \
  >/dev/null
then
  echo "PASS: pre-migration S3 receipt contains exact protected object evidence"
else
  echo "FAIL: pre-migration S3 receipt evidence is incomplete"
fi

Expected:

PASS: pre-migration S3 receipt contains exact protected object evidence

Record exact S3 keys and versions:

export ERS335_DB_KEY="$(
  jq -r '.remote.backup.key' "$ERS335_BACKUP_FILE.offhost.json"
)"

export ERS335_DB_VERSION_ID="$(
  jq -r '.remote.backup.version_id' "$ERS335_BACKUP_FILE.offhost.json"
)"

export ERS335_MANIFEST_KEY="$(
  jq -r '.remote.manifest.key' "$ERS335_BACKUP_FILE.offhost.json"
)"

export ERS335_MANIFEST_VERSION_ID="$(
  jq -r '.remote.manifest.version_id' "$ERS335_BACKUP_FILE.offhost.json"
)"

Verify those exact database and manifest versions exist in Hetzner:

AWS_SHARED_CREDENTIALS_FILE="$ERS335_AWS_CREDENTIALS_FILE" \
AWS_PROFILE=ers-backup \
AWS_REGION=hel1 \
aws s3api head-object \
  --bucket ers \
  --key "$ERS335_DB_KEY" \
  --version-id "$ERS335_DB_VERSION_ID" \
  --endpoint-url https://hel1.your-objectstorage.com \
  >/dev/null \
  && echo "PASS: exact pre-migration database VersionId exists in Hetzner"

Expected:

PASS: exact pre-migration database VersionId exists in Hetzner

Run:

AWS_SHARED_CREDENTIALS_FILE="$ERS335_AWS_CREDENTIALS_FILE" \
AWS_PROFILE=ers-backup \
AWS_REGION=hel1 \
aws s3api head-object \
  --bucket ers \
  --key "$ERS335_MANIFEST_KEY" \
  --version-id "$ERS335_MANIFEST_VERSION_ID" \
  --endpoint-url https://hel1.your-objectstorage.com \
  >/dev/null \
  && echo "PASS: exact pre-migration manifest VersionId exists in Hetzner"

Expected:

PASS: exact pre-migration manifest VersionId exists in Hetzner

Confirm the full application stack is now stopped:

ERS335_RUNNING_AFTER_GATE="$(
  docker ps --format '{{.Names}}' \
    | grep -E "^${ERS335_CONTAINER_PREFIX}-(backend|frontend|caddy)$" \
    || true
)"

if [ -z "$ERS335_RUNNING_AFTER_GATE" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--z-"$ERS335_RUNNING_AFTER_GATE"-); then
  echo "PASS: complete application stack is stopped while gate is armed"
else
  echo "FAIL: application services remain running while gate is armed:"
  printf '%s\n' "$ERS335_RUNNING_AFTER_GATE"
fi

Expected:

PASS: complete application stack is stopped while gate is armed

Promotion blocker

Do not promote if the gate reaches live migration without verified backup, staged rollback evidence, or successful rehearsal.

13. Test 10 — Simulate Post-Deployment Failure and Roll Back the Armed Gate

Identity / context

No ERS login required.

Purpose

Simulate the deployment workflow starting the new revision and then discovering a later deployment failure.

Procedure

Load state:

source /tmp/ers-335-manual-session.env

Start the application while the gate remains armed:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-up \
  ENV=development

Verify health:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-backend-health \
  ENV=development

Expected:

ers335-806f2eca-backend health check passed.

Confirm the pre-gate marker exists:

docker exec "${ERS335_CONTAINER_PREFIX}-backend" \
  sqlite3 /app/data/app.db \
  "SELECT id || ':' || value FROM bite335_manual_pre_gate_probe;"

Expected:

pre-gate-1:must-survive-rollback

Create a second marker that represents post-migration/post-deployment state and must disappear after rollback:

docker exec "${ERS335_CONTAINER_PREFIX}-backend" \
  sqlite3 /app/data/app.db "
    CREATE TABLE bite335_manual_post_deploy_probe (
      id TEXT PRIMARY KEY,
      value TEXT NOT NULL
    );
    INSERT INTO bite335_manual_post_deploy_probe(id, value)
    VALUES ('post-deploy-1', 'must-disappear-on-rollback');
  "

Verify:

docker exec "${ERS335_CONTAINER_PREFIX}-backend" \
  sqlite3 /app/data/app.db \
  "SELECT id || ':' || value FROM bite335_manual_post_deploy_probe;"

Expected:

post-deploy-1:must-disappear-on-rollback

Invoke the migration/deployment rollback:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-migration-recovery-rollback \
  ENV=development

Expected output includes:

Stopping deployed application stack before migration rollback.
Verified pre-migration database restored.
Migration/deployment recovery gate rollback completed.
The application stack remains stopped. Review evidence before retrying or starting another revision.

Confirm state changed:

grep -qx 'status=rolled_back' "$ERS335_STATE_FILE" \
  && grep -q '^rolled_back_at=' "$ERS335_STATE_FILE" \
  && echo "PASS: migration gate state records rollback"

Expected:

PASS: migration gate state records rollback

Confirm the stack is stopped:

ERS335_RUNNING_AFTER_ROLLBACK="$(
  docker ps --format '{{.Names}}' \
    | grep -E "^${ERS335_CONTAINER_PREFIX}-(backend|frontend|caddy)$" \
    || true
)"

if [ -z "$ERS335_RUNNING_AFTER_ROLLBACK" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--z-"$ERS335_RUNNING_AFTER_ROLLBACK"-); then
  echo "PASS: application stack remains stopped after rollback"
else
  echo "FAIL: application service is running after rollback:"
  printf '%s\n' "$ERS335_RUNNING_AFTER_ROLLBACK"
fi

Expected:

PASS: application stack remains stopped after rollback

While stopped, compute the restored live database SHA without opening SQLite:

export ERS335_ROLLED_BACK_LIVE_SHA="$(
  cd "$ERS335_ENV_DIR" \
  && env \
      -u APP_ENV \
      -u APP_AUTO_MIGRATE \
      -u AUTHZ_ACTOR_HEADER_MODE \
      -u DEV_SEED_ADMIN \
      AUTHZ_BOOTSTRAP_ENABLED=false \
      docker compose \
        -p "$ERS335_COMPOSE_PROJECT" \
        --env-file .env.development \
        -f docker-compose.server.yml \
        run --rm --no-deps \
        --entrypoint sha256sum \
        backend \
        /app/data/app.db \
  | awk '{print $1}'
)"

Compare with the pre-migration backup SHA:

printf 'Pre-migration source SHA: %s\n' "$ERS335_SOURCE_SHA256"
printf 'Rolled-back live SHA:      %s\n' "$ERS335_ROLLED_BACK_LIVE_SHA"

if [ "$ERS335_ROLLED_BACK_LIVE_SHA" == "$ERS335_SOURCE_SHA256" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$ERS335_ROLLED_BACK_LIVE_SHA"-==-"$ERS335_SOURCE_SHA256"-); then
  echo "PASS: rollback restored the exact pre-migration database bytes"
else
  echo "FAIL: rolled-back database differs from pre-migration source"
fi

Expected:

PASS: rollback restored the exact pre-migration database bytes

Query the stopped database read-only to prove the pre-gate marker survived:

ERS335_PRE_GATE_COUNT="$(
  cd "$ERS335_ENV_DIR" \
  && env \
      -u APP_ENV \
      -u APP_AUTO_MIGRATE \
      -u AUTHZ_ACTOR_HEADER_MODE \
      -u DEV_SEED_ADMIN \
      AUTHZ_BOOTSTRAP_ENABLED=false \
      docker compose \
        -p "$ERS335_COMPOSE_PROJECT" \
        --env-file .env.development \
        -f docker-compose.server.yml \
        run --rm --no-deps \
        --entrypoint sqlite3 \
        backend \
        -readonly \
        /app/data/app.db \
        "SELECT COUNT(*) FROM bite335_manual_pre_gate_probe WHERE id='pre-gate-1' AND value='must-survive-rollback';" \
  | tail -1
)"

if [ "$ERS335_PRE_GATE_COUNT" == "1" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$ERS335_PRE_GATE_COUNT"-==-"1"-); then
  echo "PASS: rollback preserved pre-migration logical state"
else
  echo "FAIL: pre-migration logical state was not restored"
fi

Expected:

PASS: rollback preserved pre-migration logical state

Prove the post-deploy marker disappeared:

ERS335_POST_DEPLOY_TABLE_COUNT="$(
  cd "$ERS335_ENV_DIR" \
  && env \
      -u APP_ENV \
      -u APP_AUTO_MIGRATE \
      -u AUTHZ_ACTOR_HEADER_MODE \
      -u DEV_SEED_ADMIN \
      AUTHZ_BOOTSTRAP_ENABLED=false \
      docker compose \
        -p "$ERS335_COMPOSE_PROJECT" \
        --env-file .env.development \
        -f docker-compose.server.yml \
        run --rm --no-deps \
        --entrypoint sqlite3 \
        backend \
        -readonly \
        /app/data/app.db \
        "SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name='bite335_manual_post_deploy_probe';" \
  | tail -1
)"

if [ "$ERS335_POST_DEPLOY_TABLE_COUNT" == "0" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$ERS335_POST_DEPLOY_TABLE_COUNT"-==-"0"-); then
  echo "PASS: rollback removed post-deployment logical state"
else
  echo "FAIL: post-deployment logical state survived rollback"
fi

Expected:

PASS: rollback removed post-deployment logical state

Verify the Bite 33.4 rollback report:

if jq -e \
  '
    .status == "verified"
    and .environment == "development"
    and .restored_target.path == "/app/data/app.db"
    and .restored_target.integrity_check == "ok"
    and .restored_target.foreign_key_check == "ok"
  ' \
  "$ERS335_ROLLBACK_GATE_DIR/deployment-rollback.json" \
  >/dev/null
then
  echo "PASS: deployment rollback report is verified"
else
  echo "FAIL: deployment rollback report is missing or invalid"
fi

Expected:

PASS: deployment rollback report is verified

Explicitly restart only after the rollback evidence is verified:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-up \
  ENV=development

Then:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-backend-health \
  ENV=development

Expected:

ers335-806f2eca-backend health check passed.

PASS criteria

Rollback must restore the exact source bytes while stopped, preserve the pre-gate marker, remove the post-deploy marker, and leave services stopped until the explicit operator restart.

14. Test 11 — Migration Rehearsal Failure Blocks Before Live Mutation

Identity / context

No ERS login required.

Purpose

Force a deterministic preflight failure by requesting a final migration that does not exist in the repository.

The gate must fail during isolated rehearsal while the currently running application remains available.

Procedure

Load state:

source /tmp/ers-335-manual-session.env

Confirm backend is healthy before the failed rehearsal:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-backend-health \
  ENV=development

Run the gate with an impossible expected final migration:

set +e

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  DEPLOYMENT_FINAL_MIGRATION="000999_bite335_manual_missing.up.sql" \
  server-migration-recovery-gate \
  ENV=development \
  > /tmp/ers-335-rehearsal-failure.out 2>&1

ERS335_REHEARSAL_FAILURE_STATUS=$?

set -e

Display:

cat /tmp/ers-335-rehearsal-failure.out
printf 'Exit status: %s\n' "$ERS335_REHEARSAL_FAILURE_STATUS"

Expected output includes:

Rehearsing repository migrations against an isolated copy before live database mutation.
Expected final migration is not present in the repository: 000999_bite335_manual_missing.up.sql

Assert:

if [ "$ERS335_REHEARSAL_FAILURE_STATUS" -ne 0 ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$ERS335_REHEARSAL_FAILURE_STATUS"--ne-0-); then
  echo "PASS: invalid migration contract failed during rehearsal"
else
  echo "FAIL: invalid migration contract unexpectedly succeeded"
fi

Expected:

PASS: invalid migration contract failed during rehearsal

Confirm backend is still running:

if docker ps --format '{{.Names}}' \
  | grep -qx "${ERS335_CONTAINER_PREFIX}-backend"
then
  echo "PASS: backend remained running after rehearsal failure"
else
  echo "FAIL: backend was stopped by a rehearsal-only failure"
fi

Expected:

PASS: backend remained running after rehearsal failure

Confirm health still passes:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-backend-health \
  ENV=development

Expected:

ers335-806f2eca-backend health check passed.

Confirm the pre-gate marker still exists:

docker exec "${ERS335_CONTAINER_PREFIX}-backend" \
  sqlite3 /app/data/app.db \
  "SELECT id || ':' || value FROM bite335_manual_pre_gate_probe;"

Expected:

pre-gate-1:must-survive-rollback

The old state must remain rolled_back; the failed rehearsal must not create a new armed state:

grep -qx 'status=rolled_back' "$ERS335_STATE_FILE" \
  && echo "PASS: failed rehearsal did not arm a new migration gate"

Expected:

PASS: failed rehearsal did not arm a new migration gate

Promotion blocker

If the application is stopped before rehearsal succeeds, do not promote.

15. Test 12 — Successful Existing-Database Deployment and Gate Commit

Identity / context

No ERS login required.

Purpose

Mirror the successful migrate-in-place deployment sequence.

Procedure

Load state:

source /tmp/ers-335-manual-session.env

Run the normal gate:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-migration-recovery-gate \
  ENV=development

Expected output includes:

Creating verified online pre-migration backup before taking the application offline.
Off-host backup verification passed:
Migration rehearsal passed:
Stopping the deployed application stack before live migration.
Migration/deployment recovery gate ARMED.

Confirm:

grep -qx 'status=armed' "$ERS335_STATE_FILE" \
  && grep -qx 'first_deployment=0' "$ERS335_STATE_FILE" \
  && echo "PASS: successful existing-database gate is armed"

Expected:

PASS: successful existing-database gate is armed

Record this gate directory:

export ERS335_COMMIT_GATE_DIR="$(
  grep '^gate_dir=' "$ERS335_STATE_FILE" | tail -1 | cut -d= -f2-
)"

printf "export ERS335_COMMIT_GATE_DIR='%s'\n" \
  "$ERS335_COMMIT_GATE_DIR" \
  >> /tmp/ers-335-manual-session.env

Start the new application revision:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-up \
  ENV=development

Run the same important deployment checks:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-backend-health \
  ENV=development

Expected:

ers335-806f2eca-backend health check passed.

Run:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-migrated-db-verify \
  ENV=development

Expected output includes:

Migrated database verification passed.
Final migration: 000077_journey_bonus_award_approval.up.sql
Integrity check: ok
Foreign key check: clean

Run:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-env-caddy-health \
  ENV=development

Expected:

ers335-806f2eca-caddy internal health check passed.

Confirm backend startup did not auto-migrate:

if docker logs "${ERS335_CONTAINER_PREFIX}-backend" 2>&1 \
  | grep -Fq \
    'APP_AUTO_MIGRATE=false: refusing implicit schema mutation; verifying deployed database before API startup.'
then
  echo "PASS: deployed backend verified schema instead of implicitly migrating"
else
  echo "FAIL: expected APP_AUTO_MIGRATE=false startup evidence is missing"
fi

Expected:

PASS: deployed backend verified schema instead of implicitly migrating

Commit:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-migration-recovery-commit \
  ENV=development

Expected:

Migration/deployment recovery gate COMMITTED.
Backend health, SQLite integrity/foreign keys, and migration history are verified.

Confirm state:

if grep -qx 'status=committed' "$ERS335_STATE_FILE" \
  && grep -q '^committed_at=' "$ERS335_STATE_FILE"
then
  echo "PASS: existing-database migration gate committed after deployment verification"
else
  echo "FAIL: existing-database migration gate did not commit"
fi

Expected:

PASS: existing-database migration gate committed after deployment verification

Attempt rollback after commit:

set +e

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-migration-recovery-rollback \
  ENV=development \
  > /tmp/ers-335-rollback-after-commit.out 2>&1

ERS335_ROLLBACK_AFTER_COMMIT_STATUS=$?

set -e

Display:

cat /tmp/ers-335-rollback-after-commit.out

Expected output includes:

Migration gate is not armed; current status: committed

Assert:

if [ "$ERS335_ROLLBACK_AFTER_COMMIT_STATUS" -ne 0 ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$ERS335_ROLLBACK_AFTER_COMMIT_STATUS"--ne-0-); then
  echo "PASS: committed migration gate cannot be rolled back accidentally"
else
  echo "FAIL: rollback unexpectedly succeeded after commit"
fi

Expected:

PASS: committed migration gate cannot be rolled back accidentally

16. Test 13 — Verify Live Database Health After Successful Commit

Identity / context

No ERS login required.

Procedure

Integrity check:

ERS335_FINAL_INTEGRITY="$(
  docker exec "${ERS335_CONTAINER_PREFIX}-backend" \
    sqlite3 /app/data/app.db 'PRAGMA integrity_check;'
)"

if [ "$ERS335_FINAL_INTEGRITY" == "ok" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$ERS335_FINAL_INTEGRITY"-==-"ok"-); then
  echo "PASS: committed database integrity_check is ok"
else
  printf 'FAIL: integrity_check returned: %s\n' "$ERS335_FINAL_INTEGRITY"
fi

Expected:

PASS: committed database integrity_check is ok

Foreign keys:

ERS335_FINAL_FOREIGN_KEYS="$(
  docker exec "${ERS335_CONTAINER_PREFIX}-backend" \
    sqlite3 /app/data/app.db 'PRAGMA foreign_key_check;'
)"

if [ -z "$ERS335_FINAL_FOREIGN_KEYS" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--z-"$ERS335_FINAL_FOREIGN_KEYS"-); then
  echo "PASS: committed database foreign_key_check is clean"
else
  echo "FAIL: committed database has foreign-key violations:"
  printf '%s\n' "$ERS335_FINAL_FOREIGN_KEYS"
fi

Expected:

PASS: committed database foreign_key_check is clean

Final migration:

ERS335_FINAL_MIGRATION_COUNT="$(
  docker exec "${ERS335_CONTAINER_PREFIX}-backend" \
    sqlite3 /app/data/app.db \
    "SELECT COUNT(*) FROM schema_migrations WHERE filename='000077_journey_bonus_award_approval.up.sql';"
)"

if [ "$ERS335_FINAL_MIGRATION_COUNT" == "1" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$ERS335_FINAL_MIGRATION_COUNT"-==-"1"-); then
  echo "PASS: committed database contains exactly one final migration record"
else
  printf 'FAIL: final migration record count is %s\n' \
    "$ERS335_FINAL_MIGRATION_COUNT"
fi

Expected:

PASS: committed database contains exactly one final migration record

Pre-gate logical marker:

ERS335_FINAL_PRE_GATE_COUNT="$(
  docker exec "${ERS335_CONTAINER_PREFIX}-backend" \
    sqlite3 /app/data/app.db \
    "SELECT COUNT(*) FROM bite335_manual_pre_gate_probe WHERE id='pre-gate-1' AND value='must-survive-rollback';"
)"

if [ "$ERS335_FINAL_PRE_GATE_COUNT" == "1" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/-"$ERS335_FINAL_PRE_GATE_COUNT"-==-"1"-); then
  echo "PASS: committed database retains expected pre-gate application data"
else
  echo "FAIL: expected pre-gate marker is missing after successful deployment"
fi

Expected:

PASS: committed database retains expected pre-gate application data

17. Test 14 — Re-Run Automated Recovery Gates

Identity / context

No ERS login required.

Procedure

From the repository root, run:

make production-environment-guardrails-check

Expected:

Bite 33.1 Production environment/destructive-operation guardrails verified.

Run:

make backup-creation-retention-verification-check

Expected successful Bite 33.2 completion.

Run:

make off-host-backup-protection-check

Expected:

Bite 33.3 off-host backup protection verified.

Run:

make restore-tooling-recovery-verification-check

Expected:

Bite 33.4 restore tooling and recovery verification verified with Go Production tooling.

Run:

make migration-deployment-recovery-gate-check

Expected:

Bite 33.5 migration/deployment recovery gate verified.

Finally:

make local-check

Expected: complete LOCAL gate passes.

Retain the already-passing CI result for this exact SHA as promotion evidence.

18. Test 15 — Retain Non-Secret Evidence and Clean Up Only the Isolated Environment

Identity / context

No ERS login required.

Procedure

Reload all persisted state:

source /tmp/ers-335-manual-session.env

Create evidence directory:

export ERS335_EVIDENCE_DIR="$HOME/Downloads/ers-bite-335-manual-evidence-806f2eca"

rm -rf "$ERS335_EVIDENCE_DIR"
mkdir -p "$ERS335_EVIDENCE_DIR"

Copy the final gate state:

cp \
  "$ERS335_STATE_FILE" \
  "$ERS335_EVIDENCE_DIR/final-migration-gate.state"

Copy the verified rollback report from Test 10:

if [ -s "$ERS335_ROLLBACK_GATE_DIR/deployment-rollback.json" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--s-"$ERS335_ROLLBACK_GATE_DIR/deployment-rollback.json"-); then
  cp \
    "$ERS335_ROLLBACK_GATE_DIR/deployment-rollback.json" \
    "$ERS335_EVIDENCE_DIR/"
  echo "PASS: deployment rollback report retained"
else
  echo "FAIL: deployment rollback report is unavailable"
fi

Copy the pre-migration S3 receipt:

if [ -s "$ERS335_BACKUP_FILE.offhost.json" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--s-"$ERS335_BACKUP_FILE.offhost.json"-); then
  cp \
    "$ERS335_BACKUP_FILE.offhost.json" \
    "$ERS335_EVIDENCE_DIR/pre-migration-backup.offhost.json"
  echo "PASS: pre-migration S3 receipt retained"
else
  echo "FAIL: pre-migration S3 receipt is unavailable"
fi

Write a non-secret summary:

cat > "$ERS335_EVIDENCE_DIR/summary.txt" <<EOF
Branch=943-bite-335-migration-deployment-recovery-gate
SHA=806f2ecab3db30612004e98de4dbb70c1ff436f2
ComposeProject=$ERS335_COMPOSE_PROJECT
ContainerPrefix=$ERS335_CONTAINER_PREFIX
S3Region=hel1
S3Endpoint=https://hel1.your-objectstorage.com
S3Bucket=ers
S3Prefix=$ERS335_S3_PREFIX
RollbackGateDir=$(basename "$ERS335_ROLLBACK_GATE_DIR")
RollbackSourceBackup=$(basename "$ERS335_BACKUP_FILE")
RollbackSourceSHA256=$ERS335_SOURCE_SHA256
DatabaseObjectKey=$ERS335_DB_KEY
DatabaseVersionId=$ERS335_DB_VERSION_ID
ManifestObjectKey=$ERS335_MANIFEST_KEY
ManifestVersionId=$ERS335_MANIFEST_VERSION_ID
FinalState=$(grep '^status=' "$ERS335_STATE_FILE" | cut -d= -f2-)
FinalExpectedMigration=$(grep '^expected_final_migration=' "$ERS335_STATE_FILE" | cut -d= -f2-)
EOF

Inspect:

cat "$ERS335_EVIDENCE_DIR/summary.txt"

Check retained evidence for obvious secrets:

if grep -R -Eiq \
  'aws_secret_access_key|PRIVATE KEY|JWT_SECRET|LEDGER_CORRECTION_KEY|LEDGER_SETTLEMENT_KEY|OPENAI_API_KEY' \
  "$ERS335_EVIDENCE_DIR"
then
  echo "FAIL: retained evidence appears to contain secret material"
else
  echo "PASS: retained evidence contains no obvious secret material"
fi

Expected:

PASS: retained evidence contains no obvious secret material

Stop and delete only the dedicated Bite 33.5 Compose resources:

make \
  SERVER_ROOT="$ERS335_SERVER_ROOT" \
  COMPOSE_PROJECT="$ERS335_COMPOSE_PROJECT" \
  CONTAINER_PREFIX="$ERS335_CONTAINER_PREFIX" \
  server-down-volumes \
  ENV=development

Verify no dedicated containers remain:

ERS335_REMAINING_CONTAINERS="$(
  docker ps -a --format '{{.Names}}' \
    | grep -E "^${ERS335_CONTAINER_PREFIX}-(backend|frontend|caddy)$" \
    || true
)"

if [ -z "$ERS335_REMAINING_CONTAINERS" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--z-"$ERS335_REMAINING_CONTAINERS"-); then
  echo "PASS: Bite 33.5 manual-test containers removed"
else
  echo "FAIL: Bite 33.5 manual-test containers still exist:"
  printf '%s\n' "$ERS335_REMAINING_CONTAINERS"
fi

Expected:

PASS: Bite 33.5 manual-test containers removed

Verify no dedicated volumes remain:

ERS335_REMAINING_VOLUMES="$(
  docker volume ls --format '{{.Name}}' \
    | grep "^${ERS335_COMPOSE_PROJECT}_" \
    || true
)"

if [ -z "$ERS335_REMAINING_VOLUMES" ](/RodrigoMattosoSilveira/EnterpriseRemoteSystems/wiki/--z-"$ERS335_REMAINING_VOLUMES"-); then
  echo "PASS: Bite 33.5 manual-test volumes removed"
else
  echo "FAIL: Bite 33.5 manual-test volumes still exist:"
  printf '%s\n' "$ERS335_REMAINING_VOLUMES"
fi

Expected:

PASS: Bite 33.5 manual-test volumes removed

For information only, show the normal Development deployment:

docker ps -a --format '{{.Names}}' \
  | grep -E '^ers-dev-(backend|frontend|caddy)$' \
  || true

Existing ers-dev-* names are expected and are not a failure.

Remove the disposable filesystem tree and temporary credentials copy:

rm -rf "$ERS335_SERVER_ROOT"
rm -rf "$ERS335_SECRET_DIR"
rm -f /tmp/ers-335-manual-session.env

Do not attempt to bypass Hetzner Object Lock to delete objects under:

$ERS335_S3_PREFIX

Those objects may remain until normal retention expiration.

PASS criteria

The dedicated Docker resources and temporary credential copy are removed, the normal Development deployment remains untouched, and non-secret evidence is retained in Downloads.

19. Promotion Gate

Promote Bite 33.5 only when every item below is true:

  • Branch is exactly 943-bite-335-migration-deployment-recovery-gate.
  • HEAD is exactly 806f2ecab3db30612004e98de4dbb70c1ff436f2.
  • Existing ers-dev-* resources were not reused or deleted.
  • Isolated ers335-806f2eca images build successfully.
  • First-deployment gate creates a database with final migration 000077.
  • First-deployment gate records status=armed.
  • First-deployment gate records first_deployment=1.
  • First-deployment gate has no rollback source.
  • Application remains stopped while the first gate is armed.
  • A second gate is refused while an earlier gate is armed.
  • Commit is refused while backend is stopped.
  • Backend starts successfully with APP_AUTO_MIGRATE=false.
  • Backend logs prove it verifies schema rather than implicitly migrating.
  • First-deployment gate commits only after health/schema verification.
  • Existing-database gate creates a verified pre-migration backup.
  • Existing-database gate creates the backup while the running backend is still online.
  • S3 replication succeeds before live migration.
  • S3 receipt records exact DB and manifest VersionIds.
  • Exact DB VersionId exists in Hetzner.
  • Exact manifest VersionId exists in Hetzner.
  • Bite 33.4 rollback candidate is staged and byte-identical to the pre-migration backup.
  • Isolated migration-probe evidence exists.
  • Migration-probe SHA matches gate evidence.
  • Full application stack is stopped before live migration is considered complete.
  • Post-deployment rollback restores the exact pre-migration database bytes while stopped.
  • Rollback preserves pre-migration logical state.
  • Rollback removes post-deployment logical state.
  • Rollback leaves application services stopped.
  • Application restarts only after explicit operator action.
  • Invalid expected migration fails during isolated rehearsal.
  • Rehearsal failure leaves the currently running backend available.
  • Rehearsal failure does not arm a new gate.
  • Successful existing-database gate starts the new revision and passes backend health.
  • Successful existing-database gate passes migrated database verification.
  • Successful existing-database gate passes Caddy/internal environment health.
  • Successful gate commit records status=committed.
  • Rollback is refused after commit.
  • Final live database integrity check is ok.
  • Final foreign-key check is clean.
  • Final migration 000077_journey_bonus_award_approval.up.sql is recorded exactly once.
  • Bite 33.1 regression passes.
  • Bite 33.2 regression passes.
  • Bite 33.3 regression passes.
  • Bite 33.4 regression passes.
  • Bite 33.5 regression passes.
  • make local-check passes.
  • CI passes for this exact SHA.

20. Promotion-Blocking Defects

Do not promote Bite 33.5 if any of the following occurs:

  • backend startup mutates schema while APP_AUTO_MIGRATE=false;
  • a live database is migrated before the migration rehearsal succeeds;
  • a pre-migration backup is missing or fails verification;
  • enabled off-host protection is skipped;
  • migration proceeds before off-host replication succeeds;
  • the rollback candidate is missing or differs from the pre-migration source;
  • a second gate can replace an already-armed gate;
  • commit succeeds while backend is stopped or unhealthy;
  • commit succeeds with SQLite integrity or foreign-key failures;
  • commit succeeds with an incomplete migration chain;
  • the application remains running during live SQLite migration;
  • rollback does not stop the application stack;
  • rollback restores anything other than the exact staged pre-migration source;
  • post-deployment state survives rollback when it was absent from the source;
  • pre-migration application data disappears after rollback;
  • rollback automatically restarts the new application revision;
  • a rehearsal failure stops a healthy running deployment before rehearsal succeeds;
  • a failed rehearsal writes a new armed state;
  • rollback is permitted after a committed gate;
  • S3 receipt lacks exact VersionIds;
  • retained recovery evidence contains credentials/secrets;
  • any 33.1–33.5 automated recovery regression fails;
  • LOCAL or CI fails.

21. Evidence to Retain

Retain these non-secret values:

Branch:
SHA:

Dedicated server root:
Dedicated Compose project:
Dedicated container prefix:

Hetzner region:
Hetzner endpoint:
Hetzner bucket:
Manual S3 prefix:

First-deployment gate:
  status:
  first_deployment:
  expected_final_migration:

Existing-database rollback gate:
  gate directory:
  pre-migration backup filename:
  pre-migration backup SHA-256:
  staged rollback candidate:
  migration-probe SHA-256:
  database S3 key:
  database S3 VersionId:
  manifest S3 key:
  manifest S3 VersionId:
  rollback report:

Preflight failure:
  exit status:
  backend remained running:
  health passed:

Final successful gate:
  status:
  committed_at:
  expected_final_migration:

Automated checks:
  production-environment-guardrails-check:
  backup-creation-retention-verification-check:
  off-host-backup-protection-check:
  restore-tooling-recovery-verification-check:
  migration-deployment-recovery-gate-check:
  local-check:
  CI:

Do not retain:

  • AWS secret access key;
  • AWS credentials file;
  • JWT secret;
  • ledger correction/settlement keys;
  • passwords;
  • OpenAI API keys.

22. Glossary Impact

none