Bite 33.3 — Off-Host Backup Protection

Revised Manual Verification Checklist — Hetzner Object Storage / S3

Branch: 935-bite-333-off-host-backup-protection
Validated HEAD SHA for this revision: 589773522e9351ba93212878026074328ce42d6a

Execution prerequisite

This checklist is the manual acceptance contract for the S3-enabled revision of Bite 33.3. Do not execute the ERS-specific S3 steps against the earlier SSH-only implementation. Before promotion, record the exact S3-enabled HEAD SHA used for testing.

1. Purpose

Bite 33.3 must prove that a verified Bite 33.2 backup can leave the ERS application host, be stored in Hetzner Object Storage through its S3-compatible API, be downloaded back, and pass the complete Bite 33.2 verification contract.

The Production design adopted for ERS is:

ERS Production host
    │
    ├── live SQLite database
    │
    ├── Bite 33.2 verified local backup
    │       ├── app-<timestamp>.db
    │       └── app-<timestamp>.db.manifest.json
    │
    └── Bite 33.3 S3 replication
            ↓
      Hetzner Object Storage
      private bucket
      versioning enabled
      Object Lock enabled
            │
            ├── backup database object
            ├── manifest object
            └── off-host receipt object
            ↓
      exact S3 object versions downloaded again
            ↓
      SHA-256 comparison
            ↓
      full Bite 33.2 SQLite/schema/data verification

A successful manual verification must prove all of the following:

the source backup and manifest are valid before replication;
the Hetzner bucket is private;
bucket Versioning is enabled;
Object Lock is enabled;
a default Governance retention policy protects newly uploaded backup objects;
the database and manifest are uploaded together under the ERS backup prefix;
ERS records the exact S3 VersionId for the database and manifest;
the remote database SHA-256 exactly matches the source database SHA-256;
the remote manifest SHA-256 exactly matches the source manifest SHA-256;
the S3 ETag is not used as a substitute for SHA-256;
ERS downloads the recorded database and manifest versions back and verifies them;
ERS creates an off-host receipt locally and in Object Storage;
the receipt contains recovery evidence but not the S3 secret key;
off-host verification succeeds even when the local backup database and manifest are temporarily unavailable;
retrying replication of the same accepted backup does not silently create replacement database/manifest versions;
an exact retained backup version cannot be deleted normally while Object Lock retention is active;
the original known-good remote backup remains verifiable after all negative tests;
Production refuses to run with off-host protection disabled or with a non-S3 transport after ERS adopts Hetzner Object Storage as its Production backup transport.

2. Hetzner-Specific Contract Used by This Suite

Hetzner Object Storage currently exposes location-specific S3-compatible endpoints:

fsn1 → https://fsn1.your-objectstorage.com
nbg1 → https://nbg1.your-objectstorage.com
hel1 → https://hel1.your-objectstorage.com

For this ERS Production design:

the Bucket must be private;
Object Lock must be enabled when the Bucket is created;
Versioning must be enabled;
the default Object Lock retention used by this test is:
- Mode: GOVERNANCE
- Days: 30
no lifecycle rule is required by Bite 33.3;
S3 credentials must not be committed to the ERS repository;
the AWS CLI is used only as an independent operator verification tool.

3. Expected ERS S3 Configuration Contract

The S3-enabled Bite 33.3 implementation should preserve:

make server-offhost-backup
make server-offhost-backup-verify

and select S3 through configuration equivalent to:

SERVER_OFFHOST_BACKUP_ENABLED=true
SERVER_OFFHOST_BACKUP_TRANSPORT=s3

SERVER_OFFHOST_BACKUP_S3_ENDPOINT=https://hel1.your-objectstorage.com
SERVER_OFFHOST_BACKUP_S3_REGION=hel1
SERVER_OFFHOST_BACKUP_S3_BUCKET=<configured-bucket>
SERVER_OFFHOST_BACKUP_S3_PREFIX=ers-backups

AWS_SHARED_CREDENTIALS_FILE=<absolute-path-outside-source-control>
AWS_PROFILE=ers-backup

The expected S3 object layout is:

<ERS-S3-PREFIX>/
└── <environment>/
    └── <backup-filename>/
        ├── <backup-filename>
        ├── <backup-filename>.manifest.json
        └── <backup-filename>.offhost.json

The off-host receipt must record at minimum:

transport = s3
endpoint
region
bucket
database object key
database VersionId
database SHA-256
manifest object key
manifest VersionId
manifest SHA-256
verification timestamp
verification method

4. Safety Rules

These tests must not use:

the Production database;
the Test database;
Production backup objects;
a shared object prefix containing customer backups.

The source database is:

backend/data/app.db

The test uses a unique prefix:

manual-validation/<UTC-timestamp>

The bucket itself is not deleted.

Because Object Lock retention is deliberately tested, some test object versions can remain until retention expires.

Do not use --bypass-governance-retention merely to clean up test objects early.

5. Test 00 — Confirm Branch, S3-Enabled HEAD, and Required Tools

Run from the ERS repository root:

git branch --show-current
git rev-parse HEAD

Expected branch:

935-bite-333-off-host-backup-protection

Record:

S3-enabled Bite 33.3 HEAD:

Confirm tools:

for command in sqlite3 aws jq shasum python3 make; do
  command -v "$command" >/dev/null 2>&1     && echo "PASS: $command is available"     || echo "FAIL: $command is missing"
done

Create workspace:

rm -rf /tmp/ers-333-s3-manual
mkdir -p /tmp/ers-333-s3-manual/aws
mkdir -p /tmp/ers-333-s3-manual/local
mkdir -p /tmp/ers-333-s3-manual/downloaded
mkdir -p /tmp/ers-333-s3-manual/server-root/development
chmod 700 /tmp/ers-333-s3-manual/aws

Verify:

test -d /tmp/ers-333-s3-manual/aws   && test -d /tmp/ers-333-s3-manual/local   && test -d /tmp/ers-333-s3-manual/downloaded   && echo "PASS: Bite 33.3 S3 workspace created"

Expected:

PASS: Bite 33.3 S3 workspace created

PASS: correct branch, S3-enabled SHA recorded, all tools available, workspace created.

6. Test 01 — Configure Hetzner S3 Credentials Once

In Hetzner Console, create or select an S3 access-key/secret-key pair authorized to use the ERS backup bucket.

Run in zsh:

read "ERS333_S3_REGION?Enter Hetzner Object Storage region (fsn1, nbg1, or hel1): "

Validate:

case "$ERS333_S3_REGION" in
  fsn1|nbg1|hel1)
    echo "PASS: supported Hetzner Object Storage region selected"
    ;;
  *)
    echo "FAIL: region must be fsn1, nbg1, or hel1"
    exit 2
    ;;
esac

Derive endpoint:

export ERS333_S3_ENDPOINT="https://${ERS333_S3_REGION}.your-objectstorage.com"
printf 'Hetzner S3 endpoint: %s\n' "$ERS333_S3_ENDPOINT"

Enter bucket and credentials:

read "ERS333_S3_BUCKET?Enter the private Hetzner Object Storage bucket name for ERS backups: "
read "ERS333_S3_ACCESS_KEY?Enter the Hetzner S3 access key: "
read -s "ERS333_S3_SECRET_KEY?Enter the Hetzner S3 secret key: "
printf '\n'

Create temporary AWS credentials:

cat > /tmp/ers-333-s3-manual/aws/credentials <<EOF
[ers-backup]
aws_access_key_id=$ERS333_S3_ACCESS_KEY
aws_secret_access_key=$ERS333_S3_SECRET_KEY
EOF

chmod 600 /tmp/ers-333-s3-manual/aws/credentials
unset ERS333_S3_SECRET_KEY

Create AWS config:

cat > /tmp/ers-333-s3-manual/aws/config <<EOF
[profile ers-backup]
region=$ERS333_S3_REGION
output=json
EOF

chmod 600 /tmp/ers-333-s3-manual/aws/config

Set session:

export AWS_SHARED_CREDENTIALS_FILE="/tmp/ers-333-s3-manual/aws/credentials"
export AWS_CONFIG_FILE="/tmp/ers-333-s3-manual/aws/config"
export AWS_PROFILE="ers-backup"
export AWS_REGION="$ERS333_S3_REGION"
export ERS333_S3_PREFIX="manual-validation/$(date -u +%Y%m%dT%H%M%SZ)"

Persist only non-secret values:

cat > /tmp/ers-333-s3-manual/session.env <<EOF
export ERS333_S3_REGION='$ERS333_S3_REGION'
export ERS333_S3_ENDPOINT='$ERS333_S3_ENDPOINT'
export ERS333_S3_BUCKET='$ERS333_S3_BUCKET'
export ERS333_S3_PREFIX='$ERS333_S3_PREFIX'
export AWS_SHARED_CREDENTIALS_FILE='/tmp/ers-333-s3-manual/aws/credentials'
export AWS_CONFIG_FILE='/tmp/ers-333-s3-manual/aws/config'
export AWS_PROFILE='ers-backup'
export AWS_REGION='$ERS333_S3_REGION'
EOF

Test access:

aws s3api head-bucket   --bucket "$ERS333_S3_BUCKET"   --endpoint-url "$ERS333_S3_ENDPOINT"

Expected exit code:

echo $?

If the bucket does not yet exist, create it once with Object Lock enabled:

aws s3api create-bucket   --bucket "$ERS333_S3_BUCKET"   --region "$ERS333_S3_REGION"   --object-lock-enabled-for-bucket   --endpoint-url "$ERS333_S3_ENDPOINT"

Then repeat head-bucket.

PASS: credentials work, secret is outside repository and not in session.env, bucket is reachable.

7. Test 02 — Verify Versioning and Object Lock

Load:

source /tmp/ers-333-s3-manual/session.env

Enable versioning:

aws s3api put-bucket-versioning   --bucket "$ERS333_S3_BUCKET"   --versioning-configuration Status=Enabled   --endpoint-url "$ERS333_S3_ENDPOINT"

Assert:

test "$(
  aws s3api get-bucket-versioning     --bucket "$ERS333_S3_BUCKET"     --endpoint-url "$ERS333_S3_ENDPOINT"     | jq -r '.Status'
)" = "Enabled"   && echo "PASS: bucket versioning is enabled"

Expected:

PASS: bucket versioning is enabled

Verify Object Lock:

aws s3api get-object-lock-configuration   --bucket "$ERS333_S3_BUCKET"   --endpoint-url "$ERS333_S3_ENDPOINT"   | jq .

If this fails because Object Lock is absent, STOP. Create a new bucket with Object Lock enabled; Hetzner cannot add Object Lock retroactively.

Configure 30-day Governance retention:

aws s3api put-object-lock-configuration   --bucket "$ERS333_S3_BUCKET"   --object-lock-configuration '{
  "ObjectLockEnabled": "Enabled",
  "Rule": {
    "DefaultRetention": {
      "Mode": "GOVERNANCE",
      "Days": 30
    }
  }
}'   --endpoint-url "$ERS333_S3_ENDPOINT"

Assert:

aws s3api get-object-lock-configuration   --bucket "$ERS333_S3_BUCKET"   --endpoint-url "$ERS333_S3_ENDPOINT"   | jq -e '
      .ObjectLockConfiguration.ObjectLockEnabled == "Enabled"
      and .ObjectLockConfiguration.Rule.DefaultRetention.Mode == "GOVERNANCE"
      and .ObjectLockConfiguration.Rule.DefaultRetention.Days == 30
    '

Expected:

echo $?

PASS: Versioning enabled, Object Lock enabled, Governance 30 days.

8. Test 03 — Confirm Bucket Is Not Public

Run:

source /tmp/ers-333-s3-manual/session.env

aws s3api get-bucket-acl   --bucket "$ERS333_S3_BUCKET"   --endpoint-url "$ERS333_S3_ENDPOINT"   > /tmp/ers-333-s3-manual/bucket-acl.json

Check:

if jq -e '
  [.Grants[]? |
    select(
      .Grantee.URI == "http://acs.amazonaws.com/groups/global/AllUsers"
      or
      .Grantee.URI == "http://acs.amazonaws.com/groups/global/AuthenticatedUsers"
    )
  ] | length > 0
' /tmp/ers-333-s3-manual/bucket-acl.json >/dev/null; then
  echo "FAIL: bucket ACL contains a public group grant"
else
  echo "PASS: bucket ACL contains no public group grant"
fi

Expected:

PASS: bucket ACL contains no public group grant

9. Test 04 — Start the Normal LOCAL Backend

Terminal 1:

unset APP_ENV
make local-backend

Keep it running.

Terminal 2:

test -f backend/data/app.db   && echo "PASS: normal LOCAL database exists"

Expected:

PASS: normal LOCAL database exists

10. Test 05 — Create and Verify Source Bite 33.2 Backup

Load:

source /tmp/ers-333-s3-manual/session.env

Create:

export ERS333_GOOD_BACKUP="/tmp/ers-333-s3-manual/local/app-$(date -u +%Y%m%dT%H%M%SZ).db"

sqlite3 backend/data/app.db ".backup '$ERS333_GOOD_BACKUP'"

Confirm:

test -s "$ERS333_GOOD_BACKUP"   && echo "PASS: source backup exists and is non-empty"

Create manifest:

./scripts/run-backup-go-tool.sh ers-backup create-manifest \
  --backup "$ERS333_GOOD_BACKUP" \
  --environment development \
  --source-container manual-local-backend \
  --source-database-path "$(pwd)/backend/data/app.db"

Verify:

./scripts/run-backup-go-tool.sh ers-backup verify \
  --backup "$ERS333_GOOD_BACKUP" \
  --manifest "$ERS333_GOOD_BACKUP.manifest.json" \
  --expected-environment development

Expected:

Backup verification passed

Record SHA-256:

export ERS333_GOOD_BACKUP_SHA256="$(
  shasum -a 256 "$ERS333_GOOD_BACKUP" | awk '{print $1}'
)"

export ERS333_GOOD_MANIFEST_SHA256="$(
  shasum -a 256 "$ERS333_GOOD_BACKUP.manifest.json" | awk '{print $1}'
)"

Persist:

{
  printf "export ERS333_GOOD_BACKUP='%s'\n" "$ERS333_GOOD_BACKUP"
  printf "export ERS333_GOOD_BACKUP_SHA256='%s'\n" "$ERS333_GOOD_BACKUP_SHA256"
  printf "export ERS333_GOOD_MANIFEST_SHA256='%s'\n" "$ERS333_GOOD_MANIFEST_SHA256"
} >> /tmp/ers-333-s3-manual/session.env

PASS: source backup/manifest verify and both SHA-256 values are recorded.

11. Test 06 — Create ERS Development S3 Configuration

Load:

source /tmp/ers-333-s3-manual/session.env

Create:

cat > /tmp/ers-333-s3-manual/server-root/development/.env.development <<EOF
APP_ENV=development

SERVER_OFFHOST_BACKUP_ENABLED=true
SERVER_OFFHOST_BACKUP_TRANSPORT=s3

SERVER_OFFHOST_BACKUP_S3_ENDPOINT=$ERS333_S3_ENDPOINT
SERVER_OFFHOST_BACKUP_S3_REGION=$ERS333_S3_REGION
SERVER_OFFHOST_BACKUP_S3_BUCKET=$ERS333_S3_BUCKET
SERVER_OFFHOST_BACKUP_S3_PREFIX=$ERS333_S3_PREFIX

AWS_SHARED_CREDENTIALS_FILE=/tmp/ers-333-s3-manual/aws/credentials
AWS_PROFILE=ers-backup
EOF

Confirm secret is not embedded:

if grep -q 'aws_secret_access_key'   /tmp/ers-333-s3-manual/server-root/development/.env.development; then
  echo "FAIL: secret key was embedded in ERS environment file"
else
  echo "PASS: ERS environment file references credentials without embedding the secret key"
fi

Expected:

PASS: ERS environment file references credentials without embedding the secret key

12. Test 07 — Replicate Verified Backup to Hetzner S3

make server-offhost-backup is a deployed-server target. It sets ERS_BACKUP_ENV_DIR and expects the selected server environment directory to contain docker-compose.server.yml. This manual suite deliberately uses a synthetic LOCAL environment directory only for S3 configuration, so the LOCAL test must invoke the Go backup runner directly.

Run:

source /tmp/ers-333-s3-manual/session.env

./scripts/run-backup-go-tool.sh ers-offhost-backup replicate \
  --environment development \
  --env-file /tmp/ers-333-s3-manual/server-root/development/.env.development \
  --backup "$ERS333_GOOD_BACKUP"

Expected:

exit 0;
S3 round-trip verification success;
local receipt at <backup>.offhost.json.

Verify:

test -s "$ERS333_GOOD_BACKUP.offhost.json"   && echo "PASS: local S3 off-host receipt exists"

Expected:

PASS: local S3 off-host receipt exists

13. Test 08 — Inspect Receipt and Record Exact S3 VersionIds

Display:

jq . "$ERS333_GOOD_BACKUP.offhost.json"

Assert:

jq -e '
  .status == "verified"
  and .transport == "s3"
  and .remote.endpoint != null
  and .remote.region != null
  and .remote.bucket != null
  and .remote.backup.key != null
  and .remote.backup.version_id != null
  and .remote.backup.sha256 != null
  and .remote.manifest.key != null
  and .remote.manifest.version_id != null
  and .remote.manifest.sha256 != null
' "$ERS333_GOOD_BACKUP.offhost.json"

Expected:

echo $?

Capture:

export ERS333_DB_KEY="$(jq -r '.remote.backup.key' "$ERS333_GOOD_BACKUP.offhost.json")"
export ERS333_DB_VERSION_ID="$(jq -r '.remote.backup.version_id' "$ERS333_GOOD_BACKUP.offhost.json")"
export ERS333_MANIFEST_KEY="$(jq -r '.remote.manifest.key' "$ERS333_GOOD_BACKUP.offhost.json")"
export ERS333_MANIFEST_VERSION_ID="$(jq -r '.remote.manifest.version_id' "$ERS333_GOOD_BACKUP.offhost.json")"

Persist:

{
  printf "export ERS333_DB_KEY='%s'\n" "$ERS333_DB_KEY"
  printf "export ERS333_DB_VERSION_ID='%s'\n" "$ERS333_DB_VERSION_ID"
  printf "export ERS333_MANIFEST_KEY='%s'\n" "$ERS333_MANIFEST_KEY"
  printf "export ERS333_MANIFEST_VERSION_ID='%s'\n" "$ERS333_MANIFEST_VERSION_ID"
} >> /tmp/ers-333-s3-manual/session.env

Secret check:

if grep -q 'aws_secret_access_key' "$ERS333_GOOD_BACKUP.offhost.json"; then
  echo "FAIL: receipt contains S3 secret-key material"
else
  echo "PASS: receipt contains no S3 secret-key material"
fi

Expected:

PASS: receipt contains no S3 secret-key material

Promotion blocker: receipt has object keys but no VersionIds.

14. Test 09 — Independently Download Exact Recorded S3 Versions

The Bite 33.2 manifest intentionally records the original backup filename. Therefore the independently downloaded database and manifest must retain their original filenames. Do not rename the database to backup.db.

Load:

source /tmp/ers-333-s3-manual/session.env

Derive the original backup filename from the verified local backup path:

export ERS333_BACKUP_NAME="$(basename "$ERS333_GOOD_BACKUP")"
export ERS333_DOWNLOADED_BACKUP="/tmp/ers-333-s3-manual/downloaded/$ERS333_BACKUP_NAME"
export ERS333_DOWNLOADED_MANIFEST="$ERS333_DOWNLOADED_BACKUP.manifest.json"

printf 'Downloaded DB path:       %s\n' "$ERS333_DOWNLOADED_BACKUP"
printf 'Downloaded manifest path: %s\n' "$ERS333_DOWNLOADED_MANIFEST"

Expected paths resemble:

/tmp/ers-333-s3-manual/downloaded/app-20261009T201450Z.db
/tmp/ers-333-s3-manual/downloaded/app-20261009T201450Z.db.manifest.json

Remove any stale copies from a previous Test 09 attempt:

rm -f \
  "$ERS333_DOWNLOADED_BACKUP" \
  "$ERS333_DOWNLOADED_MANIFEST"

Download the exact recorded database VersionId while preserving its original filename:

aws s3api get-object \
  --bucket "$ERS333_S3_BUCKET" \
  --key "$ERS333_DB_KEY" \
  --version-id "$ERS333_DB_VERSION_ID" \
  --endpoint-url "$ERS333_S3_ENDPOINT" \
  "$ERS333_DOWNLOADED_BACKUP" \
  > /tmp/ers-333-s3-manual/downloaded/backup-get.json

Download the exact recorded manifest VersionId:

aws s3api get-object \
  --bucket "$ERS333_S3_BUCKET" \
  --key "$ERS333_MANIFEST_KEY" \
  --version-id "$ERS333_MANIFEST_VERSION_ID" \
  --endpoint-url "$ERS333_S3_ENDPOINT" \
  "$ERS333_DOWNLOADED_MANIFEST" \
  > /tmp/ers-333-s3-manual/downloaded/manifest-get.json

Confirm both downloaded files exist and are non-empty:

test -s "$ERS333_DOWNLOADED_BACKUP" \
  && echo "PASS: exact S3 database version downloaded"

test -s "$ERS333_DOWNLOADED_MANIFEST" \
  && echo "PASS: exact S3 manifest version downloaded"

Expected:

PASS: exact S3 database version downloaded
PASS: exact S3 manifest version downloaded

Compute independent SHA-256 values:

ERS333_DOWNLOADED_DB_SHA256="$(
  shasum -a 256 "$ERS333_DOWNLOADED_BACKUP" | awk '{print $1}'
)"

ERS333_DOWNLOADED_MANIFEST_SHA256="$(
  shasum -a 256 "$ERS333_DOWNLOADED_MANIFEST" | awk '{print $1}'
)"

printf 'Source DB SHA-256:       %s\n' "$ERS333_GOOD_BACKUP_SHA256"
printf 'Downloaded DB SHA-256:   %s\n' "$ERS333_DOWNLOADED_DB_SHA256"
printf 'Source manifest SHA-256: %s\n' "$ERS333_GOOD_MANIFEST_SHA256"
printf 'Downloaded manifest SHA: %s\n' "$ERS333_DOWNLOADED_MANIFEST_SHA256"

Compare:

test "$ERS333_DOWNLOADED_DB_SHA256" = "$ERS333_GOOD_BACKUP_SHA256" \
  && echo "PASS: exact S3 database version matches source SHA-256"

test "$ERS333_DOWNLOADED_MANIFEST_SHA256" = "$ERS333_GOOD_MANIFEST_SHA256" \
  && echo "PASS: exact S3 manifest version matches source SHA-256"

Expected:

PASS: exact S3 database version matches source SHA-256
PASS: exact S3 manifest version matches source SHA-256

Now run the full Bite 33.2 Go verification against the independently downloaded pair:

./scripts/run-backup-go-tool.sh ers-backup verify \
  --backup "$ERS333_DOWNLOADED_BACKUP" \
  --manifest "$ERS333_DOWNLOADED_MANIFEST" \
  --expected-environment development

Expected:

Backup verification passed

Immediately run:

echo $?

Expected:

Important: do not use the S3 ETag as the database checksum. The PASS condition is based on independently computed SHA-256 values.

PASS: the exact recorded S3 database and manifest versions preserve the original filenames, reproduce the original SHA-256 values, and pass the complete Bite 33.2 verification contract.

15. Test 10 — Verify Object Lock on Exact DB Version

Inspect retention:

aws s3api get-object-retention   --bucket "$ERS333_S3_BUCKET"   --key "$ERS333_DB_KEY"   --version-id "$ERS333_DB_VERSION_ID"   --endpoint-url "$ERS333_S3_ENDPOINT"   | jq .

Assert Governance:

test "$(
  aws s3api get-object-retention     --bucket "$ERS333_S3_BUCKET"     --key "$ERS333_DB_KEY"     --version-id "$ERS333_DB_VERSION_ID"     --endpoint-url "$ERS333_S3_ENDPOINT"     | jq -r '.Retention.Mode'
)" = "GOVERNANCE"   && echo "PASS: DB version is protected by Governance retention"

Attempt ordinary exact-version delete:

set +e

aws s3api delete-object   --bucket "$ERS333_S3_BUCKET"   --key "$ERS333_DB_KEY"   --version-id "$ERS333_DB_VERSION_ID"   --endpoint-url "$ERS333_S3_ENDPOINT"   >/tmp/ers-333-s3-manual/delete-protected.out 2>&1

ERS333_DELETE_STATUS=$?

set -e

Assert:

test "$ERS333_DELETE_STATUS" -ne 0   && echo "PASS: retained DB version cannot be deleted normally"

Expected:

PASS: retained DB version cannot be deleted normally

Re-download exact version and compare SHA-256.

16. Test 11 — Verify Off-Host Recovery Without Local DB/Manifest

This test deliberately moves the local .db and .manifest.json away from their normal paths. The final restoration steps are mandatory before Test 12.

Load the persisted session:

source /tmp/ers-333-s3-manual/session.env

Derive the original filename again. This is intentionally recomputed so the test remains correct even after restarting Terminal:

export ERS333_BACKUP_NAME="$(basename "$ERS333_GOOD_BACKUP")"
export ERS333_HIDDEN_BACKUP="/tmp/ers-333-s3-manual/local-hidden/$ERS333_BACKUP_NAME"
export ERS333_HIDDEN_MANIFEST="$ERS333_HIDDEN_BACKUP.manifest.json"

Create the holding directory:

mkdir -p /tmp/ers-333-s3-manual/local-hidden

Move the local pair away from its normal location:

mv "$ERS333_GOOD_BACKUP" "$ERS333_HIDDEN_BACKUP"
mv "$ERS333_GOOD_BACKUP.manifest.json" "$ERS333_HIDDEN_MANIFEST"

Confirm both normal paths are absent:

test ! -e "$ERS333_GOOD_BACKUP" \
  && test ! -e "$ERS333_GOOD_BACKUP.manifest.json" \
  && echo "PASS: local backup pair is unavailable"

Expected:

PASS: local backup pair is unavailable

Verify the remote exact versions using only the retained off-host receipt:

./scripts/run-backup-go-tool.sh ers-offhost-backup verify-replica \
  --environment development \
  --env-file /tmp/ers-333-s3-manual/server-root/development/.env.development \
  --receipt "$ERS333_GOOD_BACKUP.offhost.json"

Expected output includes:

Off-host replica verification passed:

Immediately run:

echo $?

Expected:

Mandatory restoration before Test 12

If Terminal has been restarted since the files were moved, reload the session and recompute the filenames first:

source /tmp/ers-333-s3-manual/session.env

export ERS333_BACKUP_NAME="$(basename "$ERS333_GOOD_BACKUP")"
export ERS333_HIDDEN_BACKUP="/tmp/ers-333-s3-manual/local-hidden/$ERS333_BACKUP_NAME"
export ERS333_HIDDEN_MANIFEST="$ERS333_HIDDEN_BACKUP.manifest.json"

Show the current state before restoring:

printf 'Normal backup:    %s\n' "$ERS333_GOOD_BACKUP"
printf 'Hidden backup:    %s\n' "$ERS333_HIDDEN_BACKUP"
printf 'Normal manifest:  %s\n' "$ERS333_GOOD_BACKUP.manifest.json"
printf 'Hidden manifest:  %s\n' "$ERS333_HIDDEN_MANIFEST"

ls -l \
  "$ERS333_HIDDEN_BACKUP" \
  "$ERS333_HIDDEN_MANIFEST"

Restore the database:

if [[ ! -e "$ERS333_GOOD_BACKUP" && -f "$ERS333_HIDDEN_BACKUP" ]]; then
  mv "$ERS333_HIDDEN_BACKUP" "$ERS333_GOOD_BACKUP"
fi

Restore the manifest:

if [[ ! -e "$ERS333_GOOD_BACKUP.manifest.json" && -f "$ERS333_HIDDEN_MANIFEST" ]]; then
  mv "$ERS333_HIDDEN_MANIFEST" "$ERS333_GOOD_BACKUP.manifest.json"
fi

Assert that both files are back in their original locations:

test -s "$ERS333_GOOD_BACKUP" \
  && echo "PASS: local backup restored"

test -s "$ERS333_GOOD_BACKUP.manifest.json" \
  && echo "PASS: local manifest restored"

Expected:

PASS: local backup restored
PASS: local manifest restored

Confirm the restored bytes still match the SHA-256 values recorded before off-host replication:

test "$(
  shasum -a 256 "$ERS333_GOOD_BACKUP" | awk '{print $1}'
)" = "$ERS333_GOOD_BACKUP_SHA256" \
  && echo "PASS: restored backup SHA-256 is unchanged"

test "$(
  shasum -a 256 "$ERS333_GOOD_BACKUP.manifest.json" | awk '{print $1}'
)" = "$ERS333_GOOD_MANIFEST_SHA256" \
  && echo "PASS: restored manifest SHA-256 is unchanged"

Expected:

PASS: restored backup SHA-256 is unchanged
PASS: restored manifest SHA-256 is unchanged

Finally, re-run Bite 33.2 verification on the restored local pair:

./scripts/run-backup-go-tool.sh ers-backup verify \
  --backup "$ERS333_GOOD_BACKUP" \
  --manifest "$ERS333_GOOD_BACKUP.manifest.json" \
  --expected-environment development

Expected:

Backup verification passed

Only after this succeeds should you continue to Test 12.

PASS: remote verification succeeds while the local DB/manifest are unavailable, and the original local pair is then restored byte-for-byte and re-verifies successfully.

17. Test 12 — Retry Is Idempotent

This test requires the local database and manifest restored by the final section of Test 11.

Load persisted state:

source /tmp/ers-333-s3-manual/session.env

Confirm the local pair exists before continuing:

test -s "$ERS333_GOOD_BACKUP" \
  && echo "PASS: source backup is available for retry" \
  || { echo "FAIL: source backup is missing; complete the Test 11 restoration first"; return 2 2>/dev/null || exit 2; }

test -s "$ERS333_GOOD_BACKUP.manifest.json" \
  && echo "PASS: source manifest is available for retry" \
  || { echo "FAIL: source manifest is missing; complete the Test 11 restoration first"; return 2 2>/dev/null || exit 2; }

Expected:

PASS: source backup is available for retry
PASS: source manifest is available for retry

Capture the S3 version inventory before the retry:

aws s3api list-object-versions \
  --bucket "$ERS333_S3_BUCKET" \
  --prefix "$ERS333_S3_PREFIX" \
  --endpoint-url "$ERS333_S3_ENDPOINT" \
  > /tmp/ers-333-s3-manual/versions-before.json

Count the exact database-key versions before the retry:

ERS333_DB_VERSION_COUNT_BEFORE="$(
  jq --arg key "$ERS333_DB_KEY" \
    '[.Versions[]? | select(.Key == $key)] | length' \
    /tmp/ers-333-s3-manual/versions-before.json
)"

Count the exact manifest-key versions before the retry:

ERS333_MANIFEST_VERSION_COUNT_BEFORE="$(
  jq --arg key "$ERS333_MANIFEST_KEY" \
    '[.Versions[]? | select(.Key == $key)] | length' \
    /tmp/ers-333-s3-manual/versions-before.json
)"

Print the counts:

printf 'DB versions before retry:       %s\n' "$ERS333_DB_VERSION_COUNT_BEFORE"
printf 'Manifest versions before retry: %s\n' "$ERS333_MANIFEST_VERSION_COUNT_BEFORE"

Repeat the exact same ERS replication request:

./scripts/run-backup-go-tool.sh ers-offhost-backup replicate \
  --environment development \
  --env-file /tmp/ers-333-s3-manual/server-root/development/.env.development \
  --backup "$ERS333_GOOD_BACKUP"

Expected output includes:

Off-host backup verification passed using existing exact S3 versions:

The command must exit 0.

Capture the S3 version inventory again:

aws s3api list-object-versions \
  --bucket "$ERS333_S3_BUCKET" \
  --prefix "$ERS333_S3_PREFIX" \
  --endpoint-url "$ERS333_S3_ENDPOINT" \
  > /tmp/ers-333-s3-manual/versions-after.json

Count database versions after the retry:

ERS333_DB_VERSION_COUNT_AFTER="$(
  jq --arg key "$ERS333_DB_KEY" \
    '[.Versions[]? | select(.Key == $key)] | length' \
    /tmp/ers-333-s3-manual/versions-after.json
)"

Count manifest versions after the retry:

ERS333_MANIFEST_VERSION_COUNT_AFTER="$(
  jq --arg key "$ERS333_MANIFEST_KEY" \
    '[.Versions[]? | select(.Key == $key)] | length' \
    /tmp/ers-333-s3-manual/versions-after.json
)"

Compare:

test "$ERS333_DB_VERSION_COUNT_BEFORE" = "$ERS333_DB_VERSION_COUNT_AFTER" \
  && echo "PASS: retry did not create another DB version"

test "$ERS333_MANIFEST_VERSION_COUNT_BEFORE" = "$ERS333_MANIFEST_VERSION_COUNT_AFTER" \
  && echo "PASS: retry did not create another manifest version"

Expected:

PASS: retry did not create another DB version
PASS: retry did not create another manifest version

PASS: ERS re-verifies the exact already-recorded S3 versions and does not create replacement database or manifest versions.

18. Test 13 — Production Rejects Off-Host Protection Disabled

Create:

cat >/tmp/ers-333-s3-manual/production-disabled.env <<'EOF'
APP_ENV=production
SERVER_OFFHOST_BACKUP_ENABLED=false
SERVER_OFFHOST_BACKUP_TRANSPORT=s3
SERVER_OFFHOST_BACKUP_S3_ENDPOINT=https://hel1.your-objectstorage.com
SERVER_OFFHOST_BACKUP_S3_REGION=hel1
SERVER_OFFHOST_BACKUP_S3_BUCKET=invalid-negative-test
SERVER_OFFHOST_BACKUP_S3_PREFIX=ers-backups
EOF

Run:

set +e
make server-environment-contract-probe   ENV=production   SERVER_ENV_PROBE_FILE=/tmp/ers-333-s3-manual/production-disabled.env
ERS333_PROD_DISABLED_STATUS=$?
set -e

Expected non-zero.

Assert:

test "$ERS333_PROD_DISABLED_STATUS" -ne 0   && echo "PASS: Production cannot disable off-host backup protection"

19. Test 14 — Production Rejects Non-S3 Transport

Create:

cat >/tmp/ers-333-s3-manual/production-ssh.env <<'EOF'
APP_ENV=production
SERVER_OFFHOST_BACKUP_ENABLED=true
SERVER_OFFHOST_BACKUP_TRANSPORT=ssh
EOF

Run exactly:

set +e

make server-environment-contract-probe \
  ENV=production \
  SERVER_ENV_PROBE_FILE=/tmp/ers-333-s3-manual/production-ssh.env

ERS333_PROD_SSH_STATUS=$?

set -e

Then inspect the exit code:

printf 'Exit status: %s\n' "$ERS333_PROD_SSH_STATUS"

Expected: non-zero. Then assert the PASS condition:

test "$ERS333_PROD_SSH_STATUS" -ne 0 \
  && echo "PASS: Production rejects non-S3 off-host backup transport" \
  || echo "FAIL: Production accepted non-S3 off-host backup transport"

Expected:

PASS: Production rejects non-S3 off-host backup transport

Assert:

PASS: Production rejects non-S3 off-host backup transport

20. Test 15 — Final Known-Good S3 Backup Still Verifies

Run:

./scripts/run-backup-go-tool.sh ers-offhost-backup verify-replica \
  --environment development \
  --env-file /tmp/ers-333-s3-manual/server-root/development/.env.development \
  --receipt "$ERS333_GOOD_BACKUP.offhost.json"

Expected exit 0.

Independently download the exact recorded database VersionId one final time and compare:

final exact-version SHA-256
=
original source DB SHA-256

Expected:

PASS: final off-host DB version still matches original SHA-256

21. Test 16 — Re-Run Automated Gates

Run:

make off-host-backup-protection-check
make backup-creation-retention-verification-check
make production-environment-guardrails-check
make local-check

Then push the branch and require CI to pass.

22. Cleanup

Remove LOCAL temporary material:

rm -rf /tmp/ers-333-s3-manual

Do not delete the Production-intended bucket.

Do not use Governance bypass merely to erase protected test versions.

The manual-validation/<timestamp> objects may remain until their retention period expires.

23. Promotion Gate

Promote only if all of the following are true:

24. Promotion-Blocking Defects

Do not promote if:

Production can run without off-host protection.
Production can silently use SSH after S3 adoption.
Bucket is public.
Versioning is disabled.
Object Lock is absent.
Secret key is committed or embedded in ERS environment configuration.
ERS reports upload success without round-trip verification.
Receipt records keys but not VersionIds.
Later verification uses latest object instead of recorded VersionId.
Remote DB SHA-256 differs from source.
Remote manifest SHA-256 differs from source.
ETag is treated as SHA-256.
Retained DB version can be normally deleted during retention.
Off-host verification requires the original local backup pair.
Retry silently creates replacement DB/manifest versions.
Any 33.1/33.2/33.3/LOCAL/CI gate fails.

25. Manual Evidence to Retain

Branch:
S3-enabled SHA:

Hetzner Object Storage region:
Hetzner S3 endpoint:
Bucket:
Bucket Versioning status:
Bucket Object Lock status:
Default retention mode:
Default retention days:

Manual-test S3 prefix:

Source backup filename:
Source DB SHA-256:
Source manifest SHA-256:

S3 DB object key:
S3 DB VersionId:
S3 DB SHA-256:

S3 manifest object key:
S3 manifest VersionId:
S3 manifest SHA-256:

Initial S3 replication completed:
Independent exact-VersionId verification completed:
Final verification completed:

make off-host-backup-protection-check:
make backup-creation-retention-verification-check:
make production-environment-guardrails-check:
make local-check:
CI:

Do not retain:

S3 secret key;
application passwords;
JWT secrets;
ledger keys;
API keys.

Bite 33.3 — Off‐Host Backup Protection - RodrigoMattosoSilveira/EnterpriseRemoteSystems GitHub Wiki

Bite 33.3 — Off-Host Backup Protection

Revised Manual Verification Checklist — Hetzner Object Storage / S3

1. Purpose

2. Hetzner-Specific Contract Used by This Suite

3. Expected ERS S3 Configuration Contract

4. Safety Rules

5. Test 00 — Confirm Branch, S3-Enabled HEAD, and Required Tools

6. Test 01 — Configure Hetzner S3 Credentials Once

7. Test 02 — Verify Versioning and Object Lock

8. Test 03 — Confirm Bucket Is Not Public

9. Test 04 — Start the Normal LOCAL Backend

10. Test 05 — Create and Verify Source Bite 33.2 Backup

11. Test 06 — Create ERS Development S3 Configuration

12. Test 07 — Replicate Verified Backup to Hetzner S3

13. Test 08 — Inspect Receipt and Record Exact S3 VersionIds

14. Test 09 — Independently Download Exact Recorded S3 Versions

15. Test 10 — Verify Object Lock on Exact DB Version

16. Test 11 — Verify Off-Host Recovery Without Local DB/Manifest

Mandatory restoration before Test 12

17. Test 12 — Retry Is Idempotent

18. Test 13 — Production Rejects Off-Host Protection Disabled

19. Test 14 — Production Rejects Non-S3 Transport

20. Test 15 — Final Known-Good S3 Backup Still Verifies

21. Test 16 — Re-Run Automated Gates

22. Cleanup

23. Promotion Gate

24. Promotion-Blocking Defects

25. Manual Evidence to Retain

⚠️ GitHub.com Fallback ⚠️

Bite 33.3 — Off‐Host Backup Protection - RodrigoMattosoSilveira/EnterpriseRemoteSystems GitHub Wiki

Bite 33.3 — Off-Host Backup Protection

Revised Manual Verification Checklist — Hetzner Object Storage / S3

1. Purpose

2. Hetzner-Specific Contract Used by This Suite

3. Expected ERS S3 Configuration Contract

4. Safety Rules

5. Test 00 — Confirm Branch, S3-Enabled HEAD, and Required Tools

6. Test 01 — Configure Hetzner S3 Credentials Once

7. Test 02 — Verify Versioning and Object Lock

8. Test 03 — Confirm Bucket Is Not Public

9. Test 04 — Start the Normal LOCAL Backend

10. Test 05 — Create and Verify Source Bite 33.2 Backup

11. Test 06 — Create ERS Development S3 Configuration

12. Test 07 — Replicate Verified Backup to Hetzner S3

13. Test 08 — Inspect Receipt and Record Exact S3 VersionIds

14. Test 09 — Independently Download Exact Recorded S3 Versions

15. Test 10 — Verify Object Lock on Exact DB Version

16. Test 11 — Verify Off-Host Recovery Without Local DB/Manifest

Mandatory restoration before Test 12

17. Test 12 — Retry Is Idempotent

18. Test 13 — Production Rejects Off-Host Protection Disabled

19. Test 14 — Production Rejects Non-S3 Transport

20. Test 15 — Final Known-Good S3 Backup Still Verifies

21. Test 16 — Re-Run Automated Gates

22. Cleanup

23. Promotion Gate

24. Promotion-Blocking Defects

25. Manual Evidence to Retain

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️