An Azure Blob Storage archive holds a copy of every stored document original. It exists so a document can be put back if a job overwrites the original in the object store. This page covers the account, who can touch it, how to take a backup, and how to check one document. Reverting and the restore drill are in Document-Backup-Restore.

Scripts are in src/scripts/: backup-originals.js (backup, drill) and restore-originals.js (revert). Infrastructure is azure/modules/document-backup.bicep, one param file per environment. The PDF title job refuses to write a document unless src/helpers/backup-check.js finds a matching backup blob.

Measured 2026-10-06 to 2026-10-08, test and prod.

Step	State
Account deployed on test (c4b0a8-test-rg)	Done, nine setting checks pass
Copy and verify of a 255-object sample on test	Passed 2026-10-05, 0 gaps; a rerun skips existing blobs
Restore drill read-back check	Passed 2026-10-06, 20 of 20 matched
Full copy of the test bucket	Passed 2026-10-06: 69,153 objects, 207.0 GB, 0 failed, 0 gaps
Revert tool reads (plan, status, dry runs)	Ran 2026-10-06; see Document-Backup-Restore
Revert tool writing back to the object store	Not run; unit tests use fakes
Grant removal on test (step 7)	Run 2026-10-07. Afterwards the writer held none of its three assignments, the lock was back as CanNotDelete, and the devbox carried no writer identity
Account deployed on prod (rg-demi-prod)	Done 2026-10-07 22:15 UTC as eagle-document-backup-prod with grantWriter=true: Standard_GRS, shared key off, public access disabled, three containers, lock, private endpoint eagle-backup-pe-prod
Full copy of the prod bucket	Done 2026-10-07 from tag v0.164.0. Bucket 96,561 objects (335.6 GB), 61,746 rows, 28,393 objects (131.0 GB) excluded; writer eagle-backup-writer-prod on demi-devbox-prod. Dry run planned 68,168, failed 0. Live copy 22:42 to 23:55 UTC at about 48 MB/s: copied 68,168 (204,587,151,962 bytes), mismatch 0, failed 0, missing in source 0
Verify on prod	Passed 2026-10-07 23:57 UTC: exit 0, runId 2026-10-07T23-57-15-271Z, blobs 68,168 equal to the bucket, cosmosKeys 61,746 all backed, gaps 0, manifest sha256 ea03ba96934d1bd842a5d016fad80b0bfdc404b2e16825a95386cceadcc7325a
Restore drill on prod	--sample 10 --live started 2026-10-07 23:57 UTC. restore-originals.js --check --live passed 2026-10-08: 10 of 10 matched
Prod backup and write gate	Prod archive backup complete and verified 2026-10-08. The API's backup gate (BACKUP_ACCOUNT, BACKUP_CONTAINER) is on in prod, so a title write is refused for any document with no backup copy
Grant removal on prod	Done 2026-10-08 (step 7). The eagle-backup-writer-prod identity is detached, its three role grants are deleted, and the infra is redeployed with grantWriter=false. 0 writer assignments remain

One storage account per environment, named eaglebak<env><suffix>. Test is eaglebaktestvymaysch2agd; prod is eaglebakproduvtikwlcqtpg. The account is StorageV2, geo-redundant (Standard_GRS), Canada Central, with no public network access, a private endpoint, shared keys off and TLS 1.2. A CanNotDelete lock, eagle-backup-nodelete, is on the account. Soft delete for blobs and containers keeps items 30 days. Azure blob versioning is off.

Container	Holds	Tier	Retention
originals	One blob per object, named by s3Key	Archive	730 days prod, 30 days test, unlocked
manifests	<env>/<runId>/objects.jsonl.gz and summary.json from each verify run	Cool	Same as originals
restore	Rehydrated copies for the drill and for reverts	Cool	None. A lifecycle rule deletes blobs 30 days after creation

The retention policy blocks deleting or overwriting a blob in originals until it expires. It stays unlocked on purpose: a locked policy cannot be shortened, so a document that must be destroyed could not be. Extend the policy before it expires if the originals are still needed.

Each backup blob carries metadata keys sourcebucket, sourceetag, md5, sha256, size, documentids (cut short; the manifest has the full list) and copiedat.

Left out: every key under DO_NOT_DELETE_restic_backup/, a separate backup repository in the bucket, not a document store (owner decision 2026-10-05). It is excluded with --exclude-prefix DO_NOT_DELETE_restic_backup/ on copy and verify.

Identity	Access	When
eagle-backup-writer-<env>	Storage Blob Data Contributor on originals, manifests and restore	Only during a run. Removed after it
demi-identity-<env> (API)	Storage Blob Data Reader on originals	Always. Reads blob properties for the title job's check. Archived content cannot be read
Anyone doing a revert	Data Reader on originals and manifests, Data Contributor on restore	Granted by the owner per incident

No person holds standing access. Reads, writes and deletes go to the audit workspace demi-audit-<env>. The activity-log alert eagle-backup-changes-<env> fires once for each finished (Succeeded or Failed) Administrative operation on the account whose name is in an allow-list, and sends it to the action group demi-alerts-<env>. The list covers:

• storage account write, delete and key regeneration
• blob service and container write and delete
• immutability policy write, delete, lock and extend
• legal hold set and clear
• lifecycle management policy write and delete
• private endpoint connection write and delete
• role assignment write and delete
• resource lock write and delete
• diagnostic setting write and delete

Azure Policy evaluations, Defender for Storage settings writes, Event Grid subscription writes and the platform's private endpoint proxy validate action do not fire it. A Bicep deploy of the module still sends several mails, one per resource it writes. A subscription owner can still turn shared keys on or grant themselves a role; the alert reports it, nothing blocks it.

Never change a blob's tier in place: that makes it readable to the API reader.

A run copies every object in the bucket to originals, then proves the copy is complete. It runs on the devbox demi-devbox-<env>. Every script is a dry run unless given --live.

The landing-zone policy adds the private DNS record (10.46.50.41 on prod) some minutes after the endpoint exists. Until then the devbox resolves the public IP and every blob call fails with 403 AuthorizationFailure. Check getent hosts <account>.blob.core.windows.net before the dry run. Run-command on the devbox returns a stale instance view when a run-command name is reused; use a new name per run.

Subscriptions and groups are in each param file's header. Test is subscription 7897ceb1-9a86-4639-87d7-7f9ff67142b3, group c4b0a8-test-rg. Prod is subscription be5924ac-1083-4a1b-be92-7b444882cfd9, group rg-demi-prod. The owner deploys prod from a tag checkout.

Before the run, put the object-store settings in an env file on the devbox, such as /home/demi/backup-run/env.sh. It exports MINIO_HOST, MINIO_PORT, MINIO_USE_SSL and MINIO_BUCKET_NAME (values are in the app settings of demi-api-fc-<env>), and sets MINIO_ACCESS_KEY and MINIO_SECRET_KEY by reading the Key Vault demi-kv-<env> secrets minio-access-key and minio-secret-key when it runs, as demi-identity-<env>. It holds no secret value. Define ACCT as the account name and WRITER_CLIENT_ID as the writer identity's client id.

1. Turn the writer grant on. Deploy the module with grantWriter=true:

   az deployment group create -n eagle-document-backup-<env> -g <rg> \
     -f azure/modules/document-backup.bicep -p azure/document-backup.<env>.bicepparam -p grantWriter=true
   

   Save the outputs (az deployment group show ... --query properties.outputs): writerIdentityId, writerClientId, writerPrincipalId and the three writerGrantIds. A later deploy empties writerGrantIds, and step 7 needs them.

2. Attach the writer identity to the devbox. az vm identity assign fails when the VM carries a landing-zone identity from another subscription. Send only the writer identity with an ARM PATCH. Start a stopped VM first (az vm start).

   az rest --method patch \
     --url https://management.azure.com/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Compute/virtualMachines/demi-devbox-<env>?api-version=2024-07-01 \
     --body '{"identity":{"type":"SystemAssigned, UserAssigned","userAssignedIdentities":{"<writerIdentityId>":{}}}}'
   

3. List the bucket and export the Cosmos rows. export-rows reads Cosmos as the API identity, not as the writer. The bucket is zdspnb on test and ozwdez on prod.

   sudo -u demi /usr/local/bin/demi-run 'cd /opt/eagle-demi && git pull --ff-only && . /home/demi/backup-run/env.sh && node src/scripts/backup-originals.js list-bucket --bucket <bucket> --out /home/demi/backup-run/bucket.jsonl'
   sudo -u demi /usr/local/bin/demi-run 'cd /opt/eagle-demi && . /home/demi/backup-run/env.sh && node src/scripts/backup-originals.js export-rows --out /home/demi/backup-run/rows.jsonl'
   

   The list keeps every key. Pass --exclude-prefix to copy and verify, not list-bucket, so summary.json records the excluded totals.

4. Copy, as the writer identity (AZURE_CLIENT_ID=$WRITER_CLIENT_ID). Run it once without --live and read planned and failed, then add --live. A full copy outlasts one run-command call, so start it detached and write to a log. Test 2026-10-06: 93 minutes at about 37 MB/s, copied=68898 missingInSource=413 mismatch=0 failed=0, 28,393 objects excluded. The dry run took 52 seconds.

   sudo -u demi /usr/local/bin/demi-run 'cd /opt/eagle-demi && . /home/demi/backup-run/env.sh && export AZURE_CLIENT_ID=$WRITER_CLIENT_ID && setsid nohup node src/scripts/backup-originals.js copy --bucket <bucket> --account $ACCT --bucket-list /home/demi/backup-run/bucket.jsonl --rows /home/demi/backup-run/rows.jsonl --missing-out /home/demi/backup-run/missing.txt --exclude-prefix DO_NOT_DELETE_restic_backup/ --live > /home/demi/backup-run/full-copy.log 2>&1 &'
   

   Follow the log with tail -n 2 /home/demi/backup-run/full-copy.log. The summary line has total, copied, present, missingInSource, mismatch and failed. missingInSource counts Cosmos keys with no object in the bucket (413 on test); they go to --missing-out. Rerun the command to resume: existing names are skipped. Flags: --concurrency N (default 8), --limit N.

   Each object is hashed while read. The blob is committed only if its MD5 equals the source ETag, the size matches and no blob of that name exists. An ETag with a -N suffix (multipart upload) is also checked by reading the source twice and comparing sha256.

5. Verify. Run as the writer, with --live so the manifest is written to manifests. Test 2026-10-06 took 33 seconds: exit 0, cosmosKeys=61684 cosmosBacked=61271 missingInSource=413 gaps=0.

   sudo -u demi /usr/local/bin/demi-run 'cd /opt/eagle-demi && . /home/demi/backup-run/env.sh && export AZURE_CLIENT_ID=$WRITER_CLIENT_ID && node src/scripts/backup-originals.js verify --env <env> --account $ACCT --bucket-list /home/demi/backup-run/bucket.jsonl --rows /home/demi/backup-run/rows.jsonl --bucket-missing /home/demi/backup-run/missing.txt --exclude-prefix DO_NOT_DELETE_restic_backup/ --live'
   

   Exit 0 means no gaps: every bucket key has a blob of equal size and checksum, totals match, every Cosmos key has a blob or is in the missing list, and every blob is in Archive with a sha256. Exit 1 means gaps, named in the log with the runId. Without --live nothing reaches manifests.

6. Start the restore drill if one is wanted (Document-Backup-Restore). It needs the writer identity, so start it before step 7. Reading its result needs it up to 15 hours later.

7. Turn the grant off, in this order. Account locks block deleting role assignments, so the lock goes first.

   1. Detach the identity: the step 2 ARM PATCH with null in place of {}. Deallocate the VM if it was stopped before the run (az vm deallocate).
   2. Delete the lock: az lock delete -n eagle-backup-nodelete -g <rg> --resource-name $ACCT --resource-type Microsoft.Storage/storageAccounts
   3. Delete the grants: az role assignment delete --ids <the three writerGrantIds>
   4. Redeploy with grantWriter=false, the step 1 command with that value. This puts the lock back.

   Deleting the lock needs a person with the owner role. Automation's CLI was refused on test, so the owner runs sub-step 2 and automation does the rest.

8. Check the writer holds nothing. az role assignment list --assignee needs a Graph token that the devbox login lacks, so ask ARM with a principalId filter:

   az rest --method get --url "https://management.azure.com/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<account>/providers/Microsoft.Authorization/roleAssignments?api-version=2022-04-01&\$filter=principalId eq '<writerPrincipalId>'" --query "value[].id"
   

   The answer is an empty list. With demi-identity-<env>'s principal id it shows the one reader assignment on originals.

Blobs written by a run cannot be deleted until retention ends.

Three ways, from least to most access. Way 1 ran on test 2026-10-06 as demi-identity-test and answered unchanged; ways 2 and 3 are not run as written.

1. Ask the revert tool. plan is read-only and needs the manifest (Document-Backup-Restore step 1). The document's row says unchanged when the bucket object matches its backup, restore when it differs, and refused with reason no-backup when the manifest has no verified entry for its key:

   node src/scripts/restore-originals.js plan --id <docId> --manifest objects.jsonl.gz
   

2. Search the manifest. Each line of objects.jsonl.gz has key, size, checksums, document ids, copy time and status, which must be ok.

   zcat objects.jsonl.gz | grep '<docId>' | head -n 5
   

3. Read the blob's properties. This needs Data Reader on originals. The blob name is the document's s3Key. A good backup shows tier Archive, a size equal to the document's file size, and the metadata keys listed above.

   az storage blob show --auth-mode login --account-name <account> -c originals -n <s3Key> \
     --query "{tier:properties.blobTier,size:properties.contentLength,md5:metadata.md5,sha256:metadata.sha256}"
   

The title job runs check 3 itself, through assertBackedUp. It refuses a document with 409 no-backup when the blob is missing or its size or checksum differs, or BACKUP_ACCOUNT is empty.

Design estimates at about two write operations per object, prices not re-read from the Azure price API. Archive has a 180-day minimum storage charge. Revert costs: Document-Backup-Restore.

Data	Objects	Writes	Storage per month	180-day minimum
200 GB	100,000	$4.60	$0.84	$5.00
1 TB	500,000	$23	$4.30	$26

Prod (2026-10-07): 61,746 documents, 68,168 objects, 204.6 GB excl. restic; a year in Archive is about USD 14.

• Documents added after a backup have no backup blob, and the title job refuses them. Rerun copy then verify.
• If an object changes after its backup, the title job's check fails with md5-mismatch. The backup blob cannot be overwritten.
• Blobs in originals and manifests cannot be deleted until retention ends, including the backup of a document that must be destroyed. The policy stays unlocked for that reason.
• Every devbox infrastructure deploy drops a hand-attached writer identity.

Related: Azure-Environments (cost), Operating-DEMI-in-Azure (deploy), Bulk-Download-Operations (demi-run).