Public Read Backfill - digitalspace/eagle-demi GitHub Wiki
How to fill and repair the four containers behind ADR-011-Public-Read-Surface —
commentPeriods, comments, notifications and lists — and the matching eagle-api script that
re-sends what the push missed. Design and containers: Sync-Architecture. Query shape:
Search-Index-Reference.
eagle-api pushes commentPeriod, comment, organization and projectNotification to DEMI on
every write, the same way it already does for project and document. The push covers everything
written from here on. It does not cover what predates it: the corpus at cutover, and List, which
has no eagle-api write controller at all — list items change only by migration, so nothing ever
pushes them.
src/scripts/seed-public-reads.js, run on the devbox: Operating-DEMI-Data-Plane#reach-the-data-plane
or the repo README's "Running anything against the database". Reads eagle-api over HTTPS
(EAGLE_API_BASE), writes Cosmos (COSMOS_ENDPOINT, COSMOS_NOSQL_DATABASE) through the same
mirrorFromEagle function the push handlers call, so a backfilled row and a pushed row are
identical.
npm run db:seed-public-reads # dry run: fetches, counts, writes nothing
npm run db:seed-public-reads -- --live
npm run db:seed-public-reads -- --live --only lists
npm run db:seed-public-reads -- --live --since 2026-01-01
npm run db:seed-public-reads -- --live --state ./repair.state.jsonStages run in dependency order and --only selects among them without reordering:
lists → organizations → notifications → updates → commentPeriods → comments. Projects
are not this script's: run db:seed-nosql first.
Resumable. A finished stage is recorded in the state file (./seed-public-reads.state.json, or
--state <path>) and a later run skips it. The comments stage records each comment period whose
comments all wrote, so a run killed part way through resumes at the period it stopped on. A stage
that logged any error is not marked complete, so a partial stage is retried in full next time.
--only overrides the state file: naming a stage runs it whether or not it is marked done, which is
how a repair is asked for. --only comments also ignores the per-period checkpoint, so it re-walks
every period rather than only the ones the plain run had not yet reached.
Rerunning --only updates rewrites existing rows. A legacy Update with a missing read (absent
or null) takes its parent's read on the rerun. read: [] does not, because eagle-api hides it. See Eagle-Mirror-Reference#updates-publish-gate-and-announce.
A stage that skipped rows because their parent was not yet in DEMI is not checkpointed complete
either — those rows are still owed, not permanently absent, and a plain rerun retries them once the
parent lands. The log line and the state file count this as dropped, apart from skipped: a row
outside --since was never wanted, a dropped row still is.
Fetching a comment page treats a missing x-total-count header, or a body shaped as
[{total_items, results}] instead of a bare comment array, as an error rather than a page of zero
comments. Either shape means eagle-api broke the count=true contract this stage depends on; before
this was a silent zero that still checkpointed the period as done.
Parent admission. A comment period's parent is a project or a ProjectNotification — Eagle's
project reference holds either id, and a notification wins when both exist for the same id
(src/helpers/parent-admit.js, pickParent/admitParent). A period or comment whose parent is not
in DEMI is skipped, not written with no ACL.
src/scripts/reconcile-eagle.js diffs the ids Eagle publishes against the rows DEMI holds, for
projects, documents, commentPeriods, both lists kinds, notifications and updates on
every run. It reports drift and deletes nothing.
node src/scripts/reconcile-eagle.js # comment periods, lists, notifications, updates
node src/scripts/reconcile-eagle.js --comments # and commentscomments is swept only behind --comments, because the sweep costs one eagle-api request per
comment period plus one Cosmos query per period. The nightly reconcileEagle timer calls it with no
flags, so its alert line reports comments: skipped rather than a false zero.
Three findings, not two:
-
unresolvedParent— a period or comment Eagle publishes whose own parent project is unpublished or gone. The seed drops these too, so they are reported separately and excluded fromdrift=. -
misfiledParent— a row that IS mirrored, but sits under a parent the admission rule would not choose today. This is the one check that catches the notification-vs-project collision: a Track project can carry aProjectNotificationid inepic_guid, and before the notification was made to win, a period referencing that id was filed under the Track project's partition and ACL instead. Both sides hold the id, so an id-set diff alone calls it clean;misfiledParentis what finds it, and it counts towarddrift=.
The repair, in order:
npm run db:seed-nosql -- --live --only documents
npm run db:seed-public-reads -- --live --only commentPeriods,comments --state ./repair.state.jsonDocuments first, because a document moves partition on its own re-seed. The period re-mirror then
moves the row to the notification partition, deletes the row left under the project, and cascades
the corrected access list to its comments; the comments stage follows because a comment stores its
period's projectId and nothing else rewrites it. A fresh --state path makes the comments stage
revisit periods an earlier run already checkpointed.
eagle-api allows 200 requests a minute per client (ratelimit-policy: 200;w=60). A comment sweep or
the comments backfill stage can spend that inside a minute; the shared fetch helper
(src/seed/sources.js) waits a 429 out rather than failing the run. Order it tries, per response:
-
retry-after, if the response names one. -
ratelimit-reset(seconds left in the current window). - The window length off
ratelimit-policy(200;w=60). - A 5 s, 15 s, 60 s ladder, if none of the above is present.
Every wait is floored at 1 s and capped at 120 s, so a ratelimit-reset: 0 cannot retry instantly
and a bad header cannot park the run for an hour. It gives up after 5 waits on the same request.
Each wait logs one line — a run gone quiet for a minute at a time is being throttled, not stuck.
Rows seeded into DEMI before eagle-api started pushing are missing the push-only enrichment a
controller write adds on the way out — a project's applicableRegulation, resolved pins,
featuredDocuments and per-legislation proponent labels, added in api/helpers/demiPush.js, never
stored in Mongo. Nothing re-pushes a record that has not been edited since, so this script walks a
Mongo collection and pushes every record through the same helper the controllers use.
node scripts/demi-repush.js --kind project # dry run, default
node scripts/demi-repush.js --kind project --live
node scripts/demi-repush.js --kind document --live --state ./repush.state.json
node scripts/demi-repush.js --kind commentPeriod --live --concurrency 1--kind takes project, document, commentPeriod, comment, organization or
projectNotification (default project). --since <ISO> filters on the record's own update
timestamp where one exists; projectNotification has none and refuses --since. --concurrency
sets pushes in flight, default 4. --state <path> checkpoints the last settled _id; a rerun with
the same path resumes after it. The write is a PUT on the record id, so a repeated push is harmless.
Exit codes: 0 everything pushed, 1 the run itself failed, 2 bad arguments or DEMI_API_BASE/
DEMI_APIM_KEY not set, 3 the run finished with records DEMI did not accept (see the logged
failed ids and retry with --state, which resumes past the last clean batch).
Concurrency 4 hit APIM's rate limit on test. A full --kind project re-push at the default
concurrency 4 was throttled partway through; rerunning with --concurrency 1 from the same
--state file finished the remaining records (376/376, measured 2026-09-08). Start a fresh
environment's first repush at --concurrency 1 rather than assuming the default is safe there.
- Sync-Architecture — the push and the four mirrored kinds
- Search-Index-Reference — the Cosmos-served datasets these containers answer
- ADR-011-Public-Read-Surface — why reads and writes split across two systems
-
eagle-demi/docs/public-read-backfill.md— the same detail, kept in the repo for the devbox session that runs it without a wiki checkout