This page is the authority on how records get into DEMI. If another page contradicts it, this page is right.

eagle-api pushes to DEMI on write, authenticated. It is not a MongoDB change-stream worker and not a poller; both were considered and rejected below.

Track   ──(5-min poll, nightly sweep)──▶ DEMI Cosmos ──(indexers, PT5M)──▶ Azure AI Search
Eagle   ──(push on write)─────────────▶      ▲   │
ENGAGE  ──(push on write)─────────────▶      │   └──(sync-out queue)──▶ eagle-api HTTP API
                                             └── seed / --reconcile, run by hand

Nothing in DEMI writes Eagle Mongo. DEMI writes ENGAGE-owned comment periods into Eagle only through eagle-api's HTTP API (sync-out). Eagle's MongoDB is being retired; until then its projects and document metadata are mirrored into DEMI Cosmos, and indexing runs against Cosmos.

PUT /api/eagle/projects/{eagleId} and PUT /api/eagle/documents/{eagleId} take it, both behind authMiddleware, requireWrite and requireEagleMirror — the last pins every /eagle/* write to eagle-api alone — and audited on every call. The wire contract — bodies, responses and error codes, and the mirror's auth gate — is in Connecting-an-Application-to-DEMI.

The body carries the raw Eagle record, so DEMI holds the merge and transform rules and a push and a re-seed produce the same row. DEMI, not the push, decides identity, visibility and extraction state, and any enrichment block already on the row survives.

On the eagle-api side the caller is api/helpers/demiPush.js, called from the project and document controllers on every write, publish and unpublish. It is fire and forget: nothing awaits it, so a DEMI outage never fails an Eagle write. It retries once on a network error or a 5xx, gives up on any 4xx except a missing-parent 404 (parked and retried, see Eagle-Mirror-Reference), and logs the failure. It makes no call at all unless both DEMI_API_BASE and DEMI_APIM_KEY are set, which is how the push stays off in an environment that has no DEMI.

Document pushes also carry labels: the resolved List names for the record's type, milestone, projectPhase and documentAuthorType ObjectId references, so DEMI stores each label beside its id and needs no mirror of the List collection.

A project push also resolves fields Mongo cannot serve on its own: each legislation block's applicableRegulation becomes {_id, name, item}, its proponent reference becomes proponentId and proponentName, pins becomes [{_id, name, province}], and featuredDocuments becomes a plain id list. DEMI's project merge flattens these onto the top-level record — see EAGLE_ONLY_FIELDS and PUSH_ONLY_FIELDS in src/merge/project.js. proponentId is also what lets and[proponent]=<ObjectId> be expressed as a filter — see Search-Index-Reference.

The merge reads every legislation_YYYY key on the pushed project, so a new Act needs no change in src/merge/project.js. flattenEagleProject picks one block as the project's content:

1. The block named by currentLegislationYear, when it matches legislation_YYYY and is not empty. It is used whether or not it has a name.
2. Otherwise the only non-empty block that has a name.
3. Otherwise, with no top-level name and at least one legislation_YYYY key present, the merge throws an error with status 400 and the push is rejected. With a top-level name, or with no legislation_YYYY key at all, the record is used as it stands.

Other mirrored kinds, refusal codes, the parked retry and the migration re-push rule: Eagle-Mirror-Reference.

ENGAGE pushes each engagement to PUT /api/engage/engagements/{engagementId}. DEMI stores it as a comment period it owns (sourceSystem: 'engage') and queues the row for every enabled consumer. The Eagle consumer writes the period into eagle-api. Why: ADR-014-Sync-Out. Fields, join rules and reconcile: ENGAGE-Rows-Reference. The request: Connecting-an-Application-to-DEMI.

• Queue. A Storage queue named by SYNC_OUT_QUEUE, with a <queue>-poison queue beside it. A message carries the consumer name, the row id, the project id and the attempt number, never row data. Empty SYNC_OUT_QUEUE registers no worker and queues nothing.
• Worker. The syncOutWorker storage queue trigger (api/index.js) calls src/sync-out. It re-reads the row, so the newest state is sent whatever the message order.
• Consumer. src/sync-out/eagle.js signs in to Keycloak as demi-sync-out with client credentials and calls eagle-api: PUT /commentperiod/{id} when the row has an eagleId; otherwise it looks for a period under the project with the same metURL and updates it, or creates one with POST /commentperiod. It stores the created period's eagleId on the row. A deleted row sends DELETE. A claim written to the row before the lookup stops two deliveries from both creating the period. The claim holds for 180 s and is renewed just before the POST.
• Create guards. A row with no metURL is never created in Eagle: it is recorded skipped and logged as an error. A metURL match that DEMI already holds as an Eagle-owned row is not adopted: the row is recorded conflict with the candidateEagleId, and reconcile-engage reports it.
• Project move. The consumer records the Eagle project it sent under in syncOut.eagle.projectId; before the first send it reads the project from the Eagle copy in sources.eagle. When the row's eagleProjectId differs, it deletes the Eagle period and creates one under the new project, because eagle-api's PUT keeps a period's project.
• Idempotency. syncVersion is the row's version; every ENGAGE write raises it. After a send the worker records the version it sent as syncOut.eagle.sentVersion, and a message for a row already sent at that version or later is skipped. If the row's version rose during the send, the worker queues a resend that skips this check. sentVersion never moves back.
• Retry. A failed send is queued again with a visibility timeout of 30 s, doubled for each attempt. Attempts per message are host.json's maxDequeueCount (3). The last attempt does not throw, whether it failed in the send or before it, such as an unreadable message or a failed row read: it records failed on the row, moves the message to <queue>-poison itself, then logs [sync-out] job failed once. The alert demi-sync-out-failed-<env> fires on that line. When a newer send is already recorded, or another delivery holds a live create claim, the last attempt records nothing and logs a warning instead, since that other send settles the row. The host redelivering a last attempt that already failed at the same version is dropped with a warning, unless the message is a resend or a --repair message. A failed enqueue at ingest is logged and the push still succeeds; reconcile-engage.js --repair queues the row again.
• Echo. eagle-api pushes the period back through PUT /api/eagle/commentperiods/{eagleId}. The mirror writes only the Eagle-owned fields onto an ENGAGE-owned row and queues nothing.
• Flags. SYNC_OUT_QUEUE turns the queue on. SYNC_OUT_EAGLE_ENABLED=true turns the Eagle consumer on; the queue can run with it off. All settings: ENGAGE-Rows-Reference#settings.

Never two systems send periods to Eagle at once. A gap of a few minutes with none is acceptable.

1. Deploy DEMI with the queue on and SYNC_OUT_EAGLE_ENABLED=false.
2. ENGAGE pushes to both Eagle and DEMI. A backfill pushes every engagement that has a project; rows adopt their Eagle period through trackingId.
3. Run reconcile-engage.js until it reports no drift against Eagle. The consumer is still off.
4. Outside ENGAGE's scheduled publish and close-out jobs: switch ENGAGE to DEMI only, set SYNC_OUT_EAGLE_ENABLED=true, then run reconcile-engage.js --repair once.

Test is cut over before prod.

1. Set SYNC_OUT_EAGLE_ENABLED=false.
2. Run the ENGAGE backfill job so each engagement carries its current Eagle period id as trackingId.
3. Switch ENGAGE back to pushing to Eagle.

eagle-search runs one (eagle-search/worker/), and it works: Mongo change streams, a resume token, freeze-on-error, and operationType: 'delete'. It would have to be re-homed, given Mongo credentials, and deployed into OpenShift — investment in a database intended for deletion. More importantly, most of what that worker does, DEMI does not need. See below.

src/seed/sources.js already reads eagle-api over HTTPS rather than Mongo, and eagle-api's authenticated GET /api/search returns the unpublished records /api/public/search hides, so a poller was cheap to build. It fails on deletes.

eagle-api hard-deletes. findOneAndDelete, no tombstone. A deleted record leaves nothing behind: no row to poll, no flag to notice, and no push to send. A deleted document therefore stays in DEMI and stays searchable.

The only thing that can catch that is an id-set reconciliation between eagle-api and Cosmos. A Functions timer (reconcileEagle, prod only) runs one nightly; the seed's --reconcile mode does the same thing by hand.

src/scripts/reconcile-eagle.js covers projects, documents, commentPeriods, both lists kinds, notifications and updates on every run. It reports drift and deletes nothing. Drift classes and the stored report: Eagle-Mirror-Reference#drift-classes. comments is swept only with its --comments flag — one eagle-api round trip per comment period — so the nightly timer, which calls it with no flags, does not sweep comments. A comment period or comment whose parent is unpublished or gone is reported separately under unresolvedParent, not counted as drift, because the backfill and the seed both drop those rows too.

This is the biggest difference from eagle-search, and it is worth stating precisely because copying that design in would be a large, security-sensitive mistake.

Chunk visibility in DEMI is derived from the parent DOCUMENT at query time. Both chunk paths in src/controllers/search.js fetch the parent documents under the caller's own access and drop any chunk whose parent is not among them, logging the withheld count. A caller who cannot see the document cannot see its text, and the chunk's own read[] is treated as an ingest-time snapshot that may lag. A stale chunk ACL therefore cannot leak text on its own — which is the property eagle-search buys with resync.js, stamp.js, awaitParents and the narrows gate.

Do not port that machinery into DEMI. It has no counterpart here.

Project names are resolved live by labelWithProjectNames under the caller's own access, so a rename cascades nothing either.

Documents are gated on their own stamped read[] in the index, so a project's ACL change must reach them. cascadeProjectVisibility in src/controllers/nosql/project.js does this, on either publish transition, so a re-publish restores the documents a previous unpublish narrowed. It runs on updateProject and on a project push whose isPublished changed.

The formula is constrainToProject (src/repositories/documents.js) — the intersecting version of eagle-search's worker/transform.js, with the same fail-closed branches. A document's own narrowing survives and the cascade cannot widen an ACL. ownRead is captured lazily on the first cascade, so every later one re-derives from the seeded value rather than from an already-narrowed one. The derived ACLs are written to Cosmos and to the documents index in the same pass, because no document list is a live read.

The seed enforces the same invariant before it writes: transformDocument narrows each document against its project's read[] (notification-parented rows have no project row and stay verbatim), and verifyItems fails the run — dry run included — on any document that is public under a non-public project.

DEMI pulls from Track; Track does not push. Two Functions timers read GET /api/v1/projects from epictrack-api with the Track client credentials (TRACK_API_BASE, TRACK_CLIENT_ID, TRACK_CLIENT_SECRET) and write the Track-owned fields onto DEMI's project rows:

A timer is registered only when its setting is set. A Track edit to name, description, proponentName, type, address or state reaches the DEMI row within five minutes, because Track wins those fields (TRACK_PRECEDENCE in src/merge/project.js). eagle-public reads DEMI directly, so the edit shows on the public site at the same time. An Eagle push leaves a Track row's stored phases in place.

A push route exists too. DEMI's project id is the Track project id — GET /api/projects returns id: "1", trackProjectId: 1 — so PUT /api/projects/<trackProjectId> is the existing update route, behind authMiddleware + requireWrite, and createProject rejects a project with no trackProjectId. Track does not call it. ADR-009-Track-Feed holds the options considered for the feed.

src/scripts/seed-nosql.js is the only bulk Eagle→Cosmos path: full-corpus, dry-run by default, run by hand on the devbox VM (Operating-DEMI-Data-Plane § Reach the data plane), invoked by no workflow, keeping no cursor. It reads eagle-api's public HTTP API at EAGLE_API_BASE, not Mongo, and --reconcile deletes DEMI rows whose upstream is gone, under a ceiling. It preserves extraction state, so re-running it does not orphan chunks.

It is the disaster-recovery path for projects and documents, both of which are fully regenerable from public endpoints plus two checked-in files. chunks is not — see Azure-Environments for the backup posture.

• Identity-Model — which id means what, and the Eagle-only project rule
• Sources-and-Status — every source and surface, each in one state
• Eagle-Mirror-Reference — mirrored kinds, refusals, drift classes, migration re-push rule
• Architecture — containers, the merge, the ACL model
• Public-Read-Backfill — filling the four containers, and the misfiledParent repair
• ADR-014-Sync-Out — why ENGAGE flows through DEMI to Eagle
• ENGAGE-Rows-Reference — ENGAGE-owned rows: fields, joins, reconcile
• ADR-002-Async-Processing-MongoDB-Queue — the superseded Mongo queue design