Eagle Mirror Reference - digitalspace/eagle-demi GitHub Wiki

Eagle Mirror Reference

What eagle-api mirrors into DEMI besides projects and documents, and how drift and refusals show up. The push design is in Sync-Architecture.

Mirrored kinds

demiPush.js also pushes four more record kinds, one call each, fire-and-forget and gated the same way. Each takes the raw Eagle record; DEMI stores it under sources.eagle and derives the row's read[] and isPublished from it, as for project and document:

Kind Eagle model DEMI route Cosmos container Partition key
commentPeriod CommentPeriod PUT /api/eagle/commentperiods/{eagleId} commentPeriods /projectId (the DEMI project id)
comment Comment PUT /api/eagle/comments/{eagleId} comments /periodId
organization Organization PUT /api/eagle/organizations/{eagleId} lists (kind: 'Organization') /kind
projectNotification ProjectNotification PUT /api/eagle/notifications/{eagleId} notifications /id

Five more kinds mirror users, groups and inspections, sent only when DEMI_PUSH_OPT_IN_KINDS names the kind: PUT /api/eagle/users/{eagleId}, /groups/, /inspections/, /inspection-elements/ and /inspection-items/. Containers: Architecture. A child that arrives before its parent gets a 503, which eagle-api retries once. A row's read[] is capped by its parent (group and inspection by the project, element by its inspection, item by its element). reconcile-eagle.js checks each row against its stored Eagle read and parent, and reports aclMismatch and missingParent. Field levels: Attribute-Level-Access. Reads: Search-Query-Construction.

A comment period's read[] is constrainToProject-ed against its parent project's, the same formula documents use, so a period can never out-rank its project. A comment period with no parent project in DEMI is refused with a 404, as a document is.

The 404 body carries a code (see Refusal codes) but not the reason. The log has it: each refusal writes one warning, [parent-admit] parent not admitted, with the parent's Eagle id (eagleId), the refused record's Eagle id (childId), and one reason per container checked (project and notification, or period for a comment). The childId values can be collected into an ids file for a targeted repush. The reasons are:

Reason Meaning
missing No row with that id.
hidden The row exists but has no read[] or is DEMI-sealed (level 0 with sealedAt).
eagle-sealed The row is at level 0 without sealedAt: an Eagle push carrying compliance sealed it. It heals on its own next push.
visible The row was written after the push checked for it.
unknown The check failed or took longer than 2 s. A could not classify parent warning carries the error.
malformed-ref The parent reference is not a 24-character hex id. No lookup is made and the reference is not logged.

The reason costs one extra read, made only on a refusal. The warning [parent-admit] hidden notification passed its children to a same-id project means a hidden notification let a same-id Track project take the child.

A level-0 row without sealedAt was sealed by an Eagle push and takes the read of its next push; only a DEMI seal outlasts a push. Until then its children are admitted at its Eagle read minus compliance, and each admission logs [parent-admit] project stored sealed by an Eagle push, admitted at its Eagle read.

A read that held only compliance is stored as ['sysadmin'], as is ['compliance', 'sysadmin'], so only a privileged caller reads the row, never the project team. When a project or parent caps a row to level 1, a read of privileged roles only is kept, not widened to team (capRead).

A fifth kind, config, mirrors eagle-api's public config, not a record. With no Eagle _id, the route takes a fixed id (PUT /eagle/config/public, config container, item public) and the body is the payload itself, not a { doc } envelope. It fires from GET /api/config on a change, as Config has no write path in eagle-api. See ADR-012-Public-Config-Surface.

Updates: publish gate and announce

Updates (Eagle RecentActivity) arrive on PUT /api/eagle/updates/{eagleId} into updates (partition /id). An Update can carry status (draft, published, archived) and publishDate besides read[]. Callers below staff level (IDIR and public) see a row that has a status only while status is published and its publishDate has passed. The gate reads publishDate alone, so Cosmos can serve it from the index. The mirror stores dateAdded as publishDate when none is sent; src/scripts/backfill-update-publish-date.js fills older rows, and a gated row with no publishDate stays hidden. A row with no status follows read[] alone. The gate is evaluated when each query runs, with nothing stamped on rows: Cosmos reads add it in src/repositories/updates.js, and keyword searches and counts over the activities index add the OData version of it.

An Update can never out-rank its parent project (ADR-004 ยง4, the same rule as documents and periods). At push time, src/controllers/nosql/update.js resolves the Update's parent through readParent (src/helpers/update-parent.js), which reads the project and the notification with that Eagle id unfiltered; a notification wins the id. When the parent sits at a lower level than Eagle's read[] minus compliance, the Update's read[] is capped to the parent's level (readUnder); otherwise it keeps Eagle's own. A notification parent caps the same way. A parent an Eagle push sealed (level 0, no sealedAt) caps at its own Eagle read minus compliance, not at sealed; a DEMI-sealed parent caps to sealed. Eagle's own ACL, minus compliance, is kept underneath as sources.eagle.read so a later project change can re-derive from it. An Update with no parent DEMI holds keeps Eagle's read[] minus compliance. This matches eagle-api's parentReadMatch, which hides a record whose parent the caller cannot read.

An Update with a missing Eagle read (absent or null), whose status is null and whose active is true takes its parent's ceiling read as is (inheritsParentRead, updateRead). Legacy 2017 migration rows look like this. eagle-api serves rows with no read field but hides read: [], so DEMI does the same. Reconcile skips Eagle rows with an empty or missing read, so these never show as aclMismatch.

When a project's visibility changes, cascadeProjectVisibility (src/controllers/nosql/project.js) recaps that project's Updates the same way it recaps documents and periods, through src/helpers/update-acl.js. The helper re-queries updates.projectId, skips rows whose id belongs to a notification (a notification wins that id space over a same-id Track project, so its Updates were never capped by the project), and patches read[] and isPublished on the rest. A legacy open Update (missing read, null status, active: true) inherits the project's ceiling read here too.

Every RecentActivity read in src/controllers/search.js also drops, at query time, any Update whose parent project the caller cannot read (unreadableProjects, called from updateProjects), so older rows need no repush. Known gap: the count field on a filtered list, and the count behind the activities keyword index, are not re-filtered, so they can include a hidden row.

Sorts on publishDate (the home strip, and sortBy=publishDate on the list) run on dateAdded while the app setting UPDATES_PUBLISH_DATE_FALLBACK is on (the default), because a Cosmos ORDER BY leaves out rows that lack the field. Turn it off (bicep updatesPublishDateFallback = false) once a backfill dry run reports undated=0.

DEMI emails an Update through eagle-notify at most once. Before sending, a writer takes a claim with a conditional patch that succeeds only if the row is still published, past its publish time, and not claimed. Cosmos checks this at write time, so an Update archived or rescheduled after the timer listed it is not sent, and the email carries the row as it is at claim time. A claim records notifiedBy: 'demi', notifyClaimedAt and notifyAttempts, and a successful send records notifySentAt.

When eagle-notify answers 4xx, DEMI records notifyFailedAt and never retries; the same body would be refused again. When it answers 5xx or does not answer, the claim stays as a 30-minute lease. Once the lease runs out, the timer can take the claim over and try again, up to 3 attempts in total. The same rule recovers a run that was stopped in the middle of a send.

The push announces a published Update only when it is inside the 7-day notify window: an unclaimed row counts from publishDate, a dead claim from notifyClaimedAt. A push outside the window still mirrors the row โ€” a bulk repush of old Updates carries them all โ€” but leaves it unclaimed, so it is never emailed by the push or by the timer. eagle-api does not push again when a scheduled Update's publishDate arrives, so the announceUpdates timer (src/scripts/announce-updates.js) handles that case. Every 5 minutes it takes up to 20 rows, retries first, and handles them together:

  • withdrawn Updates with a DEMI claim whose cancellation has not gone out yet;
  • DEMI claims whose lease ran out with no send, no refusal and fewer than 3 attempts;
  • unclaimed published Updates whose publishDate has passed, oldest first.

Each list is a query on plain comparisons of indexed fields, read page by page until it has enough rows, for at most 10 pages. Rows without a status are listed only to retry a DEMI send. ANNOUNCE_UPDATES_SCHEDULE comes from the bicep parameter announceUpdatesSchedule. When it is empty, the schedule is 0 */5 * * * * where notifyApiBase is set, and there is no timer where it is not.

Some claims are not DEMI sends. A pushed record with Eagle's own notifiedAt fills an empty claim and is marked notifiedBy: 'eagle'. The public-read backfill claims every live Update it carries (notifiedBy: 'backfill') and leaves scheduled ones for the timer.

Withdrawing an Update (unpublishing or archiving it) sends one cancellation whenever DEMI holds the claim (notifiedBy: 'demi'), even when no send was confirmed, as an unanswered send may have gone out. A claim with no notifiedBy predates this bookkeeping and is treated like a backfill claim: no cancellation. The time is recorded in notifyCancelledAt, also when eagle-notify refuses the cancellation. A cancellation with no answer is left unrecorded, and the timer retries it. A cancellation never gives the claim back, so publishing the Update again sends nothing. Moving publishDate into the future after the email went out sends nothing either: no cancellation, and no second email when the rescheduled date passes.

Wire contract, request/response shapes and the credential this push authenticates with: Connecting-an-Application-to-DEMI.

ENGAGE-owned comment periods

A comment period with sourceSystem: 'engage' is written by the ENGAGE ingest, not by this mirror. An Eagle push to such a row writes only the Eagle-owned fields, and reconcile-eagle.js leaves the row out of its diff. Fields, join rules, tombstones and the reconcile-engage report: ENGAGE-Rows-Reference.

Refusal codes

A child push whose parent is not admitted gets a 404 with body { error, code }. The code tells eagle-api whether waiting can help:

code Cause Retry helps
PARENT_NOT_FOUND The parent id is valid but no admissible parent row exists. Yes, once the parent lands.
PARENT_REF_INVALID The parent reference is empty or not a 24-character hex id. No.

Document, comment period, comment, group and inspection pushes send both codes.

Parked retry in eagle-api

api/helpers/demiPush.js parks a document, comment period, comment, group or inspection push that gets PARENT_NOT_FOUND, and sends it again through the same mirror path (the row is re-read from Mongo). Any other 4xx is dropped.

  • A landed push of a project, notification or comment period in the same pod retries the records parked on it.
  • Otherwise the retry runs after 30 s, then after 5 min, for a parent another pod pushed.
  • Parked records have their own map, capped at DEMI_PUSH_QUEUE_MAX; a full map drops the refused record.
  • A record still refused after the second retry, or still parked at a clean shutdown, logs push-dropped <kind> <id>: rejected 404 (parent not found). A pod killed without the shutdown path loses parked records silently.

The durable backstop is the push-missed-parent-in-demi class below.

Drift classes

src/scripts/reconcile-eagle.js puts each drifted id into one class with classify from src/scripts/parity-map.js. A class names the likely cause. Nothing is fixed automatically.

Class Meaning
demi-only DEMI row whose id is not an Eagle id: Track-only, or created in DEMI.
list-id-other-env The same List row (name, type, legislation) sits on the other side under another id.
parent-not-public Child missing in DEMI; its parent exists but is not public.
orphan-parent-missing-in-eagle The parent ref is empty or malformed, or names a row in none of Eagle, DEMI and Track.
push-missed-parent-in-demi Eagle child missing in DEMI, parent present in DEMI. A repush should fix it: feed the ids to demi-repush.js --ids-file.
push-missed Any other missing row.

track-mastered is field-level (DEMI takes the field from Track). Only parity-eagle.js produces it; id-set reconcile cannot, and cannot see field drift such as a changed update status.

The run logs the classes on a [reconcile] classes ... line. The drift= line and its alert are unchanged.

Stored report

reconcile-eagle.js --store writes one row to the config container. The nightly timer passes the flag. A failed write is logged and the run still ends.

Field Content
ranAt Start of the run.
<kind> projects, documents, commentPeriods, lists, notifications, updates, and comments when swept.
<kind>.counts Count per class.
<kind>.ids At most 200 ids per class.
<kind>.truncated true when a class held more than 200.

GET /api/admin/reconcile (admin role) returns { success, storedAt, report }. It answers 404 no reconcile report stored before the first stored run and 503 when the row cannot be read.

Migration re-push rule

A migration writes Mongo directly, so nothing is pushed. Each push carries the whole stored record, so changing any field leaves DEMI behind until the record is pushed again.

  • Every eagle-api migration from 20260923000000 that names a mirrored _schemaName carries a DEMI: header note. It names scripts/demi-repush.js and each kind, or says DEMI: none with a reason. test/migrations/demi-mirror-note.test.js enforces it.
  • List has no write controller in eagle-api, no push and no PUT /eagle/lists/* route. Its rows (kind: 'List' in lists) come only from seed-public-reads.js --only lists in eagle-demi (Public-Read-Backfill, Search-Index-Reference). A migration that renames List entries also re-pushes projects and documents, because their labels copy List names.
  • Config needs no note: eagle-api pushes it again on the next GET /api/config.

Wire contract, request and response shapes, and the credential: Connecting-an-Application-to-DEMI. Why the push is built this way: Sync-Architecture.

โš ๏ธ **GitHub.com Fallback** โš ๏ธ