Connecting an Application to DEMI - digitalspace/eagle-demi GitHub Wiki

Connecting an Application to DEMI

How another EAO application gets a credential for DEMI, and which one to ask for. The reasoning behind this is ADR-007-Service-to-Service-Credentials.

No credential values appear on this wiki. They live in app settings and OpenShift Secrets. This page is procedure only.

Which credential do I need?

You are Use
An application that can hold a Keycloak client (Track, Submit…) A Keycloak service account. This is the standard
A machine that genuinely cannot hold a Keycloak client (the extraction host) A registry API key
ENGAGE, pushing engagements An APIM subscription; see Pushing engagements from ENGAGE
A person Your own IDIR login. Nothing on this page applies

If you are reaching for a registry key because obtaining a CSS client is slower, ask for the client anyway. The key path exists for callers Keycloak cannot serve, not for callers in a hurry.

An uncredentialed request to GET /api/projects is capped at 100 rows a page, and asking for more is refused with a 400 rather than quietly truncated. Any verified credential raises the cap to 1000, where a larger pageSize is clamped instead. Page past either cap with the x-continuation-token header.

GET /api/documents pages with page and count=true instead (x-total-count carries the total). Continuation tokens exist only on GET /api/documents/scan, which needs a credential. See Search-Query-Construction "Native GET /documents".

Which roles do I ask for?

Ask for the least that works. Most integrations read.

Role Grants
demi-service-read Reads everything the ACL allows. Cannot write anything
demi-service-write Reads and writes data. Cannot reach /admin/*
staff / demi-admin / sysadmin Read and write, including delete, and /admin/*
compliance Reads records carrying compliance in their read[]
project:<id> Restricts the caller to that project. Combines with the above

A read-only integration asking for sysadmin will be sent back. demi-service-read exists so that "we only need to read" is expressible rather than aspirational.

Path A β€” Keycloak service account (the standard)

  1. Request a client through BC Gov CSS for realm eao-epic, in each environment you need. Service accounts enabled. Note the client id β€” it becomes your azp, which is what DEMI logs and what the allowlist matches.

  2. Ask the DEMI team to assign roles to the service account, from the table above.

  3. Get a token with client_credentials:

    POST https://<loginproxy-host>/auth/realms/eao-epic/protocol/openid-connect/token
    grant_type=client_credentials&client_id=<your-client>&client_secret=<secret>
    
  4. Cache it. Tokens are short-lived and a token request per API call is wasteful on both sides. met-api caches for 55s against a 60s lifetime β€” copy that shape.

  5. Call DEMI with Authorization: Bearer <token>.

  6. Store the client secret in your platform's secret store β€” an OpenShift Secret on Silver, an app setting on Azure. Never in a repo, never in a Helm values file, never in a pipeline variable.

DEMI_ALLOWED_CLIENTS is the allowlist of permitted azp values. It is empty, which admits every verified client; a client outside a non-empty list keeps its identity but has its privileges stripped. See the ADR for why empty is the shipped default.

Path B β€” Registry API key

Only for callers that cannot hold a Keycloak client.

  1. Ask the DEMI team. Someone with a privileged credential issues one:

    POST /api/admin/api-keys
    { "name": "extraction-host", "roles": ["demi-service-read"] }
    

    Granting a write-capable role additionally requires "allowWrite": true in the body, so a writer is never issued by accident.

  2. The plaintext key is in that response and nowhere else, ever. It is not stored and cannot be recovered. Losing it means being reissued, not looked up.

  3. Send it as X-Api-Key: demi_<env>_<keyId>_<secret>.

  4. Keys expire after 90 days by default, matching the landing zone's maximum secret lifetime. Diarise it. An expired key fails closed with a 401 that reads exactly like a wrong key.

Rotating

Issue the new key, deploy it, confirm traffic has moved (lastUsedAt on the old key stops advancing), then revoke the old one:

DELETE /api/admin/api-keys/<keyId>

There is no update endpoint on purpose β€” this order is what makes rotation possible with no window where you have no working credential.

Revocation is immediate on the instance that processed it and takes up to 60 seconds elsewhere, because verified lookups are cached. If a key is believed compromised, revoke it and treat that minute as real.

Pushing engagements from ENGAGE

ENGAGE reaches DEMI through API Management, not Keycloak. azure/modules/apim.bicep creates one subscription per machine consumer, eagle-api and engage, so each key rotates or revokes alone. APIM asserts the caller as the registry principal apim:engage. The route is behind authMiddleware, requireWrite and requireEngageWriter, which admits only:

  • a registry identity whose id is in DEMI_ENGAGE_PRINCIPALS (default apim:engage; set but empty refuses everyone);
  • that holds demi-service-write;
  • with no project scope.

The registry row apim:engage therefore holds demi-service-write and carries no project scope. eagle-api's principal gets 403 on this route, and apim:engage gets 403 on /api/eagle/*.

PUT /api/engage/engagements/{engagementId}:

{
  "pushedAt": "2026-10-08T17:00:00Z",
  "engagement": {
    "id": 42, "projectId": "588511d0aaecd9001b825604", "trackingId": null,
    "name": "...", "description": "...", "status": 2,
    "start": "2026-10-10 07:00:00", "end": "2026-11-10 07:00:00",
    "metURL": "...", "metURLAdmin": "...", "bannerUrl": "...",
    "isPublished": true, "isDeleted": false
  }
}

engagement.id must equal engagementId in the path. projectId is the Eagle id of the parent project. status is ENGAGE's status_id. start and end are ISO 8601 or YYYY-MM-DD HH:MM:SS; a time with no zone is read as UTC, and any other text is refused. metURL is required unless the engagement is unpublished or deleted. trackingId is the Eagle comment period the engagement already has, if any; the first push adopts that row. pushedAt is an ISO 8601 time, not the millisecond number the Eagle push uses. A push older than the stored engagePushedAt is refused; an equal one writes again. Field storage and ownership: ENGAGE-Rows-Reference.

Status code Meaning
200 An existing row was updated.
201 A row was created.
400 ENGAGEMENT_ID_MISMATCH, PUSHED_AT_INVALID, TRACKING_ID_INVALID, PUSHED_DATES_INVALID, METURL_REQUIRED The body is wrong. Retrying does not help.
403 The caller is not in DEMI_ENGAGE_PRINCIPALS.
404 PARENT_NOT_FOUND, PARENT_REF_INVALID The parent is in neither projects nor notifications. Retry only for PARENT_NOT_FOUND.
409 STALE_PUSH A newer push already landed. The body carries id and engagePushedAt. The only 409: treat it as already current.
422 TRACKING_ID_CLAIMED Another engagement owns that Eagle period.
422 PARENT_KIND_UNSUPPORTED The parent is a project notification. Send that engagement to Eagle directly.
500 DUPLICATE_ID The engagement is stored in more than one row. Nothing was written.
500 The row is stored and queued; the comments under it were not all re-derived. Or the write failed.
503 The row changed while the push retried. Push it again.

A 200 or 201 body is { id, engagementId, eagleId, engagePushedAt, queued }. eagleId is null until sync-out creates the Eagle period. queued names the consumers the row was queued for; it is empty when none is enabled or the enqueue failed (Sync-Architecture).

To rotate the key, read the subscription's keys with az apim subscription list-keys, give ENGAGE the second key, then regenerate the first. The key lives in ENGAGE's secret store, never in a repo.

Reading the public config (no credential)

GET /config/public needs no credential β€” it is the one read on this list that is meant to be called anonymously, by eagle-public at boot. It serves DEMI's live mirror of eagle-api's /api/config, filtered to an allowlist of keys that are already public, and answers 503 rather than a default payload when that mirror is missing. See ADR-012-Public-Config-Surface and Seeding for how the mirror stays current.

Checking it works

Do not accept a probe that cannot fail. GET /api/projects returns data anonymously, so a 200 there proves nothing about your credential β€” it proves DEMI is up.

Use a discriminator instead: call an endpoint whose response differs by privilege, and compare against the same call with no credential. If the two agree, your credential is not being applied.

  • A privileged read that anonymous cannot make: GET /api/search/summary
  • A write, if you were granted one: it should succeed for a writer and return 403 Forbidden. This credential is read-only. for demi-service-read

That 403 is the expected, correct answer for a read-only consumer. It is not a misconfiguration.

Mirroring records from eagle-api

eagle-api pushes each project and document it writes into DEMI, fire and forget: a DEMI outage never fails an Eagle write. All /eagle/* routes below take those pushes, behind authMiddleware, requireWrite, and requireEagleMirror. requireEagleMirror is pinned to eagle-api: the caller must be a registry identity (an APIM subscription or a presented registry key, never a Keycloak user token) whose principal id is listed in the app setting DEMI_EAGLE_MIRROR_PRINCIPALS (default apim:eagle-api, which is eagle-api arriving through API Management), hold demi-service-write, and carry no project scope. Setting DEMI_EAGLE_MIRROR_PRINCIPALS to empty refuses every caller rather than admitting every writer. Everyone else β€” staff users, the break-glass ADMIN_API_KEY, the extractor key, any other demi-service-write key β€” gets 403. eagle-api authenticates with a demi-service-write key, held in the OpenShift secret demi-push-secret in 6cdc9e-prod.

PUT /api/eagle/projects/{eagleId} Mirror one project
PUT /api/eagle/documents/{eagleId} Mirror one document
PUT /api/eagle/commentperiods/{eagleId} Mirror one comment period
PUT /api/eagle/comments/{eagleId} Mirror one comment
PUT /api/eagle/organizations/{eagleId} Mirror one organization
PUT /api/eagle/notifications/{eagleId} Mirror one project notification
PUT /api/eagle/updates/{eagleId} Mirror one recent-activity item

eagleId is the Eagle _id of the record. The body is:

{
  "doc": { "_id": "588511d0aaecd9001b825604", "name": "...", "read": ["public"] },
  "labels": { "type": "Application", "milestone": "Application Review" },
  "pushedAt": 1757894400000
}

doc is the raw Eagle record, exactly as eagle-api stores it β€” DEMI holds the merge and transform rules, so a push and a re-seed produce the same row. doc._id must equal eagleId in the path; a mismatch is a 400. labels applies to documents only: it carries the resolved List names for the record's type, milestone, projectPhase and documentAuthorType ObjectId references, and DEMI stores each label beside its id.

pushedAt is when eagle-api sent the push, in milliseconds since the epoch. DEMI stores it as eaglePushedAt on the row and uses it to order pushes for the same record across eagle-api pods. A push whose pushedAt is strictly older than the stored eaglePushedAt is refused: nothing is written, nothing cascades, and the answer is 200 {"ok": true, "ignored": "stale", "eaglePushedAt": ...}. An equal stamp writes β€” two pushes sharing a millisecond came from one pod, which ordered them itself. pushedAt is optional; a push with none writes too and leaves the stored stamp as it was, so an older client can still update a row a newer client had stamped. A staff caller writing through the ordinary record routes can never set or clear eaglePushedAt (or the cascadePendingAt marker below) β€” both are carried off the stored row on every write that is not an eagle-api push.

A push that lands writes the row under the same etag guard as every other whole-item write in DEMI: up to three tries, each rebuilt against whatever Cosmos actually has after the last one lost its race. A push that loses all three gets 503 {"error": "The record is being written by another request. Push it again."} β€” push the same body again. Every /api/eagle/* route answers with this shape on a lost race, and every route answers a landed push with 200 {"id": "...", "action": "upsert"} ("delete" for a delete push on documents and comment periods). Pushing the same eagleId again replaces the record, so a retry after any answer other than a lost etag race is safe.

A project push that lands but whose visibility cascade onto its documents fails answers 500, and DEMI marks the project cascadePendingAt so the cascade is not lost. The next push for that project that lands runs the cascade again and clears the marker.

A comment period, comment, organization or notification push whose required parent is not yet in DEMI (a comment period's project, a comment's period) is refused with a 404: push the parent first. The 404 body is { error, code }: code is PARENT_NOT_FOUND when no admissible parent row exists (retry once the parent lands) and PARENT_REF_INVALID when the parent reference is empty or not a 24-character hex id (retry cannot help). Details: Eagle-Mirror-Reference#refusal-codes. List has no push and no route β€” Eagle has no write controller for it, so DEMI's only writer of List rows is the backfill β€” see Public-Read-Backfill.

What DEMI decides, not the push:

  • Identity. A project matched to Track keeps the Track project id; an unmatched one is keyed eagle-<eagleId>. A document keeps its Eagle _id.
  • Visibility. read[] comes from the pushed record, and a document's is intersected with its parent project's β€” a document can never out-rank its project. isPublished mirrors read[]. A project whose publication state changes cascades onto its documents.
  • Extraction state. Carried off the row already in DEMI, never from the push.
  • Enrichment. Source blocks DEMI adds itself, such as sources.wildfire, survive a push.

A document push whose parent project is not in DEMI returns 404 with the same code field: push the project first.

⚠️ **GitHub.com Fallback** ⚠️