Sources and Status - digitalspace/eagle-demi GitHub Wiki

Sources and Status

Every data source and served surface in DEMI, each in exactly one state. Counts measured 2026-08-26 against the live test API. Prod holds Eagle and Track data only. The demo frontend and every enrichment source are a test-only surface โ€” supported there, not deployed to prod.

States:

  • Served โ€” reachable from eagle-public or the demo frontend; must work end to end and have a test.
  • Proven, parked โ€” exists to prove a source can be added; runs on demand, writes nothing a served surface reads.
  • Gone โ€” nothing references it; code, container and Bicep go together.

Surfaces

Surface Environment Routes consumed
eagle-public (bcgov/eagle-public) test GET /api/search only, datasets Project, Document, DocumentChunk (src/app/services/api.ts:52), plus GET /api/documents/recent-uploads for the homepage panel โ€” see Recent-Uploads. Item reads, downloads, RecentActivity and ProjectNotification stay on eagle-api. Prod site search reads /demi-search, the same route as test.
Demo frontend /map, /search, /summary, /intake test only /config, /boundaries*, /search, /search/summary, /documents/:id/download
eagle-api mirror push test PUT /eagle/projects/:eagleId, PUT /eagle/documents/:eagleId โ€” see Sync-Architecture
Extractor / intake (GPU box) test POST /documents, /documents/extract, /documents/:id/chunks, PUT /documents/:id, PUT /documents/:id/published
Admin test /db/stats, /admin/index-progress, /admin/api-keys*, POST /admin/sync/wildfires, /admin/audit, /admin/analytics, /admin/cost
No consumer โ€” GET /projects, /projects/:id, /documents, /documents/:id โ€” anonymous, API-first surface; list pages capped at 100 rows without a credential. /api-docs is test-only and 404s in prod

Sources

Source State Depended on by Evidence
projects container + projects index Served both frontends 393 rows, 348 of them anonymously readable
documents + index Served both frontends, download 61,587 rows
chunks + index Served deep search, DocumentChunk dataset, summariser 1,128,576 rows, matching the index; the only extracted copy
epictrack-api GET /api/v1/projects Served Track-sourced project identity and fields read nightly by sync-track-teams.js with the demi-track-reader bearer; the mirror is sync-track-projects.js
epictrack-api GET /api/v1/works + /works/<id>/phases Served phases on project rows same run, same bearer โ€” ยง Track work phases below
Track static file src/data/track_projects_enriched.json Served (seed) 382 Track-sourced projects snapshot 2026-07-29; the fallback when no reader client is configured, and it carries no work phases
eagle-api push Served project and document mirroring PUT /eagle/{projects,documents}/:eagleId โ€” Sync-Architecture
Eagle seed src/scripts/seed-nosql.js Served (on demand) corpus refresh, disaster recovery reads prod eagle-api's public API; preserves extraction state; --reconcile deletes surplus under a ceiling
Eagle-only projects Served project list, project search, document search, point reads sourceSystem: 'eagle', id prefix eagle-; counted as unlinkedProjects by GET /api/db/stats
boundaries Served, test only demo /map 281 rows (Municipality 160, Electoral District 93, Regional District 28), simplifiedGeometry only; the prod container is declared and empty
config Served /api/config on every page load 1 row
apikeys Served (infra) authMiddleware โ€”
sources.wildfire on project rows Served, test only demo /map panel activeCountWithin50km, nearestDistanceKm, firesOfNoteNearby, lastCalculatedAt, written by sync-wildfires.js and read under those names by registry.models.ts
wildfires container + sync-wildfires.js Proven, parked nothing reads the container write-only repository; no schedule; only POST /admin/sync/wildfires
DataBC wildfire WFS Served (external, browser-side) demo /map fire layer fetched in map-explorer.component.ts:888; no DEMI storage
NRPTI Gone โ€” no ingest path; a handful of comment and fixture mentions remain

Track work phases

phases on a project row is the assessment schedule eagle-public draws its progress rail from. Track is the only source; Eagle has no equivalent. Eagle's own currentPhaseName and phaseHistory are a separate record and are untouched by this.

Two epictrack-api endpoints, on the same bearer as the project list:

  1. GET /api/v1/works, once. Each work carries project_id, work_type_id and is_active.
  2. GET /api/v1/works/<id>/phases, once per project โ€” for the one work whose work_type_id is 6 (Assessment). Where a project has two assessment works the active one wins, then the highest work id.

GET /api/v1/work-phases cannot replace the pair: its rows carry no project id, and it returns only completed, legislated phases that ran over or under budget. It is the overage report. GET /api/v1/projects?with_works=true only filters the project list; its response carries no works.

Fields, per phase, ordered by sortOrder:

Field Read from
name work_phase.name, else the phase code's name
eaActId, eaActName work_phase.phase.ea_act
workType work_phase.phase.work_type.name
startDate, endDate work_phase.start_date / end_date, ISO 8601 or null
numberOfDays work_phase.number_of_days
legislated work_phase.legislated
sortOrder work_phase.sort_order
isCompleted work_phase.is_completed

Nothing else off the Track row leaves DEMI โ€” staff, responsibility notes, milestone detail and day counts stay upstream. The whole array is public (src/vis/catalog/projects.js, defaultVis: 4).

sync-track-teams.js pulls the phases and sync-track-projects.js writes them alongside the other Track-owned fields. A run that pulls none leaves the stored ones alone, so a Track outage cannot blank the rail. trackPhases= in that run's summary line counts the projects it had phases for; reconcile-eagle.js reports the same count read-only, which is the only phase signal in prod, where the sync carries no schedule.

GET /api/projects/:id accepts an Eagle id. A 24-character hex :id is resolved through eagleId. A Track project number or eagle-<ObjectId> is a DEMI id and reads as before. The two id spaces cannot collide, and both answer under the same ACL and the same field redaction, so a caller holding an Eagle id needs no translation step to read a project.

Prod purity

sources.* on a project row is the only place non-Eagle/Track data touches a served row, and publicView (src/repositories/projects.js) allowlists it against the ENRICHMENT_SOURCES app setting โ€” wildfire in test, empty in prod. A corpus copied from test into prod therefore serves no enrichment data even if the rows carry it.

Prod Bicep declares projects, documents, chunks, config, apikeys and boundaries. deployEnrichment: false in azure/main.prod.bicepparam means prod has no wildfires container at all.

Contract for adding an enrichment source

Documented here, not as a code layer โ€” one source exists and its DataBC parsing is most of it.

  1. src/scripts/sync-<name>.js, runnable from the CLI and POST /admin/sync/<name> (authMiddleware + requireWrite). No scheduler unless a consumer needs freshness.
  2. Writes only sources.<name> on projects, always with lastCalculatedAt. A scratch container is optional and TTL'd.
  3. publicView and ENRICHMENT_SOURCES gate what leaves the API.
  4. The consumer renders the stamp and handles absence โ€” absence is what prod looks like.
  5. One test asserts the written shape is the shape the consumer reads.
โš ๏ธ **GitHub.com Fallback** โš ๏ธ