ADR 013 Stored Project Summaries - digitalspace/eagle-demi GitHub Wiki
Proposed. Demo on test, 2026-09-08: one project, Site C Clean Energy (DEMI id 272, Eagle
588511a0aaecd9001b82316d), generated once by a script and stored. No timer, no backfill, no
prod path yet — see Path to production below.
The reference page this models is a per-project status page: eyebrow, status card, timeline, conditions, First Nations consulted, compliance, key documents. Some of that page is facts already on the project record and document index. Some of it is prose or an extracted list that only a model can produce — a condition summary, a paragraph on the newest inspection, a status sentence.
DEMI already has one AI surface, the query-time summarizer over BM25 results (ADR-006-AI-Summarization-over-BM25). This is a second one, shaped differently: it summarizes one project's own documents, not a search result set, and it does not run on every page view.
Generate each project's summary once, offline, by a script. Store the result. The page reads the stored record and never calls a model.
- A generation script reads the project record, picks source documents by type and title pattern
(not by id — the ids in the prototype are Site C's, and an id-keyed picker would generate one
project and silently return nothing for the other 381), pulls every extracted chunk of each
picked document, and makes one model call per section, each call scoped to a single document's
chunks. Output is one record in Cosmos container
projectSummaries, keyed by project id. -
GET /api/projects/:id/summaryreturns the stored record, 404 when none. Facts on the page (document counts, dates) come fromGET /projects/:idand the document index, read live. - A section with no matching source document gets no model call and stores
null. A model asked to summarize nothing answers from its own knowledge, and on a regulatory registry that is the failure mode that matters most. Sections are generated only from documents whose text has already been extracted; a section with no usable source is stored asnullwith a reason recorded insectionErrors:no_document(nothing to pick from),not_extracted(a document was picked but has no chunks yet —sectionSources.<name>then carries{documentId, displayName}so the page can link the file),no_chunks,no_source,empty,not_json,truncated,context_overflow, plus the federal-only reasons below.
This is why the page is instant, cost is bounded per generation rather than per view, and the prototype needs no new infrastructure beyond one container.
facts — document counts, dates, document ids — comes from the documents index, computed in
src/ai/project-summary.js. The model never sees these numbers and cannot contradict them. Only
prose and extracted lists that no query can produce go to the model: a status sentence, grouped
condition summaries with one-liners, a compliance paragraph from the newest Inspection Record, and
First Nations names.
The generator uses only published, public-readable documents for every section and for the facts
block. The record carries sourceAccess: "public". The generator's --section merge (regenerating
one section of an existing record) refuses to merge into a stored record that lacks the field.
Every model reply passes three checks, each dropping rather than repairing:
- JSON shape. A reply that is not the section's declared shape is rejected whole.
- Citation range. A citation number outside the numbered source list is dropped, not renumbered onto a real source.
- Number and date grounding. A bullet carrying a figure or date that does not appear in the document it cites is dropped whole. The check runs against every chunk of the cited document, not only the ones shown to the model for that call — a model can see a window of a long document and cite a figure that is printed again on a page outside that window. A hallucinated condition reads exactly like a real one until its numbers are checked against the source.
Citations are numbered once, record-wide, across every section, so a section's citation list and the footer's source list mean the same source.
Key-document links (facts.keyDocuments) prefer an English title where the registry filed both; a
role filled only by a French document carries languageFlag: 'fr' so the page can say which
language it is linking.
timelineEvents reads the whole source document in batches sized to the model's context window,
rather than only its first chunks, because a project's chronology can put its last events on the
document's last page; the batches are merged back into one timeline, newest first, with duplicate
events collapsed.
DEMI holds five Decision Statements in its whole corpus, but every project that had a federal
assessment has one on the public Impact Assessment Agency of Canada registry
(iaac-aeic.gc.ca/050/evaluations, HTML only, no API). When DEMI has no extracted federal document
for a project, the generator (src/ai/federal-source.js) reads the registry instead: it resolves
the project's CEAR id from the project's CEAALink field, or by a title-matched search when that
field is blank; reads the project's key-document list and the full document list as a fallback;
picks the newest Decision Statement (an "Updated" one wins only when it is at least as new as the
plain one); and converts that document's PDF to text with pdftotext -layout, one chunk per PDF
page so a citation carries the page number the registry itself printed. FEDERAL_SOURCE=off turns
this off entirely, DEMI-only. The whole path is capped at 5 requests per project, 5 seconds apart
per the registry's robots.txt Crawl-delay, and results are cached for the run so a --section federal rerun refetches nothing.
Citations drawn from the registry carry source: 'iaac' and the PDF's url, instead of the
documentId a DEMI citation carries. pdftotext is a runtime prerequisite for this path; a host
without it answers no_pdf_extractor and the section is not generated.
Section reasons specific to federal: no_federal_decision (the registry lists the project but no
Decision Statement at all); federal_decision_unreadable (a Decision Statement row exists but its
PDF gave no text — logged at warn with which of no_pdf_link, pdf_fetch_failed:<status>,
request_budget_spent, no_pdf_extractor, pdf_extract_failed, or no_text caused it); and
federal_source_unavailable (the registry could not be reached or its page could not be parsed).
When the registry has no matching project either, the section reports the same absence DEMI would
have reported (no_document or not_extracted).
No Eagle or DEMI field says which nations a project consulted. The generator runs a keyword search
for "First Nation" over the project's chunks in the search index, reads the matching public
documents' passages, and extracts names from those, capped at 40 passages. The model returns names
and citations only, [{name, citation}]. Code then joins each name to a DEMI lists container Organization row
(kind Organization, companyType "Indigenous Group", 246 rows on test), by a normalised name:
diacritics stripped, apostrophes and colons removed, a leading "the" and trailing generic words
("First Nation", "Band", "Tribal Council") dropped, then resolved through a small alias table for
pairs no normalisation rule reduces to each other (e.g. "Beecher Bay" / "Scia'new", "Sooke" /
"T'Sou-ke"). Address and website come from the matched Organization row, never from the model — a
wrong postal address on a government page about a First Nation is a different class of error from a
wrong summary sentence, and the join makes it structurally impossible. An unmatched name renders
with its citation and no contact card; the generator logs it as input to the alias table.
PROJECT_SUMMARY_PROVIDER selects foundry (the account ADR-006-AI-Summarization-over-BM25
already runs) or ollama, a LAN host (http://192.168.3.8:11434, no auth, reachable only from a
workstation). The one-time demo run uses Ollama so it needs no devbox or Foundry access. Whatever
provider ran, the record always prices usage tokens at the Foundry CAD rates and stores which model
actually ran (model) separately from what the cost is priced against (pricedAs: 'gpt-4.1-mini'),
so a run with no cash cost and a deployed run stay comparable.
Cosmos and AI Search are private-endpoint only, so the generator does not talk to them directly. It
talks to the DEMI API instead, through an adapter (src/ai/project-summary-sources.js): GET /api/documents/:id/chunks for extracted text, PUT /api/projects/:id/summary to store the record.
The same ACL that gates a normal API caller gates the generator's reads and write.
- The page is a point read. No page view triggers a model call, no page view has model latency or per-view cost.
- Freshness is a property of when the generator last ran, not of the page. A project whose documents changed after generation shows a stale summary until regenerated. Nothing catches this in the prototype; see Path to production.
- Cost is bounded and predictable per generation, not per view. Unverified estimate: Schedule B plus a few documents runs 50-100k prompt tokens per project (0.05-0.10 CAD); backfilling all 382 projects would run about 40 CAD.
- A summary row carries no ACL of its own — it inherits the ACL of the documents it was generated
from, indirectly, through the project. The read endpoint re-checks the caller's access against
the project, not the stored record, exactly as
GET /projects/:iddoes. A caller who cannot read the project gets 404, whether or not a record exists. -
PUT /projects/:id/summaryrejects a record withoutsourceAccess: "public"(400).GET /projects/:id/summarywithholds a stored record that does not carry it, so the project-level read gate above is sufficient. - Reuses the Foundry account, token helper, and cost estimator already built for ADR-006-AI-Summarization-over-BM25; adds one Cosmos container and one adapter layer, not a second model deployment.
Rejected. It couples page latency to model latency, costs once per view instead of once per generation, and gives no natural point to add a human review gate before publication. The reference page's content (status, conditions, compliance) does not change between visits in the same day, so paying per-view buys nothing.
Rejected. ADR-009-Track-Feed does not cover this and no such feed is asked for. Track indigenous nation data is not pulled and not planned; the source of truth for which nations a project consulted is that project's own documents, not a separate feed.
Ordered by dependency. The prototype above is stage 0.
-
Source filter. Generate only from documents the public can read (
read[]public, published parent), so stored text can never carry staff-only content. - Quality gates in the generator. JSON schema validation and the grounding checks above, already built; add an eval set (Site C plus five projects of different ages, rubric scored), re-run on any prompt or model change.
-
Human review. A
reviewedflag and per-section hide on the record; an edit-and-approve screen ineagle-demi-admin; nothing shows to the public until reviewed. Alias table for nation names with an unmatched report. -
Refresh. Backfill script over all projects, then a timer (
refreshProjectSummarieson%PROJECT_SUMMARY_SCHEDULE%) for projects whose documents index high-water moved sincegeneratedAt; re-review required after refresh, old version kept. -
Serving to the public. Anonymous
GET /api/projects/:id/summaryunder the existing redaction boundary, with cache headers; eagle-public gets a summary tab behind the sameSEARCH_API_PATH-style kill switch eagle-api mode already uses. -
Prod AI account. Needs BC Gov AI Services Hub approval and the private-endpoint dance
(ADR-006-AI-Summarization-over-BM25);
summaryEnabledis off in prod today. - Ops. A trace per generation (project, tokens, cost, duration, failures); alert on failure rate and on the environment's AI budget alert; a runbook in this wiki.
- Disclosure and feedback. A consistent "machine-generated, reviewed by EAO staff on " label, a link to the cited documents, a report-an-error link, wording checked against BC Gov generative AI guidance before public release.
- Docs and tests. This ADR kept current, the Frontend page, an eagle-dev-guides.wiki entry for the eagle-public side; tests on the generator, the gates, and the screen.
- The demo covers one project, Site C. The generator picks source documents by type and title pattern rather than by id specifically so it runs unmodified against any project id, but only Site C has been exercised.
- Site C has no Section 11 Order document in DEMI's registry, so First Nations names come from the certificate or Assessment Report instead, which is a weaker source for that specific fact.
-
phaseHistoryrows carry no date; the timeline merges dated fact rows with extracted dated events from the documents, but the phase rows themselves stay undated. - Ollama's default context window is 4k tokens and truncates a longer prompt silently, with no
error and no warning — the only symptom is a short answer that looks fine. Schedule B alone runs
around 20k tokens, so
num_ctxmust be set explicitly on every Ollama call (PROJECT_SUMMARY_OLLAMA_CTX, default 65536 in this codebase).
- ADR-006-AI-Summarization-over-BM25 — the Foundry account, token helper, and cost estimator this reuses
- ADR-009-Track-Feed — why a Track nations feed is not the answer to First Nations consulted
- Frontend — the Project summary screen
-
src/ai/federal-source.js— the IAAC registry reader for thefederalsection's fallback