ADR 006 AI Summarization over BM25 - digitalspace/eagle-demi GitHub Wiki
Accepted 2026-08-04. Built and on main (verified 2026-08-26): GET /api/search/summary in
src/http/routes.js, summarizer in src/ai/summarize.js. Enabled in test
(SUMMARY_ENABLED=true, Foundry account demi-foundry-test) and off in prod
(SUMMARY_ENABLED=false, deployFoundry = false in azure/main.prod.bicepparam, no Foundry
account in rg-demi-prod). Anonymous callers get 401.
Extends ADR-005-Cosmos-Full-Text-Search-Ruled-Out and Search-and-Retrieval: retrieval is unchanged by this decision. It only adds a step after it.
Search returns three columns of hits and nothing that answers the question. The goal is a short generated summary above the results, grounded in the retrieved chunks and citable back to them.
The constraint that shapes everything below was settled before this ADR and is not reopened by it:
Retrieval stays lexical BM25. AI is a summarizer over the final top-N, not a retriever.
There are no embeddings and no vector field. Chunk queries do ask Azure's semantic ranker to
reorder what BM25 already found β semanticQuery against the <index>-semantic configuration
declared in azure/search/indexes/chunks.json, enabled by default for chunk queries in
src/search/ai-search.js. Azure rescores at most the top 50 of the BM25 result, so it reorders and
can never surface a document BM25 missed.
Bring our own summarizer: a small chat model deployed in a Foundry account, called from demi-api
over the top chunks a normal BM25 search already returned.
"pipeline safety near watercourse"
1. tokenize and build the Lucene query src/search/ai-search.js plain code
2. BM25 plus the OData ACL filter src/search/ai-search.js lexical, no AI
3. top 8 chunk ids by score no AI
4. read those 8 chunks from Cosmos src/repositories/chunks.js DB read, no AI
5. LLM: 8 chunks to 3 sentences with [n] citations the only AI call
The model never selects documents, re-ranks, rewrites the query, or sees anything outside those
eight chunks (SUMMARY_MAX_CHUNKS, 1,500 characters each). Delete the Foundry account and steps
1-4 still work; only the summary panel disappears.
| Model |
gpt-4.1-mini (2025-04-14), Standard (regional) deployment. Not gpt-5-mini or 4o-mini β both are GlobalStandard-only in canadaeast, which would move inference out of country |
| Region | canadaeast |
| Network |
publicNetworkAccess: Disabled, inbound private endpoint into the canadacentral 10.0.2.0/24 PE subnet |
| Auth | Keyless β the demi-identity-<env> UAMI with Cognitive Services User
|
| Callers | Privileged only. No anonymous or public AI in version 1 |
| Cost | ~0.0023 CAD/query, ~2.32 CAD/month at 1,000 queries. Derived, not billed: 2,835 prompt and 124 completion tokens measured 2026-08-05, priced against the canadaeast gpt 4.1 mini Inp/Outp regnl retail meters (0.70 and 2.70 CAD per 1M, checked 2026-08-06). CAD because the subscription bills in CAD |
This is Microsoft's own documented RAG pattern β search retrieves, the app calls the LLM, the LLM composes. The retrieval half is deliberately more conservative than Microsoft's guidance, which pushes hybrid BM25 plus vector.
Both were evaluated. Neither avoids provisioning our own LLM.
Extractive, no LLM to provision, 1,000 free requests a month and then per-1,000 on Basic and above. It is the only option that needs no Foundry account, and it does not do the job:
- It does not summarize. The docs are explicit: "verbatim content in your search index⦠It's not a generated answer", and the extraction model "doesn't compose new language".
- It only fires on question-shaped queries. DEMI's box is a keyword box;
pipelinereturns@search.answers: []every time, because answer processing is skipped when the string does not look like a question. - It needs
queryType: semantic, which replaces the Lucene expression and costs the fuzzy matching that ADR-005-Cosmos-Full-Text-Search-Ruled-Out rejected an entire cheaper database over. The reranker DEMI uses keeps the Lucene expression precisely to avoid this.
This one genuinely generates summaries with [ref_id:n] citations. It does not save the Foundry
account: the knowledge base specifies our deployment, the search service needs Cognitive Services User on our resource, and synthesis bills to our Azure OpenAI. It changes who places the call, not
whether we provision the model.
What it adds on top:
- A preview-only API.
- LLM query planning replacing BM25, plus mandatory semantic reranking of every subquery.
- A second token meter on AI Search, on top of the Azure OpenAI one.
- Knowledge base and knowledge source objects that live only in the live service, with no export
path back into the repository. Index, indexer and data-source definitions are exported and
committed under
azure/search/; the API offers no equivalent for knowledge bases.
A normal Azure RAG app gets chunk text back in the search response. Ours does not: the select
list on the chunk query omits content, so step 4 point-reads up to 8 chunks from Cosmos. That is
cheap β 2 to 5 ms each on a partitioned container β and it is also the second ACL gate, because
chunks.getById(access, β¦) re-applies the predicate at the database. Keep it.
Recall@10 is 0.620 on 71 labels, one standard error about 0.059 β see Search-and-Retrieval. If BM25 misses the right chunk, the model will fluently summarize the wrong ones.
The grounding probe proves the model refuses when handed nothing. It says nothing about the model being handed the wrong thing, and no cheap probe does. The honest mitigation is more labels. Do not read a confident summary as evidence that retrieval was correct.
A highlighted fragment discloses a phrase; a synthesized cross-chunk paraphrase discloses substance. Gating version 1 to privileged callers holds cost, abuse and disclosure to a small known population. Opening it later is a middleware swap plus the ACL probe in ADR-004-Read-ACL-Authorization-Model.
ADMIN_API_KEY grants sysadmin, staff and demi-admin unconditionally, so anything holding
that key gets unfiltered summaries. That predates this decision, but the summarizer widens what one
request with it can pull out.
The search leg is ACL-filtered by filterFor(access) and the Cosmos read re-checks through
chunks.getById(access, β¦). Neither is belt-and-braces; keep both. The empty flag must
short-circuit before any request is issued, because OData has no false literal β a null or empty
filter is unrestricted.
- Provisioning a
Microsoft.CognitiveServicesaccount is what the BC Gov AI Services Hub process governs (https://bcgov.github.io/ai-hub-tracking/). Submit before provisioning β see BC-Gov-Azure-Landing-Zone. -
Deny-PublicPaaSEndpointsrejects the deployment withRequestDisallowedByPolicyunlesspublicNetworkAccess: 'Disabled'. Do not declare a private DNS zone; the landing zone attaches one by policy and the A record appears about 10 minutes later, which looks like a missing zone if checked early.
Everything else in the resource group is canadacentral. canadacentral has no Standard (regional)
pay-per-token deployments β only global-standard, which routes inference to any Microsoft region
worldwide. canadaeast is the only Standard option in the Canada geography, so it is the only way to
keep inference in country. Private endpoints work across regions, so the endpoint still lands on
the canadacentral PE subnet.
- Search-and-Retrieval β the pipeline this attaches to, and the recall numbers
- ADR-004-Read-ACL-Authorization-Model β the ACL both gates compose
- ADR-005-Cosmos-Full-Text-Search-Ruled-Out β why fuzzy is a hard requirement
-
BC-Gov-Azure-Landing-Zone β
Deny-PublicPaaSEndpoints, DNS zone attachment