ADR 006 AI Summarization over BM25 - digitalspace/eagle-demi GitHub Wiki

ADR-006: AI summarization over BM25 results

Status

Accepted 2026-08-04. Built and on main (verified 2026-08-26): GET /api/search/summary in src/http/routes.js, summarizer in src/ai/summarize.js. Enabled in test (SUMMARY_ENABLED=true, Foundry account demi-foundry-test) and off in prod (SUMMARY_ENABLED=false, deployFoundry = false in azure/main.prod.bicepparam, no Foundry account in rg-demi-prod). Anonymous callers get 401.

Extends ADR-005-Cosmos-Full-Text-Search-Ruled-Out and Search-and-Retrieval: retrieval is unchanged by this decision. It only adds a step after it.

Context

Search returns three columns of hits and nothing that answers the question. The goal is a short generated summary above the results, grounded in the retrieved chunks and citable back to them.

The constraint that shapes everything below was settled before this ADR and is not reopened by it:

Retrieval stays lexical BM25. AI is a summarizer over the final top-N, not a retriever.

There are no embeddings and no vector field. Chunk queries do ask Azure's semantic ranker to reorder what BM25 already found β€” semanticQuery against the <index>-semantic configuration declared in azure/search/indexes/chunks.json, enabled by default for chunk queries in src/search/ai-search.js. Azure rescores at most the top 50 of the BM25 result, so it reorders and can never surface a document BM25 missed.

Decision

Bring our own summarizer: a small chat model deployed in a Foundry account, called from demi-api over the top chunks a normal BM25 search already returned.

"pipeline safety near watercourse"
  1. tokenize and build the Lucene query     src/search/ai-search.js   plain code
  2. BM25 plus the OData ACL filter          src/search/ai-search.js   lexical, no AI
  3. top 8 chunk ids by score                                          no AI
  4. read those 8 chunks from Cosmos         src/repositories/chunks.js  DB read, no AI
  5. LLM: 8 chunks to 3 sentences with [n] citations                   the only AI call

The model never selects documents, re-ranks, rewrites the query, or sees anything outside those eight chunks (SUMMARY_MAX_CHUNKS, 1,500 characters each). Delete the Foundry account and steps 1-4 still work; only the summary panel disappears.

Model gpt-4.1-mini (2025-04-14), Standard (regional) deployment. Not gpt-5-mini or 4o-mini β€” both are GlobalStandard-only in canadaeast, which would move inference out of country
Region canadaeast
Network publicNetworkAccess: Disabled, inbound private endpoint into the canadacentral 10.0.2.0/24 PE subnet
Auth Keyless β€” the demi-identity-<env> UAMI with Cognitive Services User
Callers Privileged only. No anonymous or public AI in version 1
Cost ~0.0023 CAD/query, ~2.32 CAD/month at 1,000 queries. Derived, not billed: 2,835 prompt and 124 completion tokens measured 2026-08-05, priced against the canadaeast gpt 4.1 mini Inp/Outp regnl retail meters (0.70 and 2.70 CAD per 1M, checked 2026-08-06). CAD because the subscription bills in CAD

This is Microsoft's own documented RAG pattern β€” search retrieves, the app calls the LLM, the LLM composes. The retrieval half is deliberately more conservative than Microsoft's guidance, which pushes hybrid BM25 plus vector.

Why not the built-in AI Search features

Both were evaluated. Neither avoids provisioning our own LLM.

Semantic answers and captions

Extractive, no LLM to provision, 1,000 free requests a month and then per-1,000 on Basic and above. It is the only option that needs no Foundry account, and it does not do the job:

  1. It does not summarize. The docs are explicit: "verbatim content in your search index… It's not a generated answer", and the extraction model "doesn't compose new language".
  2. It only fires on question-shaped queries. DEMI's box is a keyword box; pipeline returns @search.answers: [] every time, because answer processing is skipped when the string does not look like a question.
  3. It needs queryType: semantic, which replaces the Lucene expression and costs the fuzzy matching that ADR-005-Cosmos-Full-Text-Search-Ruled-Out rejected an entire cheaper database over. The reranker DEMI uses keeps the Lucene expression precisely to avoid this.

Agentic retrieval with outputMode: "answerSynthesis"

This one genuinely generates summaries with [ref_id:n] citations. It does not save the Foundry account: the knowledge base specifies our deployment, the search service needs Cognitive Services User on our resource, and synthesis bills to our Azure OpenAI. It changes who places the call, not whether we provision the model.

What it adds on top:

  • A preview-only API.
  • LLM query planning replacing BM25, plus mandatory semantic reranking of every subquery.
  • A second token meter on AI Search, on top of the Azure OpenAI one.
  • Knowledge base and knowledge source objects that live only in the live service, with no export path back into the repository. Index, indexer and data-source definitions are exported and committed under azure/search/; the API offers no equivalent for knowledge bases.

Consequences

One extra round-trip, by design

A normal Azure RAG app gets chunk text back in the search response. Ours does not: the select list on the chunk query omits content, so step 4 point-reads up to 8 chunks from Cosmos. That is cheap β€” 2 to 5 ms each on a partitioned container β€” and it is also the second ACL gate, because chunks.getById(access, …) re-applies the predicate at the database. Keep it.

The summarizer inherits BM25's recall ceiling, and no probe covers it

Recall@10 is 0.620 on 71 labels, one standard error about 0.059 β€” see Search-and-Retrieval. If BM25 misses the right chunk, the model will fluently summarize the wrong ones.

The grounding probe proves the model refuses when handed nothing. It says nothing about the model being handed the wrong thing, and no cheap probe does. The honest mitigation is more labels. Do not read a confident summary as evidence that retrieval was correct.

Privileged-only bounds the blast radius

A highlighted fragment discloses a phrase; a synthesized cross-chunk paraphrase discloses substance. Gating version 1 to privileged callers holds cost, abuse and disclosure to a small known population. Opening it later is a middleware swap plus the ACL probe in ADR-004-Read-ACL-Authorization-Model.

ADMIN_API_KEY grants sysadmin, staff and demi-admin unconditionally, so anything holding that key gets unfiltered summaries. That predates this decision, but the summarizer widens what one request with it can pull out.

Two gates, both load-bearing

The search leg is ACL-filtered by filterFor(access) and the Cosmos read re-checks through chunks.getById(access, …). Neither is belt-and-braces; keep both. The empty flag must short-circuit before any request is issued, because OData has no false literal β€” a null or empty filter is unrestricted.

Notes

  • Provisioning a Microsoft.CognitiveServices account is what the BC Gov AI Services Hub process governs (https://bcgov.github.io/ai-hub-tracking/). Submit before provisioning β€” see BC-Gov-Azure-Landing-Zone.
  • Deny-PublicPaaSEndpoints rejects the deployment with RequestDisallowedByPolicy unless publicNetworkAccess: 'Disabled'. Do not declare a private DNS zone; the landing zone attaches one by policy and the A record appears about 10 minutes later, which looks like a missing zone if checked early.

Why canadaeast and not canadacentral

Everything else in the resource group is canadacentral. canadacentral has no Standard (regional) pay-per-token deployments β€” only global-standard, which routes inference to any Microsoft region worldwide. canadaeast is the only Standard option in the Canada geography, so it is the only way to keep inference in country. Private endpoints work across regions, so the endpoint still lands on the canadacentral PE subnet.

References

⚠️ **GitHub.com Fallback** ⚠️