Search and Retrieval - digitalspace/eagle-demi GitHub Wiki
Search and retrieval
Azure AI Search Basic, canadacentral, keyless, private endpoint only — demi-search-test in test,
demi-search-prod in prod. Estates: Environment-Reality-and-Operational-Gotchas.
Retrieval is classic lexical BM25 over three indexes. There are no vector fields and no embeddings.
Chunk results are reordered by Azure's semantic ranker — a reranker over the top 50 BM25 hits,
not a second retrieval path. It cannot surface anything the keyword query missed, and the measured
found@50 is identical with it on and off. AI's role is a final summarizer over results the lexical
index already returned. Cosmos full-text search was evaluated and rejected —
ADR-005-Cosmos-Full-Text-Search-Ruled-Out.
Pipeline
The summarizer is the fifth step and the only one that touches a model. Steps 1–4 do not:
| Step | Function | AI? | |
|---|---|---|---|
| 1 | tokenize, build the Lucene query | ai-search.tokenize, ai-search.buildQuery |
no |
| 2 | BM25 + the OData ACL filter | ai-search.runSearch, access-odata.filterFor |
no |
| 3 | top N chunk ids, ranked by BM25 score | ai-search.searchChunks |
no |
| 4 | fetch those chunks' text from Cosmos | repositories/chunks.getById |
no |
| 5 | LLM composes a short answer with [n] citations |
controllers/search.summarize |
yes |
Step 4 point-reads from Cosmos as the second ACL gate. content is retrievable, so it is the
select list in searchChunks that keeps chunk text out of responses. Delete the model deployment
and steps 1–4 still work; only the summary panel disappears. Design, rejected alternatives and
limitations: ADR-006-AI-Summarization-over-BM25.
Why AI Search
Costed against the real corpus with the Azure retail pricing API, canadacentral, measured 2026-07-30.
| Option | Cost/mo | Fuzzy | Verdict |
|---|---|---|---|
| Cosmos DB full-text search | ~$9 | no — silent no-op | ruled out, ADR-005-Cosmos-Full-Text-Search-Ruled-Out |
| Azure AI Search Basic | ~$75–81 | yes, GA | chosen. Also gives highlighting, faceting and synonyms natively |
| Typesense Dedicated D8 | ~$700 | yes, GA | ruled out, ADR-001-Typesense-for-DFL-Search |
Typesense keeps indexed fields in RAM with no on-disk mode, and its own sizing rule is 2–3× RAM against searched-field size, which puts this corpus at 17–26 GB against the 8 GiB Container Apps Consumption ceiling. Its clustering is replication rather than sharding, so a 3-node cluster is 3× the RAM, not a third each.
DEMI search is not EPIC.search
EAO's other search product, EPIC.search, is a vector/RAG stack and shares none of this
implementation. Where DEMI's rules differ from it is in EPIC-Search-Business-Rules.
Detail pages
- Search-Index-Reference — indexes, indexers, data sources, api-versions, analyzers, field metadata and what the schema cannot express.
- Search-Query-Construction — how a request becomes a Lucene query and an OData filter, plus highlighting, deletes, paging and the search controller's decisions.
- Search-Measurements — the retrieval scorecard, the hypotheses tested and rejected, and the rules a probe has to follow.
See also
- Architecture — containers, ACL model, deletion semantics
- BC-Gov-Azure-Landing-Zone — private endpoints, DNS and the policies that force them
- Extraction-Pipeline — how the indexed text is produced
- ADR-001-Typesense-for-DFL-Search — superseded, kept for why Typesense was chosen and dropped