Cosmos migration record

Closed 2026-08-04. DEMI moved off Cosmos DB's MongoDB API to its NoSQL API with @azure/cosmos, replaced Typesense with Azure AI Search, and decommissioned the old account.

This page is a record of that work and the only page on this wiki that keeps dates. It describes nothing about how DEMI runs today — for that see Architecture, Search-and-Retrieval and Environment-Reality-and-Operational-Gotchas.

An earlier DEMI_PLAN.md claimed for months that this migration had already happened. It had not, and the half-finished attempt produced a SQL-string-over-Mongo-driver shim whose dropped predicates disabled all access control — the incident behind ADR-004-Read-ACL-Authorization-Model.

The system was built clean and re-seeded from source. The data was not migrated. Everything is reproducible from upstream, and the old database carried Mongoose legacy, 3,382 synthetic project rows, and contentExtracted: true flags with no chunks behind them.

Phases

Phase	State
0 — Delete unreachable code	✅ 99889e6 — −1,536 lines, 5 latent bugs fixed
1 — Infrastructure	✅ demi-identity-dev, demi-cosmos-dev (10 containers), private endpoint + policy DNS
2 — Data access + authorization	✅ Client, repositories, controllers
2b/2c — Delete semantics, object key	✅ Hard delete + index removal; downloads verified end to end
3 — Merge engine	✅ src/merge/project.js + 41 tests on the real 382-record Track dataset
3b — Blob storage	⏸ code + Bicep written, not deployed, nothing copied
4 — Seed	✅ run live — 393 projects · 60,578 documents · 281 boundaries
5 — Cut over	✅ 2026-07-30
6 — Typesense	✅ then deleted entirely 2026-07-31, replaced by Azure AI Search
F — Cosmos full-text search	❌ abandoned 2026-07-31 — see ADR-005-Cosmos-Full-Text-Search-Ruled-Out
F2 — Azure AI Search	✅ live 2026-07-31
7 — Change feed	⬜ not built. Indexers pull every 5 minutes instead
8 — Decommission Mongo	✅ complete 2026-08-04

Cutover, 2026-07-30

Verified	Result
projects / documents / boundaries in Cosmos	393 / 60,578 / 281 — exactly the dry-run figures
PUT /documents/:id/published	401 — the NoSQL-only route was live
anonymous project list	382 of 393; the 11 hidden were Eagle-only projects whose upstream ACL excluded public
isPublished vs read[] drift	0
forged x-demi-roles: sysadmin header	identical results — could not promote
document download	HTTP 200, application/pdf, 84,031 bytes

Project membership was settled on 2026-07-30: it comes from Keycloak role names, scoped with a project: prefix — project:207 scopes the caller to project 207. The prefix is required, because given a bare ajax nothing distinguishes "scoped to the Ajax project" from a role type like staff. See Identity-Model.

The seed had to run over an App Service SSH tunnel rather than through Kudu. Why, and what Kudu is still good for: Environment-Reality-and-Operational-Gotchas § Reaching the data plane. The recipe is in the repo README.

Bugs found during cutover

• COSMOS_DATABASE collision. Both data layers read it, needing epic against demi. Setting it for the NoSQL client repointed the live legacy app at the empty database, and every endpoint returned [] with HTTP 200 because queryContainer swallowed the error. Fixed by introducing COSMOS_NOSQL_DATABASE.
• Bulk writes counted as sent, not landed. The first seed reported 60,578 written when 56,317 existed. bulkVerified retries and confirms; the re-run's histogram proved it exactly — {200: 56317, 201: 4261, 429: 105}.
• isPublished derived from Track's is_active, which is orthogonal to publication. 23 public projects read isPublished: false, which also 409'd any attempt to publish a document under them. It now mirrors read[].
• Boundary GeoJSON never shipped. It lives under frontend/, which the packager excludes. Failed only in Azure, never locally.
• Oryx ran yarn install inside the VNet, which has no route to registry.yarnpkg.com. Set ENABLE_ORYX_BUILD=false; the zip already ships node_modules.
• Every POST/PUT/PATCH returned HTTP 500 with an empty body. api/index.js hands Express a hand-built req; Express reparents it onto http.IncomingMessage.prototype, whose _destroy() calls this.socket.destroy(). With socket a plain {remoteAddress} object, reaching EOF threw TypeError: this.socket.destroy is not a function from a microtask — past every try/catch — killing the Node worker, which the Functions host silently respawned. GET hid it completely. Fixed with autoDestroy: false plus a real EventEmitter socket stub, with a regression test in test/routes/functions-adapter.test.js.

Phase 8 — decommissioning the MongoDB-API account

Code deployed and verified live 2026-08-01. App settings and the private endpoint came off 2026-08-04, then the account itself.

The clean week was ended early on purpose

A soak window measures latent regressions under real traffic. DEMI has none — dev only, no users, active development. What replaced it was two measurements.

The account was already idle: exactly one non-zero hour of TotalRequests in the 48 hours to 2026-08-04 (62 requests), and zero on the day itself, before and after the change. And waiting was not buying recoverability — backup was Periodic with 8-hour retention, so after deletion there is a support-ticket restore for eight hours and then nothing. The window protected reachability, not the data, and reachability is worth little when nothing is reaching for it.

Generalising: a soak period is an instrument, and an instrument with no signal on its input measures nothing. Check whether traffic exists before spending days waiting on it.

Live evidence, five probes

Probe	Result
GET /db/stats	driver: azure-cosmos-nosql, database: demi, 393 / 60,578 / 281, ~0.7 s
ACL gate	anonymous 0 / privileged 1 on a throwaway non-public project; point read anonymous 404 / privileged 200. Probe deleted, removedFromSearch: 1, index hits 0 immediately
Fault fallback	SEARCH_ENDPOINT at a bad host + nonsense term → 10 on Document and Project (the Cosmos fallback fired; healthy was 0). DocumentChunk 0 both. That fallback was later removed — see Search-and-Retrieval
Patch-not-replace	wildfire sync then NRPTI sync; all 393 Track projects read individually. sources.track byte-identical 393/393, sources.wildfire 393/393, sources.eagle intact, embedded nrptiRecords[] 0
Removed routes	GET /admin/logs, GET /wildfires → 404 anonymous and privileged; X-Request-ID still present

Three of the five probes as originally written could not fail. That lesson is recorded in Search-and-Retrieval under Measuring, which is where it belongs.

A side effect worth knowing: the NRPTI sync grew projects 393 → 2,248 and records 0 → 48,086. That run may have been incomplete — it outlived its 504 and was still running when the app-setting restart landed on it, and finished-versus-killed is not distinguishable from outside.

The blocker was boot, not routes

Deleting src/models/* broke boot:

src/app.js  ->  src/utils/logger.js:6  ->  require('../models/log')  ->  src/db/cosmos.js

src/utils/logger.js is loaded by src/app.js, src/server.js, src/middleware/request-id.js and src/middleware/http-logger.js — core boot path, none of it legacy. It was done first, by dropping the Cosmos log transport rather than porting it, since App Service already ships stdout to Log Analytics. GET /admin/logs went with it.

Six more files read through the Mongo layer while being required unconditionally by routes/api.js. All were ported to src/repositories/* and access-sql.js.

Three things the port had to change rather than carry, each a live defect in the Mongo original:

1. sync-nrpti embedded every record object into its project, twice — nrptiRecords and sources.nrpti.records, each with the raw upstream payload. About 250 records exceeded the 2 MB Cosmos item cap; Mongo's 16 MB limit had hidden it. Now a bounded aggregate via patchNrptiStats.
2. Both syncs wrote whole items back (Project.upsert(proj)), silently discarding whatever another sync wrote in between. Both now patch a single path.
3. sync-nrpti wrote project and nrptiSchemaName. The container partitions on /projectId, and nrptiSchemaName was only ever a Typesense index field, so every Cosmos query filtering on it matched nothing. Now projectId and dataset.

Also deleted: src/scripts/{seed-and-merge,sync_from_openshift,nightly-sync,backfill-read-acl}.js and the /db/seed, /sync, /admin/sync, /admin/seed-track routes that drove them. seed-nosql.js reproduces all of it and runs inside the network, past any request timeout. readFilter tier 3 went with helpers/access.js; readClause in access-sql.js never had that tier, because every seeder writes read[] explicitly.

Dropping the driver

mongodb came out of package.json on 2026-08-04. It had stayed because src/extract.js was its only user and that file is deferred rather than dead, behind a main() guard that threw when no Mongo URI was configured — so a post-teardown run errored instead of silently connecting to localhost and reporting zero documents.

That guard went too, because what it guarded was gone. Once the account was unreachable the Mongo-driven half of extract.js could not run in either direction, so keeping it meant keeping a script whose only possible outcome was that error. The file was reduced to what the deferral was actually protecting — extractWithDocling and splitAndExtract, the docling client and the 10-page PDF batching. The query loop, markDocument, the yarn extract* scripts, and replaceChunks (a deleteMany→insertMany that left a window with zero chunks for a live document) were deleted with it, along with the dead mongoUri/cosmosDbUri builders in src/config.js that nothing read and that defaulted to localhost:27017.

splitAndExtract got its first test in the process, which caught a real bug: getPageCount() sat outside the try that was supposed to make an unparseable PDF fall back to a whole-file send, so a PDF that loads but has a broken page tree threw instead of degrading.

Teardown

The Bicep edits landed 2026-08-01. The app settings came off and demi-mongo-pe plus its NIC went earlier on 08-04, and the account demi-mongo-dev-pcbd7cygyic52 was deleted last, after TotalRequests was re-confirmed at 0 across the preceding 24 hours.

Its configuration was captured before deletion — kind: MongoDB, EnableServerless, Canada Central, Periodic backup on a 240-minute interval with 8-hour retention, Geo redundancy, publicNetworkAccess: Disabled — so the shape is reproducible even though the data is not. Three databases went with it (demi-dev, test, epic), taking an orphaned syncState container that had already been removed from the template but still existed in the account.

Verified immediately after: /projects and /documents returned 200, /search?dataset=DocumentChunk still reported count: 29392, az cosmosdb list -g c4b0a8-dev-rg returned only demi-cosmos-dev, and no orphan private endpoint or NIC remained.

Removing demi-mongo-pe cut about 10 CAD/month. The resource group held three private endpoints sharing one ~29 CAD/month Virtual Network line, so one endpoint was roughly a third of it, not the whole line. Cost facts: Azure-Environments.

Secret rotation

/home/site/wwwroot/.env shipped in every deploy from 2026-07-24 until 639269b, mode rwxrwxrwx. scripts/package-api.py had no .env exclusion; the CI workflows did, but CI is dead on the missing AZURE_CLIENT_ID, so the script was the only live path. It was never committed — a packaging leak, not a repo one — and no entrypoint loads dotenv, so the file was inert on disk.

DOCLING_API_KEY was DEMI's only admin credential. src/helpers/auth.js had it in validKeys, so the secret DEMI sends outbound to docling was simultaneously an inbound sysadmin credential — a logged request header or a compromised extraction host was full admin. It was split out to ADMIN_API_KEY and the old key now 401s.

MINIO_SECRET_KEY signs the presigned download URLs the extraction host fetches document bytes with, so its rotation was held until extraction finished. MONGODB_PASSWORD went with the account.

Phase 3b, document storage on Azure Blob, was written and validated but never deployed. Its current state is in Architecture.