How a stored PDF becomes searchable chunks. What the resulting text is like, and how good it is, is Extraction-Quality.

An extraction host posts markdown to POST /api/documents/:id/chunks, authenticated with X-Api-Key against the admin key. The server chunks it — src/chunker.js is the only chunking implementation — and copies read[] from the live document, so an extraction host can never widen a document's visibility.

Chunk ids are deterministic (<documentId>::p<page>::c<index>) and chunks.replaceForDocument reconciles, so the route is idempotent and a killed backfill just restarts.

Re-posting identical chunks writes nothing: each stored chunk carries a hash of its own content, and an upsert is skipped when the incoming chunk hashes the same. A stored chunk with no hash yet counts as changed, so the first re-post after this shipped still rewrites once.

Re-posting a failed document clears the failure. There is no guard on contentExtracted: ingestChunks replaces the chunks and patches contentExtractionError: null whatever state the document was in. A host holding its own done-list requeues by posting again — no flag reset, no admin script, no SSH tunnel.

src/scripts/purge-extraction.js --errors-only covers the one case that cannot fix itself. A document flagged extracted-with-error is invisible to the host's work list, which asks for extracted=false, and a recorded failure sets contentExtracted: true exactly like a success. The meaningful pair is (contentPageCount, contentExtractionError); the flag alone is never evidence of chunks. --error-like STR narrows the purge to failures whose message contains a substring.

Set in src/config.js, overridable by environment variable.

A chunk can therefore reach MAX_CHUNK_SIZE + OVERLAP_SIZE: the ceiling bounds a block's own text and prepended overlap sits on top of it.

Do not change these while ingest is running. Chunk ids derive from the split, so a mid-run change orphans every chunk already written instead of reconciling with it. Group any chunker change into a single re-ingest.

The corpus backfill runs off-platform on a GPU box that is not part of DEMI. It is an ordinary API client using the same public endpoints and admin key any client would:

GET  /api/documents/scan?extracted=false&pageSize=1000   build the work list
GET  /api/documents/:id/download                    presigned URL, bytes come from the object store
POST /api/documents/:id/chunks                      markdown back, JSON or application/x-ndjson

/documents/scan requires a credential; an anonymous call is a 401. It pages with the x-continuation-token header.

GET /api/documents/scan and GET /api/documents leave out documents with documentSource: 'UPDATE' — images uploaded through an Update's gallery or featured-image field — from both the extracted=false work list and any other listing. They never reach the host, so they are never extracted or chunked.

Because the download is a presigned URL, document bytes go straight from the object store and never transit Azure. No Bicep template and no app setting references the box. Its source is vendored under extraction-host/ so it does not live on one machine only; it is Python that the Node app never loads, and scripts/package-api.py excludes it from the deploy package.

Everything GPU-side — OCR, docling, page batching, tiling — happens there. The only thing crossing into Azure is one POST of markdown per document. It has to cross because Cosmos is the system of record and AI Search indexes it on a PT5M schedule.

There is no Azure-side extraction path in service. extractor/ (queue-triggered Python Azure Functions) and azure/modules/extractor.bicep exist in the repo, but the module is not referenced by azure/main.bicep and no extractor function app exists in either subscription. src/extract.js is the only in-repo docling-serve client and PDF page-batching code; it splits PDFs into DEMI_BATCH_PAGES batches, default 10.

Documents carry an extraction sidecar written by the host.

{"path": "ocr", "engine": "docling+rapidocr (cuda)", "doclingVersion": "2.116.0",
 "options": "{\"force_ocr\":false,\"batch_pages\":25,\"pdf_backend\":\"PyPdfiumDocumentBackend\",\"tiled\":false}",
 "at": "2026-08-01T05:09:15.741294+00:00"}

options comes back as a string — sanitizeExtraction flattens it with JSON.stringify and caps it at 500 characters, so nesting cannot smuggle in depth. Anything outside the five-key whitelist is dropped silently, so a typo in the sender is invisible rather than an error. An unrecognised path is recorded as unknown rather than dropped.

Three fields split any quality measurement: path (text or ocr), options.pdf_backend, and options.tiled.

GPU throughput is not the bottleneck; routing is. Running do_ocr=True on every PDF pushes digital PDFs with a perfectly good text layer through RapidOCR.

The host probes first. A pypdfium2 text-layer probe over the first PROBE_PAGES (20) pages sends digital PDFs down a CPU-only path; a PDF is treated as digital only if it clears MIN_CHARS_PER_PAGE (200) on at least MIN_TEXT_FRACTION (0.8) of probed pages. Only image formats and text-poor PDFs reach the GPU. The router is deliberately biased toward OCR: mis-routing a scan silently drops the document out of a lexical index, which is far worse than spending GPU time on a digital PDF.

msg, zip, rtf, doc, ppt, xls, eml, exe and dwg have no docling reader and are recorded as extraction errors without being downloaded.

A third outcome class exists beyond success and failure: missing(). A source 404 is neither a document defect nor a runner fault. It writes no .err, so the document stays in the work list for whenever the file appears, and it does not trip the deferral circuit breaker. A large unextracted count is that class working, not a bug.

worker.py --selfcheck is 8 assertions on decide(), the routing rule. It needs no network, GPU or docling, which is what makes it runnable anywhere. test_poolfix.py is separate and is not wired into that mode.

These are properties of the tools, not of one run. Changing any of them reintroduces a known failure.

• PDFium is not thread-safe, and docling uses pypdfium2 too. Every pdfium call — the probe and the tile render alike — belongs in the ProcessPoolExecutor, spawned not forked, because the parent holds a CUDA context. Concurrent use in the parent corrupts silently: valid PDFs return PDFium: Data format error and convert cleanly minutes later. Single-path tests cannot see it; the regression check asserts the render is submitted to the pool, because the output looks fine either way.
• The PDF backend is PyPdfiumDocumentBackend (DOCLING_PDF_BACKEND=pypdfium). DoclingParseDocumentBackend's native page decoder aborts the process on some corpus documents, status=5/TRAP at docling_parse/pdf_parser.py:757 _ensure_page_decoder. PYTHONFAULTHANDLER=1 does not cover SIGTRAP; faulthandler.register(signal.SIGTRAP) does. A native abort cannot be caught, so no outcome is recorded, the feeder requeues the same work, and the run stalls.
• An infrastructure failure must never be recorded as a data failure. A dead ProcessPoolExecutor child sets _broken and every later submit raises; catching that as a document failure marks documents extracted with zero chunks and drops them from the work list permanently. The recording is what makes it silent.
• CONVERTERS defaults to 2. Docling holds page images for the whole document, so RAM scales with page count; CONVERTERS=3 OOM-killed a 16 GiB host at 15.9 GB peak. Documents over BIG_DOC_MB (8) are serialised behind a semaphore.

The POST route hits four separate ceilings on real documents.

Raising the body limit is the wrong knob: a 60 MB body parses to a 60 MB JS string plus parser buffer on B1 Basic, and the ceiling only moves to the next document. Both doors share createChunkAccumulator, so a document chunks identically whichever it came through.

The batch bound is checked between chunks, not between markdown blocks. A 30 MB document can be six blocks of ~5 MB, each emitting well over a thousand chunks per call, so a per-block bound lets peak memory follow block size. STREAM_BATCH_CHUNKS is 200, roughly half a megabyte in flight regardless of document size.

The extractor writes a form feed (\f) between consecutive pages — never leading, never trailing — and still marks an empty page, so later pages keep their number. Both the text path (extract.py, join_pages in ocr.py) and the OCR path emit it. The extraction host mirrors this for its docling routes: worker.py calls export_to_markdown(page_no=N) per page and joins with the marker. Formats with no page map — office files, images — fall back to one whole-document export with no marker.

src/chunker.js splits on the marker before buffering. pageNumber is the real 1-based PDF page a chunk's text starts on. A page with at least MIN_CHUNK_SIZE characters closes its own block; a shorter page (a cover, a signature page, a sub-floor form) joins the block before it, so a divider or plate page never shifts the next page's citation. It merges forward, into the next page's text, only at document start — when there is no block before it yet — or when joining would push that block past MAX_CHUNK_SIZE. Overlap stays on the page the chunk it came from started on, so a chunk's quoted page is never one page ahead of the wording it carries.

Ingest stamps pageNumbered: true and pageCount on the document, and pageNumbered: true on each chunk, only when the source markdown carried at least one marker. Without a marker neither field is written, and a chunk's pageNumber is a sequence number, not a page. A single-page document never carries one, since there is no second page to break before, so it stays null the same as a document extracted before markers existed. A multi-page document re-extracted after markers went into production gets the flag and real page numbers; one not yet re-extracted keeps the sequence number until it is. contentPageCount still counts chunks; pageCount counts pages. The test corpus re-extraction to backfill markers on already-extracted documents started 2026-09-16.

The chunks search index carries pageNumbered (Edm.Boolean, filterable, retrievable) and the demi-chunks-ds data source selects it; deploy order is in azure/search/README.md. /api/search exposes pageNumbered on each row, and eagle-public labels a result "Page N" with a #page=N link when it is true, "Passage N" otherwise.

Two gaps remain. A grouped search row carries only its lead chunk's page, so per-passage page labels need a passages array — not built yet. The project summary prompt and GET /api/documents/{id}/chunks still return pageNumber without the flag, so a citation drawn from either is only verifiable once its source document has been re-extracted.

Re-ingest does not need the GPU. Extracted markdown is retained on the host under sent/, and the ingest path moves tens of thousands of chunks per hour. Verify retention across the full corpus before planning on it.

• Extraction-Quality — corpus coverage, word-joining, tiling, and what the quality metrics miss
• Search-and-Retrieval — what the chunks are searched with, and the retrieval scorecard
• Architecture — chunk container, ACL inheritance, object storage