ADR-002: Async Processing with MongoDB-Based Job Queue

Status

Superseded. Never implemented, and its premise never held in this environment. The pipeline that replaced it is Extraction-Pipeline.

Context

File processing — text extraction, OCR, rule-based classification — took 1 to 120+ seconds per document against a 20-second upload target, so it had to run asynchronously.

An earlier proposal used pg-boss, a PostgreSQL-backed queue. That was rejected because eagle-api was MongoDB-only and adding PostgreSQL for one feature was expensive overhead. eagle-api already ran MongoDB change streams for the Typesense sync.

Decision

Processing used a MongoDB document-state plus change-stream worker pattern, with no additional queue infrastructure.

• Documents carried processingStatus (pending, processing, processed, failed, dead_letter), timestamps, an error string, a retry count and the id of the claiming worker.
• A worker watched for pending documents and claimed each one with an atomic findOneAndUpdate, which prevents two workers taking the same job.
• Failures moved to a dead-letter state after three retries.
• Monitoring was a MongoDB count query; no dashboard or broker was needed.

Upload volume was 50-200 documents per day at peak, well below the point where a distributed queue earns its operating cost.

Consequences

The pattern depends on MongoDB change streams, which Cosmos DB for MongoDB API does not support. The watcher exits when db.command({hello:1}) reports no setName, and the Mongo API never reports one, so the design was inert from the day it was written. The Mongo API account was deleted on 2026-08-04 and the mongodb driver was removed from the repository.

Extraction instead runs off the platform and posts markdown to POST /api/documents/:id/chunks, which is idempotent, and Azure AI Search indexers pull from Cosmos on a PT5M schedule. There is no queue.