ADR-032: Paperless-First Invoice Creation via Stateless Extraction and Create-on-Confirm Commit

Status

Accepted

Context

Story #1679 ("Paperless-first invoice creation with LLM auto-itemize") introduces a new invoice-creation entry point: the user picks a Paperless-ngx document first, the app's LLM auto-extracts the line items, document metadata, and best-matching vendor, and the user reviews everything before the invoice is created.

The existing auto-itemize feature (EPIC-16, ADR-031, Story #1547) is keyed on an existing invoice:

• POST /api/invoices/:invoiceId/auto-itemize loads the invoice up front (invoiceAutoItemizeService.ts ~line 110), seeds extraction hints from invoice.vendorId and invoice.amount, and verifies the paperlessDocumentId is already linked to that invoice via document_links.
• On commit (dryRun: false), it creates budget lines on that invoice and optionally patches it.

The new flow is extract-first, create-on-confirm: there is no invoice (and no document link) when the user picks a document. Per the product-owner's acceptance criteria (AC23–AC26), the invoice must be created only after the human confirms, and abandoning the review must leave no orphan invoice, document link, or budget line.

Three sub-problems must be solved without breaking the existing invoice-keyed flow:

1. Stateless extraction — extract from a document with no invoice context, and let the LLM choose the best-matching app vendor from a server-provided list.
2. Atomic create-on-confirm — create the invoice, link the document, and itemize the reviewed lines in a single transaction with all-or-nothing semantics.
3. Correspondent listing — a new Paperless proxy endpoint to populate the picker's vendor-filter dropdown.

Alternatives considered

• (A) Extend the existing /:invoiceId/auto-itemize route to accept a null invoiceId. Rejected: the route's params schema requires invoiceId, the service body assumes a loaded invoice for hints and the document-link check, and overloading dry-run vs. commit semantics for two fundamentally different lifecycles (patch-existing vs. create-new) would make the service a tangle of branches. Backward-compatibility risk is high.
• (B) Persist a draft invoice immediately, then finalize on confirm. Rejected by the product-owner (AC23/AC25): a draft would be an orphan if the user abandons review, and the draft would pollute the invoices list and budget aggregations.
• (C) Two dedicated stateless endpoints — preview (extraction) and commit (atomic create) — that share extraction internals with the existing flow. Chosen. Clean separation of lifecycles, no change to the existing endpoint's contract, and maximal reuse of the LLM extraction + line-persistence internals.

Decision

Add two new stateless endpoints plus one Paperless proxy endpoint, and refactor invoiceAutoItemizeService.ts to extract its reusable internals into shared helpers.

1. Stateless extraction — POST /api/invoices/auto-itemize/preview

• No invoiceId. Request: { paperlessDocumentId, locale? }.
• Server fetches the document OCR text (existing paperlessService.getDocument), runs the LLM provider (existing getProvider), maps category names to IDs (existing mapCategoryNameToId), and computes the due-date fallback (existing computeDueDateFallback) — identical to the existing dry-run path.
• Vendor matching: the server loads all app vendors as { id, name }, injects the list into the LLM prompt, and instructs the LLM to return chosenVendorName (a verbatim copy of one provided name, or null). The server resolves chosenVendorName back to a vendor id by exact case-insensitive name match against the provided list. If the LLM returns null or a name that does not exactly match a provided vendor, suggestedVendorId is null (AC22). Matching is performed server-side against the canonical list — the LLM never invents an id. Amendment (#2148): the LLM additionally returns the raw issuer name as printed (vendorName), independent of the list; the preview surfaces it as extractedVendorName so the review screen can prefill inline vendor creation when no app vendor matches. It is never resolved to an id, so this decision is unchanged.
• No warnings array (there is no invoice total to compare against yet). The review UI computes the extracted-line total locally.
• Response: { lines, suggestedVendorId, extractedInvoiceNumber?, extractedInvoiceDate?, extractedDueDate?, extractedNotes?, extractedVendorName? }.

2. Atomic create-on-confirm — POST /api/invoices/auto-itemize/commit

A dedicated endpoint (not an extension of createInvoice, which is vendor-scoped and knows nothing about document links or itemization). In a single SQLite transaction:

1. Validate vendorId exists (vendor is required — AC21).
2. Create the invoice (createInvoice internals: vendor, amount, date, dueDate, invoiceNumber, notes, status).
3. Create the document_links row linking the selected Paperless document to the new invoice (entity_type='invoice', entity_id=<new invoice id>).
4. Itemize the reviewed lines into invoice_budget_lines / work_item_budgets using the same line-persistence helper extracted from the existing commit path (mode: 'append' semantics — there are never pre-existing lines on a brand-new invoice).
5. Validate Σ itemized ≤ invoice amount (existing ItemizedSumExceedsInvoiceError).

If any step throws, db.transaction() rolls back the entire unit: no invoice, no document link, no budget lines persist (AC24/AC26). The endpoint is not idempotent — each successful call creates a new invoice; the client must guard against double-submit (the review screen disables confirm while the request is in flight). Because there is no invoice before confirm, abandoning the review is a pure client-side discard — nothing to clean up (AC25).

Response (201 Created): { invoice, budgetLines, remainingAmount } — the created invoice plus the itemization result, so the client can navigate straight to the invoice without a follow-up fetch.

3. Correspondents — GET /api/paperless/correspondents

Mirrors GET /api/paperless/tags exactly: returns { correspondents: { id, name }[] }, requires auth, returns 503 PAPERLESS_NOT_CONFIGURED when Paperless is not configured, 502 on upstream failure. Fetched on demand (paginated page_size=1000 like tags). Not cached initially — correspondent lists are small (<~100) and the picker fetches once on open; if profiling later shows pressure, a short-TTL cache mirroring the filter-tag cache can be added (deferred, noted for future review).

4. Reuse strategy (invoiceAutoItemizeService.ts)

The existing invoice-keyed flow stays unchanged in behavior. The following internals are extracted into reusable, non-invoice-coupled helpers (same file or a sibling module) and shared by both flows:

• Line persistence — the per-line loop that resolves assignment mode, creates work_item_budgets (origin 'auto') or assigns existing budget lines, inserts invoice_budget_lines junction rows, and accumulates totalItemized. Refactored to take (db, invoiceId, vendorId, userId, lines) instead of reading them off a loaded invoice object. The existing commit path calls it with the loaded invoice's id/vendorId; the new commit path calls it with the freshly-created invoice's id/vendorId.
• Extraction core — fetch document, truncate OCR to MAX_OCR_CHARS, run provider, map categories, compute due-date fallback. Shared by both dry-run paths. The only difference is hint seeding (existing flow seeds from the loaded invoice; the new flow seeds from the document + provided locale and additionally injects the vendor list).
• Vendor-list prompt augmentation — new code: gather { id, name }[], render into the user prompt, parse chosenVendorName, resolve to id. Used only by the new preview endpoint.

Genuinely new code: the two route handlers, the two thin service entry points (previewAutoItemize, commitAutoItemizeCreate), the vendor-list prompt augmentation + parsing, and the correspondents proxy. The bulk of persistence and extraction logic is shared.

Consequences

Easier:

• The existing /:invoiceId/auto-itemize contract is untouched — zero regression risk to the EPIC-16 flow and its tests.
• Atomicity is trivially correct: a single db.transaction() wraps create + link + itemize, so partial failure cannot leave orphans (AC26), and there is no draft-cleanup machinery to maintain (AC25).
• Vendor resolution is deterministic and safe — the LLM can only echo a name from the server-provided list, and the id is resolved server-side, so the LLM cannot fabricate a vendor id.
• The correspondents endpoint follows the established /tags pattern, so it is consistent and quick to implement/test.

More difficult / trade-offs:

• Two extraction entry points now exist (/:invoiceId/auto-itemize dry-run and /auto-itemize/preview). They share an extraction core but diverge in hint seeding and the vendor-list step; this divergence must be kept in sync if the prompt evolves. Mitigated by extracting the shared core into one helper.
• The commit endpoint duplicates the "create invoice" surface that createInvoice already owns. We deliberately call createInvoice's internals rather than re-implementing validation, but the transaction boundary lives in the new endpoint. If invoice-creation rules change, both call sites must be considered (they share the underlying helper, so the risk is contained).
• Exact-match vendor resolution may under-match (e.g., "ACME GmbH" vs. "ACME"); the user override (AC21) is the safety net. Fuzzy matching is intentionally not done server-side to keep resolution predictable; the LLM is the fuzzy layer and is constrained to echo a provided name.
• Correspondents are not cached; a picker open triggers one upstream call. Acceptable at <5 users; revisit if it becomes a hot path.

Related

• ADR-015 (Paperless-ngx Integration) — proxy pattern, polymorphic document_links.
• ADR-018 (Invoice-Budget-Line Junction) — invoice_budget_lines model, ON DELETE CASCADE.
• ADR-031 (Outbound LLM Provider Integration) — OpenAI-compatible gateway, autoItemizeEnabled.

Deviation Log

Date	Section	Deviation	Resolution
2026-10-01	§4 Reuse strategy, "Line persistence" bullet	Issue #2149. "Assigns existing budget lines" was never defined. Since #1589 the shared persistLines helper's assign-existing branch also overwrote the existing budget line's fields from the extracted row, in both commit paths.	User decision 2026-10-01: assigning an existing budget line is junction-only — persistLines inserts the invoice_budget_lines row (VAT-effective itemizedAmount) and never modifies the referenced line. Orchestrator decision 2026-10-01, pending user confirmation: a line already linked to the invoice gets the amount added to its existing junction row instead of a second row or a rejection. In replace mode, a junction that existed before the save is reset to the first linking row's amount (later rows in the same call add), since replace deletes only auto-created lines. Only the invoice-keyed commit has a replace mode; the create-on-confirm commit always starts from a fresh invoice. Code fixed in #2149; full semantics on the API Contract page under POST /api/invoices/:invoiceId/auto-itemize.
2026-10-01	§4 Reuse strategy, "Line persistence" bullet	Issue #2149 / PR #2151. The previous row's pending orchestrator decision (an already-linked line accumulates into its existing junction row; replace resets then accumulates) was not confirmed.	User decision 2026-10-01: a budget line is linked at most once — to at most one invoice, by at most one row per request. persistLines validates before any write and rejects the whole request with 409 BUDGET_LINE_ALREADY_LINKED; in replace mode a surviving link to this invoice is overwritten, never summed. Accumulation removed. Full semantics on the API Contract page.