ADR 031 Outbound LLM Provider Integration - steilerDev/cornerstone GitHub Wiki

ADR-031: Outbound LLM Provider Integration via OpenAI-Compatible Gateway

Status

Accepted

Context

Cornerstone is adding auto-itemization of construction invoices via OCR text analysis (Story #1546, #1547). The system extracts line items (description, quantity, unit, price, VAT) from raw OCR text without vendor-locking to any single LLM provider.

Multiple providers expose the OpenAI chat-completions API schema:

  • OpenAI (GPT-4 Turbo, GPT-4o) — ~$0.01–$0.03 per invoice
  • Anthropic (Claude 3 Opus) — ~$0.015 per invoice via Claude API or OpenRouter
  • Google Gemini (Flash 1.5) — ~$0.001 per invoice, extremely cost-effective
  • Open-source / self-hosted (Ollama, OpenRouter, LiteLLM proxy) — variable costs, no external API calls
  • Azure OpenAI — enterprise compliance, OpenAI-compatible endpoint

Choosing a single vendor (e.g., "we only support OpenAI") would force users into that ecosystem and lock out cost-conscious deployments and air-gapped installations. The solution must support any OpenAI-compatible gateway.

Decision

1. Single OpenAI-Compatible HTTP Client

Implement BudgetExtractionService with a single createOpenAICompatibleProvider(config) factory that:

  • POSTs to {LLM_BASE_URL}/chat/completions with Authorization: Bearer {LLM_API_KEY}
  • Sends the configured {LLM_MODEL} and response_format: { type: 'json_object' }
  • Enforces a request timeout via LLM_REQUEST_TIMEOUT_MS (default 30 seconds)
  • Returns fully-typed ExtractedLine[] objects

2. Environment-Variable Configuration

Three optional env vars enable the feature:

  • LLM_BASE_URL — base URL of the LLM API (e.g., https://api.openai.com/v1)
  • LLM_API_KEY — bearer token for authentication
  • LLM_MODEL — model identifier (e.g., gpt-4-turbo, claude-3-opus-20240229)
  • LLM_REQUEST_TIMEOUT_MS — request timeout in ms (default: 30000)

Feature is disabled by default. GET /api/config returns autoItemizeEnabled: false unless all three core env vars are set. The client uses this flag to conditionally show auto-itemize UI buttons.

API key is never exposed. LLM_API_KEY is not returned to the client; only the boolean autoItemizeEnabled flag is. Error responses do not echo the request body (which could contain the prompt with vendor names, amounts, and invoice context).

3. Data Flowing Off-Host

What is sent to the LLM provider:

  • OCR text (raw text extracted from the PDF, 1–5 KB typical)
  • Invoice metadata hints (vendor name, total amount, date, locale)

What is NOT sent:

  • Binary PDF files or images (OCR happens on-host via Paperless)
  • User-identifying information beyond what appears on the invoice itself (no account IDs, user emails, etc.)
  • API keys or credentials

The system prompt scopes extraction to "German construction-invoice line items" and emphasizes JSON-only output, preventing prompt injection and PII leakage via LLM responses.

4. Failure Semantics

Three distinct LLM-related error codes:

Error Code HTTP Scenario
LLM_UNREACHABLE 502 Network timeout, DNS failure, connection refused
LLM_INVALID_RESPONSE 502 LLM returned non-JSON or invalid schema
LLM_UPSTREAM_ERROR 502 LLM returned non-200 (e.g., 429 rate-limit, 500)
LLM_NOT_CONFIGURED 503 Feature is disabled (env vars not set)

Error messages are sanitized — no API response bodies are echoed (they may contain the LLM's interpretation of the prompt). Only the HTTP status code is included for diagnosis.

5. Security & Compliance

  • API Key Storage: Kept in environment variables, never logged (config logging excludes llmApiKey). Shared with the LLM provider only via HTTP Authorization: Bearer header, never in URL or query params.
  • Request Timeout: Prevents hanging requests due to LLM provider outages. Default 30 seconds is appropriate for turn-based chat (GPT-4, Claude) and slightly generous for fast models (Gemini Flash). Users can tune via LLM_REQUEST_TIMEOUT_MS.
  • Response Validation: Strict structural validation rejects any response not matching { lines: ExtractedLine[] } schema. Confidence scores are 0–1; all monetary amounts are validated as finite numbers.
  • Prompt Injection: System prompt is immutable, user prompt is template-based (OCR text is concatenated, not interpolated into JSON or shell commands).
  • Failure Safety: If LLM is unreachable or returns invalid data, the extraction returns LLM_UNREACHABLE / LLM_INVALID_RESPONSE and stops. The invoice is not partially itemized. The user is prompted to retry or manually itemize.

6. Alternatives Considered

  1. Provider-Specific Clients (OpenAI SDK, Anthropic SDK, etc.)

    • Rejected: Requires shipping N client libraries, complicating dependency management. Locking users into one vendor contradicts the goal of flexibility. Future providers require code changes.

  2. Separate Python Service with Native Libraries

    • Rejected: Adds deployment complexity (Flask/FastAPI service, Docker Compose). Doubles the attack surface. No benefit over HTTP client in Node.js which is already built-in (fetch). Python-specific dependencies add build-time issues.

  3. Client-Side LLM Calls via CORS Proxy

    • Rejected: Exposes API key to the browser (major security flaw). CORS proxies are brittle. No server-side audit trail of LLM usage. Violates principle of least privilege.

  4. Custom NLP/Regex Extraction Engine

    • Rejected: Building a German invoice parser is a multi-month project. LLMs handle variance in formatting, currency notation, VAT, and unit conventions out-of-the-box. Maintenance burden is orders of magnitude higher.

Consequences

Positive

  • Zero Vendor Lock-In: Users pick their provider based on cost, compliance, or performance.
  • Default-Off Security Posture: LLM integration is opt-in via environment variables. Users who don't set them get a working system without external dependencies.
  • Cost Control: Users on tight budgets can use Gemini Flash (~$0.001/invoice) or self-hosted Ollama (free, local). Enterprise customers can use Azure OpenAI for compliance.
  • Future-Proof: New OpenAI-compatible providers (Deepseek, Mistral, etc.) work without code changes.
  • Reduced Blast Radius: If the LLM provider is compromised or unreachable, the system degrades gracefully (extraction returns a 502/503, invoices remain itemizable manually).

Negative

  • Model-Specific Behavior: Some models are better at German invoices than others. Gemini Flash may miss small detail lines; Claude 3 Opus is more thorough but slower. Users must experiment.
  • Cost Tracking: No built-in metering or cost-tracking dashboard. Users must monitor their LLM provider's usage dashboard separately.
  • Rate Limiting: If the user's LLM provider rate-limits, extraction fails. Retry is on the user.
  • Latency Variation: Extraction latency depends on the chosen provider and model. GPT-4 Turbo: ~10–30s per invoice. Gemini Flash: ~2–5s. Expected and documented.

Related Decisions

  • ADR-015 (Paperless-ngx Integration): OCR happens on-host via Paperless. The LLM extracts structure from Paperless' OCR output.
  • ADR-009 (Error Handling): Typed error codes and HTTP status codes follow the established error-handling convention.
  • ADR-030 (Orphan Budget Lines): Auto-itemized lines land as orphans (no parent work item) with origin: 'auto', distinct from manual itemizations.

Security Review Summary

  • ✅ Bearer token in Authorization header, not in URL or body
  • ✅ API key never logged or echoed in error responses
  • ✅ Response body not echoed in error messages (prevents PII leakage via LLM inference)
  • ✅ Strict schema validation prevents injection via malformed JSON
  • ✅ Request timeout prevents DoS via hanging connections
  • ✅ Default-off (requires explicit env var configuration)
  • ✅ No browser-side API key exposure
  • ✅ No client-side configuration of LLM endpoint (server-only)