ADR-015: Paperless-ngx Integration Architecture

Status

Accepted

Context

Cornerstone needs to integrate with Paperless-ngx for document management. Users want to link documents stored in Paperless-ngx to work items, household items, and vendor invoices. The application must browse, search, and display document metadata and thumbnails without storing documents locally.

Several architectural decisions need to be made:

1. Communication pattern: Should the browser communicate directly with Paperless-ngx, or should Cornerstone's server proxy the requests?
2. Document linking schema: Should we use a single polymorphic document_links table with an entity_type discriminator, or separate junction tables per entity type (e.g., work_item_documents, household_item_documents, invoice_documents)?
3. Caching strategy: Paperless-ngx document metadata and thumbnails are fetched on demand -- should we cache them, and if so, where?
4. Configuration: How should the Paperless-ngx connection be configured?

Decision

1. Server-Side Proxy Pattern

All Paperless-ngx API requests are proxied through Cornerstone's Fastify server. The browser never communicates directly with Paperless-ngx.

Alternatives considered:

• Direct client-to-Paperless-ngx: The browser would need the Paperless-ngx API token, exposing it in client-side JavaScript. This is a security risk. It also requires CORS configuration on Paperless-ngx and exposes the Paperless-ngx URL to the browser, which may not be routable from the client network.
• Server proxy (chosen): The Fastify server holds the Paperless-ngx API token securely. The client only talks to Cornerstone's /api/paperless/* endpoints. This centralizes authentication, allows rate limiting, and simplifies CORS.

Rationale: Security is paramount. The API token grants full access to all documents in Paperless-ngx. Keeping it server-side follows the principle of least privilege. The proxy also allows Cornerstone to transform Paperless-ngx responses into a stable contract that won't break if the upstream API changes.

2. Single Polymorphic Document Links Table

We use a single document_links table with an entity_type discriminator column rather than separate junction tables per entity type.

Alternatives considered:

• Separate junction tables (work_item_documents, household_item_documents, invoice_documents): Each table has a clear foreign key to its parent entity. This provides strong referential integrity via SQLite foreign keys. However, it requires three nearly identical tables with duplicate DDL, duplicate Drizzle schema definitions, and largely duplicate CRUD routes.
• Single polymorphic table (chosen): One document_links table with columns entity_type (enum: work_item, household_item, invoice) and entity_id (TEXT). This centralizes all document linking logic, reduces schema duplication, and makes it easy to add new entity types in the future.

Trade-off: The polymorphic approach cannot use SQLite foreign key constraints on entity_id because it references different tables depending on entity_type. Referential integrity for the entity side is enforced at the application layer (check that the referenced entity exists on insert, and cascade-delete links when an entity is deleted). The paperless_document_id column has no foreign key either (it references an external system). This is acceptable because:

• Document links are lightweight references, not critical business data
• Orphaned links (pointing to deleted entities) are harmless and can be cleaned up lazily
• The application layer already validates entity existence on link creation
• A composite unique constraint on (entity_type, entity_id, paperless_document_id) prevents duplicate links

3. No Server-Side Cache (Phase 1)

For the initial implementation, there is no caching layer. Each proxy request to Paperless-ngx results in a live upstream call.

Alternatives considered:

• In-memory LRU cache: Cache document metadata and thumbnails in a Node.js Map with TTL. This reduces latency and upstream load.
• SQLite cache table: Store fetched metadata in a local table with expiry timestamps.
• No cache (chosen for Phase 1): The simplest approach. With fewer than 5 users, the request volume to Paperless-ngx is minimal. The Paperless-ngx instance is typically on the same network (or even the same machine), so latency is low.

Future extensibility: If performance becomes a concern, an in-memory LRU cache with a configurable TTL (e.g., 5 minutes for metadata, 1 hour for thumbnails) can be added as a Fastify plugin without changing the API contract. The proxy endpoints would check the cache before calling upstream. This is noted but not built.

4. Environment Variable Configuration

Paperless-ngx connection is configured via two environment variables:

Variable	Default	Description
PAPERLESS_URL	(none)	Base URL of the Paperless-ngx instance (e.g., http://paperless:8000)
PAPERLESS_API_TOKEN	(none)	API authentication token for Paperless-ngx

The integration is enabled when both variables are set. If either is missing, all /api/paperless/* endpoints return 503 SERVICE_UNAVAILABLE with error code PAPERLESS_NOT_CONFIGURED.

Rationale: Environment variables are the established configuration pattern in Cornerstone (consistent with OIDC configuration). The API token is sensitive and should never be stored in the database or exposed to the frontend. The paperlessEnabled flag is derived at startup and made available via fastify.config.

5. Paperless-ngx API Version

Cornerstone targets Paperless-ngx API version 10. Every JSON request to Paperless-ngx, including the POST /api/documents/post_document/ upload, sends the Accept: application/json; version=10 header, derived from a single PAPERLESS_API_VERSION = 10 constant in server/src/services/paperlessService.ts. This pins Cornerstone to a known API shape and prevents breaking changes from upstream version bumps.

• Allowed versions are enforced upstream. Paperless-ngx uses DRF AcceptHeaderVersioning with DEFAULT_VERSION = "10" and ALLOWED_VERSIONS = ["9", "10"]. Any other requested version is rejected with HTTP 406 Not Acceptable. Cornerstone maps an upstream 406 to 502 PAPERLESS_ERROR with a message stating that the Paperless-ngx server does not support API version 10 (i.e. the server is too old or too new for this Cornerstone release).
• Binary passthroughs send no version. The thumbnail and preview passthroughs (fetchBinary) send no Accept header, because forcing application/json on a binary endpoint is wrong; they rely on the server's default version. Their response is raw bytes, so the API version does not affect them.
• Version-dependent field shapes Cornerstone relies on:
  • Tags expose color as a hex string (e.g. #a6cee3). The numeric colour field (1-13) was removed upstream in API v2. Cornerstone maps color directly when it matches /^#[0-9a-f]{6}$/i, otherwise null.
  • Documents expose created as a date (YYYY-MM-DD) from API v9 onward, not a datetime.
  • The full-text search hit __search_hit__.score may be a number or a numeric string; Cornerstone accepts both.
• Upgrade cadence. Paperless-ngx supports an older API version for at least one year after a newer one ships. When the upstream ALLOWED_VERSIONS drops 10, the constant has to be bumped and the field mappings above re-verified.

6. Proxy Endpoint Design

Proxy endpoints are scoped under /api/paperless/ and map to a curated subset of the Paperless-ngx API:

• GET /api/paperless/documents -- Search/browse documents
• GET /api/paperless/documents/:id -- Single document metadata
• GET /api/paperless/documents/:id/thumb -- Document thumbnail (binary passthrough)
• GET /api/paperless/documents/:id/preview -- Document preview/PDF (binary passthrough)
• GET /api/paperless/tags -- List all Paperless-ngx tags

These proxy endpoints transform Paperless-ngx responses into Cornerstone's own response shapes, decoupling the client from the upstream API format. Only the fields needed by the Cornerstone UI are included.

Consequences

Easier

• Security: The Paperless-ngx token never reaches the browser
• Consistency: All API calls go through /api/*, maintaining a single origin for the SPA
• Schema simplicity: One document_links table instead of three junction tables
• Future entity types: Adding document links to new entities (e.g., subsidy programs, budget sources) requires only a new entity_type enum value and application-layer handlers
• Configuration: Follows established env var patterns; admins familiar with OIDC config will recognize the pattern

More Difficult

• No referential integrity on entity_id: Application layer must handle orphan cleanup when entities are deleted
• Proxy maintenance: If Paperless-ngx changes their API significantly, the proxy layer needs updating (mitigated by version pinning)
• No offline access: Without caching, Paperless-ngx must be reachable for any document-related feature to work
• Binary passthrough for thumbnails/previews: The server must buffer and forward binary responses, adding memory pressure for large PDFs (mitigated by the small user count)

Deviation Log

Date	Section	Deviation	Resolution
2026-09-28	5. Paperless-ngx API Version	The ADR pinned API version 5. Current Paperless-ngx accepts only versions 9 and 10 (ALLOWED_VERSIONS) and answers any other version with 406, so every JSON proxy call failed with 502 PAPERLESS_ERROR. Separately, tag colours had been null since the integration shipped: the code read the numeric colour field, which upstream removed in API v2 in favour of the hex color field.	Both the wiki and the code were outdated (issue #2097). The ADR now targets API version 10 via a single PAPERLESS_API_VERSION constant, sent on all JSON requests including the upload; binary passthroughs send no Accept header and rely on the server default. It documents the 406 mapping to a descriptive PAPERLESS_ERROR, the hex color mapping with validation, created as a date (v9+), and a string | number search score. The API Contract already documented tag color as hex and needed no change. The same correction is recorded in the Architecture page's Deviation Log. This section did not previously exist and was created by this entry.