ADR-003: Rule-Based Classification Over LLM Inference

Status

Accepted 2026-05-26. Not implemented in code (verified 2026-08-26). Replaces the earlier "Reuse EPIC.search Embedder" ADR.

DEMI has no classifier. There is no configs/ directory, and nothing in src/ reads document-types.yaml, maps a document type to an ORCS code, or derives a retention class. orcsClassification in src/seed/transform.js is copied verbatim from Eagle's document metadata. The classify() function in src/text-quality.js scores text quality for chunking and is unrelated to this decision. The approach below is the one to build when AICT needs it.

Context

AICT needs suggested metadata for uploaded files: document type, topic tags, ORCS classification and retention class. The original DEMI brief assumed an LLM would produce them.

LLM classification is cheap. GPT-4.1-nano at May 2026 prices costs about $0.0001 per document — roughly $26 to classify the 65,000 existing Priority 1A documents, and about $7 a year at 50 uploads a day. Cost is therefore not the reason to avoid it.

The reason is the Metadata Verification Hold: a human reviews every classification before it takes effect. Suggestions only have to be helpful starting points, so 80% accuracy from rules is enough when a person confirms 100% of them.

Decision

Version 1 classifies by rules only. No LLM inference for metadata suggestions.

Methods

Metadata	Method	Expected accuracy
Document type	Keyword scoring against a controlled vocabulary	70-80%
Topic tags	Term frequency matched against tag definitions	60-70%
ORCS code	Document type to ORCS mapping table	90%, given the right type
Retention class	ORCS code to retention schedule lookup	100%, given the right ORCS code
Title	PDF Title metadata, first heading, or filename	50-70%
Creator	PDF Author metadata or email From header	80%
Date	PDF CreationDate, email Date header, or filename pattern	85%

Controlled vocabulary

# configs/document-types.yaml
- name: "Environmental Assessment Certificate"
  code: "EAC"
  indicators:
    high: ["environmental assessment certificate", "eac amendment"]
    medium: ["certificate holder", "certified project"]
    low: ["assessment", "certificate"]
  orcs_code: "15400-30"

Each high match scores 10, each medium 3, each low 1. The three highest-scoring entries with a non-zero score are offered to the reviewer, each with its score and the keyword that earned it.

Deduplication

Not part of this decision and not built. The original plan used Typesense's similar_document_id, which went with Typesense. Retrieval is lexical BM25 with no vector component, so there is no embedding to compare — see Search-and-Retrieval.

Consequences

Positive:

• No per-document cost, no API keys, no rate limits, and no external service to be down.
• Deterministic and explainable: a suggestion names the keyword that produced it.
• Fast enough to be free at this volume — string matching, under 100 ms per document.
• Tunable by editing YAML rather than code.

Negative:

• Lower accuracy than an LLM on ambiguous documents, roughly 70% against 90%.
• The keyword vocabularies have to be curated and maintained.
• Document types absent from the vocabulary get no suggestion at all.
• No use of context: rules cannot infer a subject that the text never names directly.

When to revisit

Move to LLM classification only if rule accuracy stays below 60% in use, reviewers report the suggestions are unhelpful, or volume grows enough to justify the API cost. The upgrade is an opt-in per project, using a small model only where rule confidence falls below a threshold; AI is already in the stack as a search-result summarizer, so this adds no first external dependency.

An embedding-similarity step is not an option: it would mean building the vector pipeline the project has deliberately not built.