Architecture - The-40-Thieves/obsidian-tc GitHub Wiki
Full design record: ARCHITECTURE.md. Tool surface: docs/G2.1-tools.md.
obsidian-tc is a polyglot monorepo. The server is a single Bun/Node process; the companion plugin runs inside Obsidian; the native module is Rust linked over napi-rs.
| Package | Language | Owns |
|---|---|---|
packages/server |
TypeScript (Bun + Hono; runs under Node >= 24 too) | Transport, auth, ACL, policy, router, tool impls, bridges, observability |
packages/plugin |
TypeScript | Companion Obsidian plugin (/obsidian-tc/v1/* routes) |
packages/native |
Rust (napi-rs) | cosine similarity, Unicode tokenizer, BM25 term scoring |
packages/shared |
TypeScript | Shared Zod schemas + types |
Where the native module matters: the main win is cosine similarity on the brute-force vector path (used when the bundled sqlite-vec extension can't load). The native tokenizer + BM25 power the fallback lexical ranker and the find_capability catalog search โ primary lexical ranking for search_text is SQLite FTS5's own bm25() over the trigram index. Everything works without a prebuild.
Server-side (one process, soft TS-module boundaries): Transport (JSON-RPC over STDIO / Streamable HTTP), Auth (JWT / none), ACL (scope vs path/op), Policy (idempotency, rate limit, HITL), Router, Tool impls, Plugin bridges (HTTP to the companion plugin), Embedding providers (eight registered: ollama / openai / voyage / cohere / bge-m3 / openai-compatible / model-tier / module โ none privileged in code), Native module, SQLite caches (authored cache.db + quarantined experiential.db per vault), Observability emitters.
Obsidian-side (separate process): Companion plugin and the third-party Local REST API plugin.
Every tools/call flows through seven layers; observability always fires, even on error:
-
Transport โ parse JSON-RPC, normalize to
ToolRequest, computeargs_hash. -
Auth โ validate JWT or accept
none(loopback only); buildAuthContextwith scopes. -
ACL โ evaluate the tool's declarative ACL annotation against scopes + vault + paths; honor the global
readOnlykill switch. -
Policy โ (a) idempotency-key replay, (b) per-class rate limit, (c) HITL threshold โ mint/consume
elicit_token. - Router โ static name โ impl lookup.
- Tool impl โ Zod-parse args, run logic (may call native, a bridge, an embedding provider, SQLite).
-
Observability โ OTLP span, Prometheus counter + histogram, event emit, JSONL trace,
event_logrow.
Details and the per-error short-circuits live in Security and ACL and Observability.
| Boundary | Wire | Auth |
|---|---|---|
| Server โ Companion plugin | HTTP over the Local REST API port (127.0.0.1:27124), path-versioned /obsidian-tc/v1/*
|
Shared bearer token (REST API key) |
| TypeScript โ Rust native | napi-rs FFI, synchronous, copy-on-boundary | In-process |
The native module never throws on a missing prebuild โ a hand-written loader tries a local .node, then the platform sub-package, then the numerically identical pure-JS fallback (nativeLoaded tells callers which backend is active).
Each configured vault is logically isolated: its data lives in the shared SQLite cache (cache.db + experiential.db) row-scoped by vault_id โ not a per-vault DB file (physically separate per-vault files are a planned V2 rewrite, not current) โ under its own slice of the global ACL (per-vault acl overrides supported). The embedding provider is server-global (one per server). Vault resolution order: explicit args.vault โ OBSIDIAN_TC_DEFAULT_VAULT โ config default โ the sole vault โ else invalid_input.
The converged retrieval engine shipped in the v1.4โv1.7 line: GraphRAG graph-walk via vault_graph_search (vector seeds + wikilink expansion, fused with RRF), FTS5 BM25 text search and dense-vector search (vec0 with a per-vault partition key, so KNN prunes to the query vault's shard), and gateway-optional rerank + knowledge_challenge red-team. Every ranking change is gated by an n=250 golden set with a statistical ship rule; contextual chunk enrichment (embeddings.chunkContext, default on) measured +0.223 nDCG. Mechanisms that did not earn a default ship dark behind flags, each with a one-command re-test. Rerankers and the class router have A/B results on the record; learned sparse, query decomposition and convex fusion have no recorded result (see docs/DARK_MECHANISMS.md). An experimental graph-densification pass (off by default) can add derived edges to the wikilink graph beyond authored links: frontmatter tag co-occurrence, vec0 kNN semantic neighbors (built from the embeddings already computed, no egress), and an optional LLM pass routed through the local inference gateway. The walk traverses them down-weighted vs authored links. It is derived and rebuildable, never written back into notes as wikilinks, and stays dark behind retrieval.densify.* pending its own golden-set A/B โ retrieval defaults are unchanged.
The experiential tier is a membrane: a physically separate experiential.db beside the authored cache.db, with its own migration chain. Nothing in it can reach an authored chunk except through query-time composition.
-
Write side โ
chunk_retrievalslogs every serve-path retrieval (with later-stampedfeedbackandcited_in_responseaxes; theoutcomeaxis was retired);agent_episodesauto-captures dispatch outcomes (content capture defaults off), bornpending, with a deterministic pre-ingest poison scanner that marks injection-shaped content born-ineligible. -
Evaluation (
obsidian-tc reflect, sleep-time) โ pending episodes promote under deterministic rules; an optional gateway judge can only lower; the same pass maintains a versioned preference profile updated only by typed deltas. - Read side โ the M8 tools enforce the reader contract: eligible-only, tombstones, trust floor, caller partition.
-
Deletion โ
forgetpropagates through derived state with tombstone-vs-erase modes and a hash-chainedforget_log(editing any entry breaks verification). -
Flywheel โ access statistics are views over the retrieval log (never mutated columns on the authored store); the
metrics/gapsCLIs feed a cycle-close loop that writes scorecards back into the vault.
MCP spec 2025-11-25. STDIO for local; Streamable HTTP for HTTP modes. The server advertises tools, resources, and prompts โ notes are readable as MCP Resources over obsidian-tc://<vault>/<path> URIs (read-scope + folder-ACL enforced). HITL uses a custom single-use elicit_token pattern that works with any MCP client.
The typed-atom MemIR substrate (claim atoms, authoritative_claims, derives_from provenance) remains reserved for a next substrate generation. Its qualification spike ran in 2026-07 and did not clear the pre-registered quality bar at the local-model footprint, so the program is parked with recorded numbers and an explicit re-entry trigger. v1.x is the stable line; V2 capabilities would be added additively.