Note Relatedness Signals - alexherrero/agentm GitHub Wiki
The notes_link_discovery.py script scores signals. It finds related-but-unlinked pairs among your personal notes. This corpus includes everything outside the vault root and .obsidian/, minus the areas the filing contract walls from recall. The audit never mutates a personal note. It surfaces candidate links for you to review (A3). Suggestions are strictly personal↔personal. A vault-root entry is never a source or a target (DC-2).
| Question | Answer |
|---|---|
| What runs the audit? |
harness/skills/memory/scripts/notes_link_discovery.py (the relatedness engine + report writer). |
| How do I see suggestions? |
python3 harness/skills/memory/scripts/notes_link_discovery.py --format text (or --format json). |
| Which notes are in the corpus? | Personal notes only — the Obsidian vault excluding the vault root folder, .obsidian/, .trash/, .git/ (DC-2) and every area the filing contract walls with recall_exempt_areas. |
| What does the wall keep out? |
personal/Home/Important Docs and anything else the contract names. The walk refuses those folders before reading a file, so a walled note is never scored, never named in the report, never embedded or cached, never in the --apply backup, and never written into by --apply. vault_lint's link-target walk holds the same wall: it names a walled note so links to it still resolve, without opening it. |
| What are the two signals? |
TF-IDF content overlap (lexical) + embedding cosine (semantic, opt-in via --embeddings). Folder + date proximity are weak context. |
| Does the audit ever edit a note? | Not by default — read-only / surface-only (DC-1). The opt-in --apply flag is the one exception: it writes the safe suggestions into a marked ## Related section, backup-first + idempotent (you directed it; A3 satisfied). |
| Where do outputs live? | Report → <vault>/diagnostics/lint/notes-links-<date>.md; embedding cache → ~/.local/state/agentm/notes-embeddings.json (the engine state directory; $AGENTM_STATE_DIR overrides). Never beside a personal note. This cache is this tool's own — it was never the AgentMemory index, which has since been removed entirely. |
| How do I clear stale keys out of the cache? |
--prune-cache drops every key the contract walls and every key naming a note the corpus no longer holds, then says how many of each went. It loads no model; --dry-run reports without writing. See Find missing note links. |
| How do I run the report? | See Find missing note links. |
| Related pages | Find missing note links |
Your personal-notes corpus lacks a usable graph signal. Only a handful of the ~390 notes carry tags. Only one note contains a [[wikilink]]. The frontmatter only holds title, created, and updated. Tags, links, and frontmatter fields are dead signals here. The script calculates relatedness entirely from the content.
| Signal | Role | Notes |
|---|---|---|
| TF-IDF term overlap (title + body) | Primary (always on) | Lowercase alphanumeric tokens ≥3 chars; English + Spanish stopwords stripped; HTML/CSS/#hex/image-embed/URL markup stripped before tokenizing (many notes are clipped web content); title terms double-weighted; sublinear-tf × IDF, L2-normalized, cosine-scored via an inverted index. |
| Embedding cosine (title + lede) | Secondary (opt-in --embeddings) |
Each note embedded with the local BGE model (embed.py), full pairwise cosine. Catches same-topic/person/event pairs with little shared vocabulary — including cross-language pairs TF-IDF structurally can't see. Graceful-skips to TF-IDF-only when sentence-transformers is absent. |
| Folder context | Secondary | Same-folder pairs are labelled same folder in the report (navigational grouping); cross-folder pairs show A ⇄ B. |
| Date proximity | Available |
created / updated are parsed into the corpus but not scored in v1. |
Existing [[wikilinks]]
|
Exclusion only | A pair already linked (by stem or relative path, either direction) is never re-suggested. |
| Tags | Dead | ~no notes carry tags — no signal in this corpus. |
| Frontmatter fields | Dead | Only title / created / updated present — no signal. |
| Flag | Default | Meaning |
|---|---|---|
--min-score |
0.18 |
TF-IDF cosine floor. Pairs below it are dropped. |
--embed-min-score |
0.70 |
Embedding cosine floor (BGE similarity runs hot — ~0.3–0.5 even for unrelated prose — so the semantic threshold sits well above the lexical one). |
--top |
40 |
Per-signal shortlist cap (0 = all). Keeps the report a high-signal shortlist, not noise. |
The corpus contains ~390 notes. This creates ≈76k candidate pairs. The TF-IDF inverted index compares only term-sharing notes. A document-frequency band drops per-note noise and ubiquitous boilerplate. The embedding pass runs a bounded O(n²) cosine over cached vectors. The report deduplicates matches. A pair found by both signals appears only once. It displays in the TF-IDF section. The report flags it with ✓ also semantically related. The embedding section lists only the pairs TF-IDF missed.
- Find missing note links — You use this operator recipe to run the audit and read the report.