agentm recall trace - alexherrero/agentm GitHub Wiki
Note
LAUNCHED (approved 2026-07-25, built same day). Widens the existing
recall-history.jsonl event ledger with the evidence behind each recall
hit, plus a small reader for it โ no new store. Rung 8 of the Loose Ends
ladder; sibling to memory index. Full build
narrative in the Amendment log below.
When a recalled memory looks wrong โ stale, irrelevant, mis-ranked โ there is
no way today to ask the system why it surfaced. The evidence exists for one
prompt turn only: recall.py's _format_recall_result prints sim= and
keywords= into the injected block, then it's gone. Auditing a bad recall
means re-running the same query later and hoping the index, the decay clock,
and the vault haven't moved on since. This design persists the evidence at
the moment recall already computes it, so a surfaced memory stays explainable
after the session that surfaced it has ended.
recall_counter.py already writes one JSONL row per prompt-submit recall
call โ query hashed, never raw text, plus the slugs that actually survived
truncation. It is the sole per-recall event log in the codebase (heat_policy's
.heat.json is a rolled-up per-slug counter, not an event log). This design
widens each row with a hits array carrying, per surfaced entry, the score
breakdown query() already produces (sim, keyword, combined, rank,
lifecycle tier, decay score) โ no new computation, just capturing what already
exists in memory at the existing call site before it is discarded. A new
memory-recall trace <slug> subcommand reads the ledger back out,
most-recent-first, for operator or console use.
The ledger is write-only today. record_recall() has been appending
since 2026-07-13 (1,422 rows as of this drafting), but nothing in production
reads it: count_since()'s only caller is its own unit test, and no health
or console surface imports the module. recall_counter.py's docstring says
it exists because "the Morning Brief's 'retrieved' count needs a real
per-recall signal" โ that consumer was never wired up. This design does not
fix that gap, and should not be read as fixing it; it is named here because
a design resting on "the ledger is already consumed" would be resting on
something untrue. The trace reader becomes the ledger's first production
reader, which is an argument for the design, not evidence the plumbing
already works.
recall.query() already returns, per hit, a dict with path, slug, sim
(vector cosine-ish score), keyword (BM25 score), combined (the final RRF
ร decay score used for ranking), and โ when lifecycle.py is importable โ
lifecycle_tier and decay_score. The trace adds exactly one derived field
that doesn't already exist on the dict: rank, the hit's 1-indexed position
in the list query() returns (already top-k'd and sorted by combined).
Rank is read by enumerating results itself, not the surviving subset, so an
unreadable-file skip or a token-budget drop never renumbers the hits that
remain. Everything else is a straight carry-over โ the trace does not
re-derive or re-score anything.
Each hit carries path, not just slug. Slugs are not unique in a real
vault: the live vault has 6,028 notes with many repeated basenames
(_index.md, _summary.md, PLAN.md, README.md, dated daily notes). The
two worst cases are _index and _summary โ precisely the
_ALTITUDE_ANCHOR_SLUGS that query() rank-boosts, so they are more likely
than average to appear in a result set. A trace keyed on slug alone would be
ambiguous exactly where it is most often consulted. The existing hit_slugs
field has this same ambiguity; it is left as-is for compatibility, and hits
carries the unambiguous vault-relative path alongside the slug.
Two fields are deliberately left out rather than invented:
-
Per-stream flags (did vec vs. BM25 independently produce this hit).
sim == 0.0/keyword == 0.0already imply "this stream didn't surface it," within the float-boundary caveat that a genuine zero score is indistinguishable from absence โ the same ambiguity_format_recall_resultalready lives with in its own header line. Not worth a second pair of fields to resolve a distinction the existing display doesn't resolve either. -
Altitude-boost flag. Whether
_ALTITUDE_BOOSTapplied is a pure function ofslug in _ALTITUDE_ANCHOR_SLUGSโ derivable at read time from the stored slug, not worth persisting.
prompt_submit() is the only call site (matches recall_counter.record_recall's
own single-call-site discipline, stated in its docstring: a lint walk, index
rebuild, or dreaming pass must never reach this). The capture point is right
after token-budget truncation decides loaded_slugs โ the same point
record_recall(prompt, loaded_slugs) already fires from.
Recovering which hits survived needs care, and the obvious method is wrong.
prompt_submit() builds raw_blocks/raw_slugs in a loop that continues
past unreadable files, then passes them through _apply_token_budget(), which
skips oversized blocks and keeps going. So the surviving loaded_slugs is an
order-preserving subsequence of raw_slugs, which is itself a subsequence
of results โ and with duplicate slugs in play, filtering results by
membership in loaded_slugs can match the wrong entry or the same entry
twice.
The fix is to carry the evidence through the budget filter rather than
reconstruct it after: build a raw_hits list index-aligned with raw_blocks
by construction (appended in the same loop iteration), then pass the slug and
its hit dict through as a single packed pair:
blocks, kept_pairs, omitted = _apply_token_budget(
raw_blocks, list(zip(raw_slugs, raw_hits)), token_budget
)
loaded_slugs = [s for s, _ in kept_pairs]
kept_hits = [h for _, h in kept_pairs]This works because _apply_token_budget measures only blocks โ the second
list is zipped along and carried, never inspected โ so packing a payload into
it cannot change a keep/skip decision. The alternative (returning kept indices)
would change the helper's return arity at both call sites plus roughly ten
test assertions, including the always-load path session_start() shares; this
version leaves the helper's logic and arity untouched and widens only its
slugs type annotation from list[str] to list.
recall_counter.record_recall's signature grows an optional hits parameter;
hit_slugs (the existing list[str]) is unchanged, so count_since and any
other reader that only cares about counts keeps working untouched. The new
row shape:
{
"ts": "...",
"query_hash": "...",
"hit_slugs": ["slug-a", "slug-b"],
"hit_count": 2,
"hits": [
{"slug": "slug-a", "path": "personal/slug-a.md", "sim": 0.81,
"keyword": 12.3, "combined": 0.0163, "rank": 1,
"lifecycle_tier": "volatile", "decay_score": 0.94},
{"slug": "slug-b", "path": "projects/agentm/slug-b.md", "sim": 0.0,
"keyword": 8.1, "combined": 0.0157, "rank": 2}
]
}lifecycle_tier / decay_score are omitted per-hit when lifecycle.py
wasn't importable at recall time (mirrors query()'s own optional-field
contract on the result dict โ no fabricated defaults). Pre-existing rows
written before this change simply lack hits; the trace reader treats a
missing key as "this event predates trace capture," not an error or a zero.
memory-recall trace <slug> [-n N] (default N=3) returns the N most recent
events whose hits contain that slug, printing each one's timestamp, the
matched entry's path, and its score breakdown โ echoing
_format_recall_result's familiar header shape (sim=/keywords=/tier:),
now durable and queryable outside the session. Scores print at fixed
precision (sim=%.2f, keyword=%.1f) rather than raw repr; keyword is a
BM25 float since V6-3, so unformatted interpolation would print a
twelve-digit tail.
It reads the whole file and filters, most-recent-first. A true reverse
streaming read would be the scalable choice, but the ledger does not warrant
it yet and the extra code would be the expensive half of this build: the file
holds 1,422 rows / 167 KB after 13 days (~110 rows/day at ~119 bytes). Adding
hits grows a row to roughly 600โ700 bytes, so ~70 KB/day, ~25 MB/year. Read
cost stays trivially small on that horizon; count_since() already reads the
whole file the same way. Re-audit trigger: if the ledger passes ~50 MB, or
a trace call takes longer than a second, switch to a reverse chunked read and
add rotation.
Degrades honestly: a missing ledger file, or a slug that never appears in it,
prints an explicit "never recalled" line rather than staying silent โ
matching console.py's honest-dark convention, where a check that has never
run says so instead of vanishing. A slug found in hit_slugs but with no
hits entry prints "recalled before trace capture landed" โ the pre-existing
rows are not retrofittable and must not read as a zero score.
No new dashboard section. console.py's "Memory activity" block already
shells out to recall.py's subcommands for its rollups; this design adds one
documented pointer there (memory-recall trace <slug> as the drill-down for
"why did entry X surface") rather than a new always-rendered row โ a per-slug
trace is a drill-down by nature, not a summary metric, and forcing it into
the always-on rollup would just be unread noise on every /console run.
recall.py (query()'s per-hit dict โ the source of every field this design
persists) ยท recall_counter.py (the ledger this design widens) ยท
lifecycle.py (optional per-hit tier/decay fields, already optional at the
source) ยท console.py's Memory activity section (gets one doc pointer, no
code dependency).
-
hits[].pathdurably discloses vault directory structure, with no redaction tied to a note's later lifecycle. Found by a pre-tag adversarial security review, missed by this design's original risk analysis.pathis a strict superset of whathit_slugsalready stored (a bare filename stem) โ it additionally reveals which group/project a note lives under, and nothing purges or redacts a ledger row when its source note is later deleted, moved, or merged by crystallization. On this operator's actual deployment (a single-user local machine where the same OS account already reads both the ledger's cache path and the vault itself) this is not a new trust-boundary crossing, so it does not block this design โ but it is a real, previously- unweighed gap, named honestly rather than assumed away, matching this design's own "the ledger has no production reader" discipline above. A bare hash (thequery_hashprecedent) isn't the fix: it would maketrace()unable to say which entry surfaced, defeating the design's whole purpose. Re-audit trigger: before this ledger's data ever leaves the single-operator/single- machine trust boundary it was built inside โ e.g., any future shared, synced, or multi-user reader. Partly resolved 2026-07-26 by recall ledger retention: rows now expire on a 90-day clock, so a path no longer outlives its note indefinitely. That bounds how long structure is disclosed; it does not change who can read it, so the trust-boundary trigger above stands unchanged. -
Ledger has no retention policy.
โ resolved 2026-07-26 by recall ledger retention. Rows past 90 days are swept fromrecall-history.jsonlgrows unboundedrecord_recall()'s own write path, with a 25 MB ceiling as a backstop. The ceiling deliberately sits below the ~50 MB re-audit trigger named above, so enforcement engages before the threshold that says "go redesign this." Measured growth is ~28 KB/day, below the ~70 KB/day this design estimated. -
The ledger's only production reader will be the one this design adds.
Everything the design claims about the ledger's shape is verified against
recall_counter.pyitself, not against a downstream consumer's expectations โ because there is no downstream consumer. If the never-wired Morning Brief "retrieved" count is ever built, it should be checked against the widened row shape rather than assumed compatible. -
hit_slugsstays ambiguous. The existing field cannot distinguish two entries sharing a basename, and this design does not change it (compatibility withcount_since). Thehitsarray is the unambiguous record; anything reasoning about which entry surfaced must readhits[].path, neverhit_slugs. -
Eval-gate discipline. This is additive instrumentation at an existing
call site โ it must not touch
query()'s ranking orprompt_submit()'s output.scripts/health/eval_v6_retrieval.py(the pinned retrieval eval) must stay green untouched; if it doesn't, the change has leaked into the ranking path and needs to be narrowed back to capture-only. Verified by this design's own build plan's task 7 โ see the Amendment log. -
Partial evidence on budget overrun.
prompt_submit()'s own time budget can already truncateresultsbeforequery()finishes every stream; the trace reflects whateverquery()actually returned in that case โ same degraded-graceful contract the rest of recall already has, not a new gap. - Built โ see the Amendment log for what shipped and where.
-
Widen the existing ledger; add no store.
recall-history.jsonlis already the per-recall event log at the exact call site where the evidence exists. A second store would need its own writer, rotation, and reader for data one field away from what is already being written. -
Capture is carry-over only. Every field except
rankalready exists onquery()'s result dict. The trace must not re-score, re-rank, or re-read the vault โ if it computes anything, it has become a second ranking implementation that can silently disagree with the first. -
Pack the payload through
_apply_token_budget; leave the helper's logic and arity alone. It is shared with the always-load path and carries ten-odd test assertions. Widening itsslugsannotation tolistis the whole change. -
hits[].pathis the identity;slugis a label. Duplicate basenames are real and concentrated in the altitude-boosted anchors. -
governs: []โ memory-system keeps governance.agentm-memory-system.mdalready stampsharness/skills/memory/scripts/, sorecall.pyandrecall_counter.pyare governed. A narrower stamp here would not trip the overlap gate (different pattern string) but would quietly redirect/planand/reviewresolution for those files to a thin sub-slice design. Not worth it for one instrumentation change. - No new console section. A per-slug trace is a drill-down; the Memory activity block gets a documented pointer, not an always-rendered row.
-
harness/skills/memory/scripts/recall.pyโquery()(the per-hit dict this design persists verbatim),prompt_submit()(the capture call site),_format_recall_result()(the header shape the trace reader echoes) -
harness/skills/memory/scripts/recall_counter.pyโrecord_recall()/count_since()(the ledger this design widens, additively) -
harness/skills/memory/scripts/heat_policy.pyโ the rolled-up per-slug counter this design deliberately does NOT duplicate (different shape, different purpose) -
harness/skills/console/scripts/console.pyโ the Memory activity section gaining the doc pointer -
memory index ยท memory system โ
sibling designs
query()'s hybrid-recall + lifecycle machinery live under -
<vault>/Projects/agentm/_harness/designs/loose-ends/SWEEP-INVENTORY.mdโ the row this design resolves ("Recall-trace substrate")
Newest first.
2026-07-26 โ Retention gap closed; two Risks bullets reconciled. The
follow-up this design flagged rather than built shipped as
recall ledger retention: rows expire on a
90-day clock, swept from record_recall()'s write path. The path-disclosure
bullet is now partly resolved โ retention bounds the exposure window, not the
trust boundary, so that trigger survives intact โ and the retention bullet is
resolved outright.
Two things this design recorded turned out to be wrong. The test-pollution note
below ("flagged as a separate follow-up, not fixed here") named one file; there
were three, found independently by two branches and fixed on main by
PR #390. That bug also stopped
being merely cosmetic once record_recall could rewrite the ledger, since an
unmocked run would read-modify-write real telemetry rather than just append to
it. And the measured growth rate is ~28 KB/day rather than the ~70 KB/day
estimated here, which puts the ~50 MB re-audit trigger years out and makes
bounded disclosure, not bounded size, the reason retention was worth building.
2026-07-25 โ Built, same day as approval. All four Design subsections shipped exactly as locked, no deviation found worth a NOTE:
-
recall_counter.record_recall()gains the optionalhitsparam (harness/skills/memory/scripts/recall_counter.py:32) โ omitted by default writes today's exact row shape (nohitskey at all, not[]). -
recall.py'sprompt_submit()packs(slug, hit)pairs through_apply_token_budget(harness/skills/memory/scripts/recall.py:1561buildsraw_hits,:1586unpackskept_hits,:1595passes them torecord_recall) โ_apply_token_budget's logic and arity are untouched, only itsslugsparameter's type annotation widened.rankis read offresults' ownenumerate(), before the read-loop's unreadable-file skip can renumber it โ proven necessary and correct by a duplicate-slug test (two entries sharing a slug, different paths) that fails under the naive "filterresultsbyloaded_slugsmembership" approach this design itself rejected below. - The reader lands as
recall.trace()(harness/skills/memory/scripts/recall.py:1356)- the
memory-recall trace <slug> [-n N]CLI subcommand โ a sized whole-file read (not the reverse-chunked scan an earlier draft wrongly promised), all three degrade cases each an explicit printed line.
- the
-
console.py's Memory Activity section gets its one documented pointer (harness/skills/console/scripts/console.py:535), not a new row.
19 new tests across three files (4 in test_recall_counter.py, 9 in the new
test_recall_trace.py, 1 in test_console.py), plus manual smoke-testing
against this machine's real 1,400+-row production ledger โ which surfaced a
genuine pre-existing bug, unrelated to this design: test_recall_token_budget.py's
prompt_submit() integration tests don't mock recall_counter, so they've
been writing test rows into the real ledger. Flagged as a separate follow-up,
not fixed here (out of this design's scope).
Eval-gate outcome (task 7). bash scripts/check-all.sh โ 34/34 green,
including the full unittest discover sweep (which already exercises
scripts/health/test_eval_v6_retrieval.py's own fixture-based suite). The
CLI (python3 scripts/health/eval_v6_retrieval.py) run directly against the
real vault exits 1, but on a cause unrelated to this design: 4 of its
pinned query set's expected-notes paths have moved or been archived since
that query set was authored โ the intentional fail-loud-on-drift behavior
v9.2.1 shipped, refusing to report a number it can't trust rather than
silently under-counting. This diff cannot be the cause (it touches zero
vault content), and the drift predates it. Flagged separately, not fixed
here. In its place: query() โ the function that does every bit of
scoring, ranking, RRF fusion, and decay/lifecycle weighting โ is byte-for-
byte untouched by this diff (confirmed directly against the base commit,
not merely asserted); _apply_token_budget's only change is a type
annotation plus docstring text, zero logic lines. Ranking cannot have
regressed by construction, independent of the live-vault eval's drift
issue.
Update, same day: the drift's own follow-up (spawned separately, not by
this design) landed as PR #391
before this release tagged. Re-run post-fix: merge_gate_passed=True,
accuracy_regressed=False, expected_files_missing=0 โ the pinned eval now
confirms this diff clean directly, not only by the code-level argument above.
2026-07-25 โ Initial draft, authored per the Loose Ends ladder's rung 8
(conditional item, handed to a dedicated session). Grounded against recall.py
/ recall_counter.py / vec_index.py / console.py as they exist today, then
corrected on a verification pass that caught four things the first draft had
wrong: the ledger has no production reader (the design had implied one), slugs
are not unique so hits must carry path, filtering results by
loaded_slugs mis-identifies entries under duplicate basenames, and the
promised "backward scan" would not have been implemented as described. Area
corrected agentm/recall-trace โ agentm/memory (the former is not in the
taxonomy check-governs-index enforces, so it would have failed the gate) and
the page linked into designs/_Sidebar.md + Designs.md (check-wiki's
no-orphan rule).