Retrieval Eval Contract - alexherrero/agentm GitHub Wiki
Note
Reference â the measurement contract for eval_retrieval_shipped.py and
every pre-registered retrieval experiment. What the instrument can resolve,
what a rule must contain before a run, and the two false nulls that made
these requirements standing rather than ad hoc.
One number with provenance. The eval issues each gold-set question exactly as
the recall hook does â hybrid mode, term extraction, Ă2 over-fetch, the
admissibility filter, temporal bounds â and scores exact-path membership at
k=5 against gold-set-v3.json (64 scored questions, 20 easy negatives, 10
near-miss negatives). Baselines embed a corpus fingerprint (document count,
embedded-in-scope count, gold-set hash, pin date); --compare refuses across
fingerprints unless --drifted-ok says the drift is understood, and a
gold-set hash mismatch is never overridable. R@5 is the product metric â the
hook injects five â and R@1 rides along as the ordering signal.
Printed on every report, so nobody can pre-register a bar the instrument cannot resolve without having seen it refuse the idea:
| scored n | smallest clean gain (6 flips) | Wilson 95% width near 75% |
|---|---|---|
| 64 (today) | +9.4pp | ~21pp |
| 100 | +6.0pp | ~17pp |
| 125 | +4.8pp | ~15pp |
| 250 | +2.4pp | ~11pp |
Six flips is derived from the exact paired test (two-sided binomial: 2·(œ)^k < 0.05 first at k=6), not chosen. A true 5pp improvement at n=64 best-cases p=0.25 â invisible. Per-target rank probes remain the sensitive instrument for sub-MDE mechanisms: they measure movement exactly, and they are how three alias strategies were shown to move nothing at all.
A RULE file (template: scripts/health/results/RULE-TEMPLATE.md) is committed
before any run, and it must contain:
-
A power-checked bar.
coin_pass_probability(bar, n) †0.05, computed by the eval's own function and quoted in the file. Both of this arc's pre-flight bars â floorless rerank's â„5-of-9 and fusion's â„3-of-5 â sit at exactly 0.50 by this arithmetic: coins with paperwork. The floorless refutation is withdrawn as unproven because the bar was its only evidence; fusion's stands only on its deterministic per-question trace. - A positive control. Something in the run that fails if the instrument is dead, named in advance. The standing controls (schema assertion, planted canary, score spread) cover the eval itself; a probe touching anything else supplies its own.
- Per-question diffs. A net "+2" once hid a +5/â3 split; totals are not results.
- A written prediction. Checked against the outcome either way, in the same file.
The field-name null, twice. expected read where the fixture says
expected_note_paths parses every expectation to an empty list, and a run
reports a clean "0 of N" that is really N comparisons against nothing. It
shipped one refutation and reproduced on the next rung's first attempt. The
schema control now aborts it (exit 4).
The keyword-soup null. The cross-encoder rung fed both models concatenated terms rather than natural questions; the technique looked flat because the query representation was. Re-measured with real phrasing before the verdict was trusted â which is what clause 2 generalizes.
None are load-bearing here, deliberately â the scoring path is deterministic. Any future judged metric must first measure operator agreement on a sampled set and publish that number beside its own. The completeness-v1 scorer is the standing reason: its gutted-note check was green in both directions while it agreed with the operator at Ï = 0.11, because the check exercised a stage downstream of the defect.
The rules above govern the offline gold-set ladder. Live traffic adds these. Each one exists because breaking it produced a wrong number once.
| rule | why | breaking it cost |
|---|---|---|
| No raw prompt text on disk | the judge's account of a gap quotes the request by construction | a repository holding the operator's prompts |
| Deterministic comparator runs and is reported apart | it fired on 7 of 3,004 injected notes â showing that is the point | a floor mistaken for a second opinion |
| Suppress the comparison when the floor never fires | the disagreement then equals the judged rate exactly | one number printed twice as though two signals agreed |
| No rate without the instrument's resolution beside it | the judge drifts ~5 points across identical runs | a precision claim the instrument cannot support |
| Always-valid interval, not fixed-horizon | the number is read whenever someone wonders | over 800 peeked streams, a Wald interval was wrong on 43â46% against a sequence's 0.0% |
| Measure at the change, paired and interleaved | the judge's ~5-point drift is common-mode and cancels in a difference | two rates a month apart, each carrying the full drift |
| A sample too small says so | ~45 judged turns clear the bar, 274 reach ±0.10 |
[0%, 54%] rendered as if it were a range |
Îș before publication does not apply here, and the exemption is recorded
rather than waived. No independent human labels are obtainable from this data â
unaided ones proved error-prone, aided ones are not independent â so the online
row carries a permanent UNVALIDATED marker with its reason attached, instead
of a Îș it cannot compute.
CI-Gates · the gate script scripts/check-retrieval-regression.sh
· results record scripts/health/results/goldv3/NOTES.md · design
agentm-hybrid-retrieval.