Exp 2026 10 12 Shared Doorbell Drain Defects - oRKLLM/ork-driver GitHub Wiki

Exp-2026-10-12 — three defects in the SHARED doorbell drain

Found while root-causing the int4 storm on .235 (6.1.115-vendor-rk35xx-fix). All three live in src/npu/core/ — the shared NONBLOCK doorbell spine — so they are not int4 bugs, they were merely exposed by an int4 workload that misses constantly. Landed 99771bc, make test ALL PASS.

1. The miss-window EWMA key omitted the geometry

miss_to comes from orki_op_predict_us(pkey), keyed on (dt, dyn_end, N, b_M, S) — no H, no Wb. So a geometry the board takes and one it will not shared a slot: the EWMA learned their average and the window was then wrong for both.

With the ladder on, a successful retry trains the key to ~4 ms and the window clamps to its floor; with the ladder off nothing succeeds, est stays 0, and the 2 s CATCHALL applies — 6 × 2 s = 17 s instead of 5.2 s.

Fix: orki_opkey takes a geom component, orki_geomkey(H, Wb).

2. The 50 ms miss floor sat below one poll pass

Each recover round reported waited 102262us/50000us. One pass over three cores' surfaces costs ~50 ms, so miss detection could not resolve finer than ~15× the op's correct runtime (3.5 ms).

Fix: the floor is now measured — 3× the first complete poll pass (ORK_MISS_PASS, ORK_MISS_FLOOR_US) — so it scales with the surface by construction instead of being a constant calibrated on one shape.

3. ★ The recover re-seeded padding, and the adjudicator could then never pass

BCHAIN writes only Hg·NBc·64 of each 4·H·Wmax·64 output slot — 75% of the slot is padding the NPU never touches (see the dispatch-envelope page for why the 4× is structural and not removable).

orki_mc_recover_resubmit's esz==2 branch re-seeded all of nout. So after any recover, ~75% of the surface sat permanently at 0x7fff, and orki_dyn_collision_landed — which scans the whole surface against a 0.1% survivor cap — was structurally unable to accept on this path ever again. Census after a re-seed: 92–94% survivors, of which ~75 pp was padding.

Fix: the recover re-seeds the written cell set (bch_db_cells mode 0), and the adjudicator uses mode 4 — a per-program "did this program write anything" — which touches no padding.

Related, found in the same place: collide_ok latched for the chain's life

Once set, it stayed set, so from recover round 1 every poll iteration ran the full-surface adjudication that its own comment says never runs on the hot path. That is what actually produced the waited 102262us/50000us overshoot in §2. It is one-shot per round now.

Why these were invisible

All three are failure-path code. A board that never misses never executes them, so they survived in the shared spine until a workload that misses constantly (.235's int4 BCHAIN, then running at ~5.2 s/call behind a ladder rescue) walked every branch. The ladder returning a bit-exact answer is exactly what kept them hidden — nothing ever reported an error.

Worth generalising: a correctness-preserving recovery path hides its own defects, because the only symptom is time. Anything that self-heals needs a timing gate as well as a correctness one.

Notes

  • tools/re/i4_storm_probe.c was written for this work: repeated, timed, per-call checksum plus a sentinel census, so self-heal / silent miscompute / value collision are distinguishable. ⚠ It grades against its own iteration 0, which is itself a geometry — adequate for storm detection, not for grading a change that alters the geometry. Use i4_e2_probe / i4_ladder_probe (absolute CPU reference) for those.
  • The in-band sentinel is a separate standing defect: ORK_DYN_SENT16 = 0x7fff is a legitimate saturated int16 output, so a saturated element is indistinguishable from "not written". It is ~33σ on real rotated data, and it has been forced and reproduced 8/8. See Exp-2026-10-13 INT4 BCHAIN Completion Gate §6 for why no out-of-band witness is constructible.