Exp 2026 10 12 Shared Doorbell Drain Defects - oRKLLM/ork-driver GitHub Wiki
Exp-2026-10-12 — three defects in the SHARED doorbell drain
Found while root-causing the int4 storm on .235 (6.1.115-vendor-rk35xx-fix). All three live in
src/npu/core/ — the shared NONBLOCK doorbell spine — so they are not int4 bugs, they were merely
exposed by an int4 workload that misses constantly. Landed 99771bc, make test ALL PASS.
1. The miss-window EWMA key omitted the geometry
miss_to comes from orki_op_predict_us(pkey), keyed on (dt, dyn_end, N, b_M, S) — no H, no
Wb. So a geometry the board takes and one it will not shared a slot: the EWMA learned their
average and the window was then wrong for both.
With the ladder on, a successful retry trains the key to ~4 ms and the window clamps to its floor;
with the ladder off nothing succeeds, est stays 0, and the 2 s CATCHALL applies — 6 × 2 s = 17 s
instead of 5.2 s.
Fix: orki_opkey takes a geom component, orki_geomkey(H, Wb).
2. The 50 ms miss floor sat below one poll pass
Each recover round reported waited 102262us/50000us. One pass over three cores' surfaces costs
~50 ms, so miss detection could not resolve finer than ~15× the op's correct runtime (3.5 ms).
Fix: the floor is now measured — 3× the first complete poll pass (ORK_MISS_PASS,
ORK_MISS_FLOOR_US) — so it scales with the surface by construction instead of being a constant
calibrated on one shape.
3. ★ The recover re-seeded padding, and the adjudicator could then never pass
BCHAIN writes only Hg·NBc·64 of each 4·H·Wmax·64 output slot — 75% of the slot is padding the
NPU never touches (see the dispatch-envelope page for why the 4× is structural and not removable).
orki_mc_recover_resubmit's esz==2 branch re-seeded all of nout. So after any recover,
~75% of the surface sat permanently at 0x7fff, and orki_dyn_collision_landed — which scans the
whole surface against a 0.1% survivor cap — was structurally unable to accept on this path
ever again. Census after a re-seed: 92–94% survivors, of which ~75 pp was padding.
Fix: the recover re-seeds the written cell set (bch_db_cells mode 0), and the adjudicator
uses mode 4 — a per-program "did this program write anything" — which touches no padding.
Related, found in the same place: collide_ok latched for the chain's life
Once set, it stayed set, so from recover round 1 every poll iteration ran the full-surface
adjudication that its own comment says never runs on the hot path. That is what actually produced
the waited 102262us/50000us overshoot in §2. It is one-shot per round now.
Why these were invisible
All three are failure-path code. A board that never misses never executes them, so they survived in
the shared spine until a workload that misses constantly (.235's int4 BCHAIN, then running at
~5.2 s/call behind a ladder rescue) walked every branch. The ladder returning a bit-exact answer is
exactly what kept them hidden — nothing ever reported an error.
Worth generalising: a correctness-preserving recovery path hides its own defects, because the only symptom is time. Anything that self-heals needs a timing gate as well as a correctness one.
Notes
tools/re/i4_storm_probe.cwas written for this work: repeated, timed, per-call checksum plus a sentinel census, so self-heal / silent miscompute / value collision are distinguishable. ⚠ It grades against its own iteration 0, which is itself a geometry — adequate for storm detection, not for grading a change that alters the geometry. Usei4_e2_probe/i4_ladder_probe(absolute CPU reference) for those.- The in-band sentinel is a separate standing defect:
ORK_DYN_SENT16 = 0x7fffis a legitimate saturated int16 output, so a saturated element is indistinguishable from "not written". It is ~33σ on real rotated data, and it has been forced and reproduced 8/8. See Exp-2026-10-13 INT4 BCHAIN Completion Gate §6 for why no out-of-band witness is constructible.