Exp 2026-10-08 — fp16 wide-K output head hangs the NPU (r104)

Verdict: FIXED. A buffer-sizing bug, not a hardware envelope bug. The M-chunk rule was computed twice from two different formulas, and the taller of the two programmed the NPU to write past the end of its output staging buffer. On this hardware that is not a fault — it is a hang.

Board 10.3.0.236, kernel 6.1.115-vendor-rk35xx-fence, base origin/main eb3e66f. (Note the kernel: the reported failure and this fix were both on -fence, not the -sram kernel AGENTS.md names. Results from different kernels are not comparable.)

Symptom

ork_f16_mm_pack + ork_f16_mm_run at M=1024, K=248320, N=1024 — the tied output head's dX, the contraction over the vocabulary — hung on run 0:

[ork] ERROR: doorbell sentinel never landed (op=run_loop dom=0 nout=360448 rep=1/1, waited 60000 ms)
[ork]   probe: commit+1 irq+0 done+0 (cnt_nojob=0) | submit_dom=0 weight_dom=0 imported=0
        core_mask=0x1 K=248320 N=1024 => COMMITTED, NO IRQ

Core 0 then sat at 99% and the board needed a power cycle. Every other shape tried passed at relerr 1.6e-7 … 2.8e-7, including M=1024 K=1024 N=248320 (the head forward) and a range of K from 256 to 6144.

Mechanism

orki_run computes the M-chunk twice:

1. a pre-loop pass that sizes the Af (activation gather) and Cc (output) staging buffers, and
2. the loop that actually emits the programs.

They had drifted. The emitting copy was moved onto the measured fp16 envelope orki_f16_mcap (2026-08-20) and then onto the N-aware orki_f16_mcap_n (r103, 2026-10-06). The sizing copy was never moved: it kept the legacy int8-derived mg_max*64. Where the fp16 envelope is the taller of the two — which is exactly where fp16 reaches the 11-bank CBUF capacity that int8 does not — the loop emitted a chunk the buffers could not hold:

K-slice	sizing pass said	emit loop said	Cc short by
Kp=512, sched=1	320 rows	352 rows	131,072 B
Kp=1024, sched=1	128 rows	176 rows	196,608 B
Kp=256, sched=1	704 rows	704 rows	—
any sched=0	16384/Kp	16384/Kp	—

The reported shape is the Kp=512 row. RK3588's fp16 K-slice is 2048, so K=248320 is 121 slices of 2048 plus a 512 tail — and that tail is where it died. The reported nout=360448 is exactly 352 × 1024, which is what identified the slice before any board run.

This is not the r103 starved-WEIGHT_BANK mechanism. That one is about which bank split the hardware can execute, and the N-aware cap already covers it; the originally-suspected "starvation is not limited to narrow N once K is sliced" is not what happened here. This is a plain buffer extent. Every way of walking off a mapped IOVA region on this NPU presents with the same "committed, no IRQ" signature, which is what made the two look alike.

Why the suite never caught it

fp16 M>1 at N>=32 goes to run_multicore → the colsplit path, which reads orki_f16_mcap_n for both planning and sizing and so was never inconsistent. orki_run's single-core run_loop is reached only when colsplit declines — and for fp16 that means a K wide enough that the slice count blows the doorbell chain cap. In other words: only the output-head shapes, and nothing else we run. Every "passing" shape in the report passed because it took a different scheduler.

Fix

Both passes now call one function, orki_run_loop_chunk(dt, Kp, Nc, RB) in src/npu.c. A sizing pass carrying its own copy of the rule is a sizing pass that will disagree with the emitter after the next measurement, which is precisely what this was.

Three further changes fall out of it:

• An extent guard, orki_check_extent (src/npu/core/submit.c), at all four emit sites in run_loop and run_fullk_dec. orki_validate_regcmd checks that base addresses land inside a known buffer; it cannot see the extent, which is the hole this fell through. The guard is called before the host gather, not just before the submit, because a short Af is a host heap overflow on the gather line and a short Cc is the NPU hang. Any future drift is now a refusal, not a power cycle.
• The int8 full-K branch's Cc is now sized explicitly. It writes rows*Nc int32 and only maxaf was ever bumped for it. It has never overrun because the K-sliced loop happens to demand at least as much at every production K — luck, not design, and luck a new SoC's K-slice would end.
• ORK_F16_HISCHED is gone. It was read only by the sizing pass, so its sole effect was to make sizing disagree with the emitter (which has no such knob). Nothing else referenced it.

Results

examples/test_f16_wide_k.c, in make test. Error is scored against sum|a_k*b_k|, not against the result: at K=248320 a random-sign dot product cancels to ~sqrt(K) of its accumulated magnitude, so a result-relative bar measures the cancellation and not the matmul (it rejects the known-good K=4096 sched=0 case at 1.3e-5). A deliberately-wrong control scores 3.3e-1 and must.

shape	before	after
M=1024 K=248320 N=1024 — head dX	HANG, power cycle	relerr 2.098e-09
M=248320 K=1024 N=1024 — head dW	never run (board wedged first)	relerr 1.005e-08
M=1024 K=1024 N=1024 single-core	would overrun Cc by 196608 B	relerr 3.143e-08
M=1024 K=2560 N=1024 (512 tail)	would overrun Cc by 131072 B	relerr 1.198e-08
M=1024 K=2304 N=1024 (256 tail)	ok	relerr 1.596e-08
M=512 K=2560 N=16 (narrow N)	ok	relerr 1.444e-08
M=1024 K=4096 N=1024 (sched=0)	ok	relerr 9.356e-09
control (reference × 1.5)	—	3.333e-01, FAILS as required

ORK_RL_DBG=1 confirms the emitted geometry and that the r103 cap still binds:

Kp=1024 sched=1 chunk=176 Nc=1024 Afsz=360448 Ccsz=720896     <- was Cc 524288
Kp=512  sched=1 chunk=352 Nc=1024 Afsz=360448 Ccsz=1441792    <- was Cc 1310720, the hang
Kp=512  sched=1 chunk=320 Nc=16                               <- r103 narrow-N cap, still binding
Kp=256  sched=1 chunk=704 Nc=1024
Kp=2048 sched=0 chunk=8   Nc=1024

Full make test: ALL TESTS PASSED, tests/sbc_attest.txt refreshed. NPU anomaly counters after the head-shape runs: unpow 0, nojob 0, no new blocked_slow.

The two full-size head shapes are opt-in (ORK_F16_HEAD_FULL=1) because they need ~1 GiB and ~2.5 GiB; the default cases reproduce the identical defect in seconds.

Lesson

The one worth carrying: a cap that is enforced in two places is a cap that will be enforced in one. r103 fixed the emitted chunk and left the buffer sizing behind it; the fix held for a day and the next shape to reach that code path wedged the board anyway. When a measured envelope moves, grep for every consumer of the quantity it bounds — and prefer a single function over agreement.

The secondary one: COMMITTED, NO IRQ now has a third cause on record, alongside the page-table mismatch and the dropped interrupt under power-down (see Vendor-Kernel-Behaviour): a program whose write window runs past its staging buffer. Job accounting is clean in this case — unpow 0, nojob 0, no NOT dispatching — which distinguishes it from the other two.