Outcome: fixed. The hang was the BCHAIN H (rows-per-weight-stream) tiling parameter exceeding a measured, K-dependent ceiling. Above that ceiling the NPU accepts the submit and never starts it. orki_i4_hcap (src/npu/i4/chain.c) now encodes the measured envelope. Full native-W4A4 prefill of qwen3.5-0.8B completes; make test ALL PASSED.

Native W4A4 prefill had been logged as an intermittent NPU wedge since 2026-08-07 (I4_GROUPED_WEDGE_WIP.md). It is none of those things:

• not a wedge — it is a bounded rc=-1 after ~15 s; the board stays healthy and the render node is released. The RKNPU: soft reset burst in dmesg is the driver's OWN recover loop (6 resets by design), not a fault. Same lesson as NPU-Quirks "gate wedge-detect on errno=110, not on the dmesg string".
• not intermittent — deterministic per (K,H): 5/5 identical outcomes.
• not full-model-specific — reproduces standalone in ~2 minutes with no model loaded.
• not the grouped path — the 2026-08-07 analysis (M-chunk swarm of ~1806 micro-submits, missing recover loop in ork_dyn_grouped_end) was about mul_mat_i4, which ggml-ork no longer dispatches to. The surviving per-channel mul_mat_i4_hadamard fails for an unrelated reason.

make i4_hcap_probe
sudo tools/util/npu_guard.sh -- env ORK_MM_TIMEOUT=2000 ./i4_hcap_probe 3072 1024 32
# before the fix: rc=-1 after ~15 s

Signature, identical standalone and in-model:

[mc-recover] doorbell MISS -> ACT_RESET + resubmit | dom=0 S=3 nc=3 esz=2     (x6)
[dyn_end] S=3 mc=1 done=0/3 last=-1 elapsed=3000103us INCOMPLETE(timeout/term)
[MC-DIAG] nc=3 submit_rc=[0 0 0 ]
[NPU-DUMP] ... hw_elapse=0 (0=>no work) | task int_status[0..3]=0x0 0x0 0x0 0x0

All three cores' submits return 0 (accepted); no output sentinel lands; hw_elapse=0 and int_status=0 say the hardware never began the job. In-model the first failing op was blk.0.ffn_down, K=3584 — everything before it (K=1024, K=1024, K=2048) succeeded.

H only re-tiles M, so every valid H must produce a BIT-IDENTICAL result. Sweep H upward and take the largest value whose checksum still agrees with the cluster below it.

Two earlier gradings both produced wrong tables:

• Grading on rc. Above the ceiling the hardware picks one of two behaviours, and only one is visible in rc:

  mode	signature	example
  silent miscompute	rc=0, DIFFERENT checksum	K=2048 H≥9; K=1024 H=5..8
  doorbell miss	rc=-1 after ~15 s	K=3072 H≥5; K=1024 H≤4

  An rc-only scan reported K=2048 H=2..20 all "ok" when in fact H≥9 miscomputes. Checksums at K=2048: H=2/4/6/8 → a50a77267779908e; H=9 → 9ae15d56…, H=10 → d812ef4b…, H=12 → 956dfda9….

• Grading on timing. Catches the self-heal (a timed-out submit retried ~800× slower) but is blind to the silent miscompute. Necessary, not sufficient.

Note the valid-H set is sparse and not an interval: at K=1024, only {10,12,16} are correct — H≤4 doorbell-miss, H=5..8 miscompute, H≥17 miscompute. H is not a tuning knob; the formula is meant to produce the single largest valid value.

Twenty-five points, and no closed form fits them. ceil(16384/K) — what shipped before this work — is wrong at eight of them, in both directions: it overshoots at 1792, 2560, 2816, 3072, 3584 and 5120 (miscompute, hang, or the non-determinism below) and undershoots at 6144. floor is wrong at 768, 1280, 1536, 4608 and 6144. 16384/next_pow2(K) fits 96..256 exactly and dies at 768 and 6144. CBUF bank-containment cannot explain K=1024 H=2, which fits the budget with room to spare and still doorbell-misses. An unmeasured K falls back to floor(12288/K) — 0.75x the naive value, the largest margin any measured point needed — which is an empirical safety factor, not a bound.

Below K=256 the envelope goes FLAT — 128 for K≤128, 64 for 128<K<256 — with no relation to 16384/K, which wants 512 at K=32. Boundaries pinned exactly: H=128 ok / 129 bad at K≤128; H=64 ok / 65 bad at K=160/192/224. So the naive rule is wrong at both ends of the range. The 64 clamp that previously covered K<256 was safe but load-bearing, not decorative: at K=160 the floor(12288/K) fallback computes 76 where the true ceiling is 64, and only the clamp caught it. It also cost 2x throughput at K≤128, now recovered.

The table IS the rule. An earlier revision of this page, with only eleven K measured, read the deviations as a contiguous band 2048<K<4096 and shipped H = in_band ? 4 : ceil(16384/K). Measuring the two gaps that rule was interpolating across killed it within the hour:

• K=2560 → 7 (H=2..7 all agree, six-way). ceil is right here; the band rule forced 4 — correct but 43% conservative.
• K=5120 → 3 (H=2,3 agree; H=4 differs). ceil gives 4, and the band rule shipped 4 → a silent miscompute, outside the band entirely.

So the deviations are not a band, and interpolation is unsafe in BOTH directions. The fractional part does not decide it either: K=768 and K=3072 both have frac .33 and round opposite ways (22 vs 4). The shipped implementation is now an exact 13-entry table, with unmeasured K falling back to floor(12288/K) — 0.75x the naive 16384/K, 0.75 being the largest margin any measured point needed (K=3072: 4 vs 5.33). That is an empirical safety factor, not a guarantee.

M-independent: K=768 re-measured at M=64 gives the same 22 as at M=32, so the table transfers across M. (This also rules out M as the explanation for the 2026-08-20 table's disagreement.)

No closed form fits all twenty points. Candidates tried and killed by a specific point:

The envelope is therefore encoded as measurement, in the same spirit as orki_f16_mcap — see Exp-2026-08-20-SDP-and-Int4-Envelopes, where the fp16 envelope was likewise non-monotonic and had to be scanned rather than derived.

bfefc64 ("H ceiling is CEIL(16384/K), not floor") pinned six K at integer resolution — 256, 512, 768, 1536, 2048, 4096 — and 3072/3584 were not among them. Under the previous floor rule K=3584 would have taken H=4 and worked; CEIL gives 5 and breaks it. K=3072 is not a regression: floor gives 5 there too and 5 also fails; the true value is 4, so that shape was already broken.

The generalisable error: a rule was extended to the whole domain from six samples that happened to share a property. The 2026-08-20 lesson was "probe between the rungs"; the 2026-08-21 lesson is "probe between the samples" — the six K left a 2048→4096 gap wide enough to hide the deviation. Note this bit twice in one day: the eleven-point band rule that replaced CEIL was itself an interpolation across the unmeasured 2560 and 5120, and 5120 broke it. Each time the fix was to measure the gap, not to reason about it.

Measuring the small-K range cost two reboots, and the cause is worth recording because it very likely explains the historical "intermittent W4A4 wedge" reports.

Running six consecutive over-ceiling configs (H=160/192/256 at K=32 and K=128) left core 0 pinned at 100% with no process holding it — an orphaned job in hardware, GUARD BLOCKED, recoverable only by sudo reboot. Every one of those six returned rc=0 individually. The wedge is cumulative, not per-config.

Proof it is the sequence and not the shape: K=224 H=64 M=256 appeared to wedge in that run, but re-run in isolation on a freshly rebooted board it passes cleanly (rc=0, NPU free) — and it is in fact exactly the ceiling there.

So a scan MUST stop at the first checksum mismatch rather than marching through the bad region, and should re-check npu_guard between K. Both scans here were rewritten that way and completed without incident; the one that marched wedged twice. This is also why "wedge" reports from full-model runs were intermittent: a model that hits several over-ceiling matmuls in a forward accumulates the same state, while any single shape probed standalone looks fine.

pack.c K-slices at 8192 (w->sliced, built whenever Sn>1 || K>8192), so a wide-K int4 shape never reaches BCHAIN with K>8192 — it arrives as ≤8192 sub-tiles, each hitting the table separately. Verified with ORK_I4_DIAG:

So the ≤8192 table is the complete answer. The real residual risk is the slice remainder, which can be any K and lands on the fallback. Checked against five non-table K — the fallback held at all five, and K=2816 has zero margin (fallback 4, true ceiling 4; naive ceil would have shipped 5 and failed):

All five are now table entries (measured beats conservative), and all five were separately confirmed stable as a slice remainder at K=8192+rem, 5/5 each — a distinct check from the standalone ceiling, for the reason in the next section.

Found while measuring the above; pre-existing, not fixed.

K=18944 — the real 7B ffn_down — returns a different result on ~40% of runs. So do 10752 (= 8192+2560) and 27136 (= 3×8192+2560), so it tracks the remainder, not the slice count. Ten runs at 105438e and ten at the current table gave the same ~40% rate, so it is not a regression from the envelope work. The majority value is the correct one.

What it is not:

• not the slice count — 20480 (3 slices, rem 4096) and 24576 (3 slices, rem 8192) are 8/8 stable;
• not a partial-last-group effect (H ∤ M) — K=18944 at M=28, where 7 does divide 28, is still unstable, and K=9728 (rem 1536, H=11, which does not divide 32) is 6/6 stable;
• not H=7 in general — K=10496 (rem 2304, also H=7) is 5/5 stable;
• not the H value being wrong — H=7 is the verified standalone ceiling for K=2560 (six-way agreeing cluster), and remainders 1024/1536/2048/4096/5120/8192 are all 8/8 stable.

What pins it: forcing ORK_I4_H=2 makes 10752 and 18944 6/6 stable on the majority value. So it is H=7 specifically, and only inside the slice int32 K-accumulate.

Per-sub-tile partial checksums settled it in one run. The two K=8192 tiles are bit-identical on every run; only the K=2560 tile varies:

[slice] ki=0 subK=8192 part=582cc6e8606eb32d   <- same every run
[slice] ki=1 subK=8192 part=318d5ebbce84d215   <- same every run
[slice] ki=2 subK=2560 part=c477629c… / 746b5811… / 9b91edef…   <- varies

So it is the BCHAIN submit itself, upstream of the int32 accumulate. Each remaining candidate was then killed by a measurement rather than an argument:

An earlier revision of this page claimed the mechanism was a kernel job timeout whose retry re-accumulated onto BCHAIN's int16 output. That was wrong, and it is worth recording how it went wrong because it is the same error this page warns about twice already.

It came from a single 8-run sample in which the corrupt runs also showed +2 dmesg soft resets and 5-7 s wall times. The resets in fact arrived in a block (runs 4-8) with the corruption inside it, and runs 4, 7 and 8 were +2 and correct. A second sample settles it: corrupt runs at the baseline +1 reset, fast, with submit-fail=0 and MC-RECOVER=0. A correlation in one sample, promoted to a mechanism.

Two further checks kill it outright:

• No retry is involved. orki_mc_recover_resubmit never fires on the corrupt runs, and the int4 branch of it already full-surface-fills the output with ORK_DYN_SENT16 before resubmitting — so even when it does fire it is already idempotent. (The int8/fp16 branch only seeds sentinel cells; that asymmetry is real but is not this bug.)
• The submit is not slow or stuck. With ORK_DYN_TRACE, H=7 standalone reports [dyn_end] S=3 mc=1 done=3/3 last=2 elapsed=496us all-done. The ~1.6-2.3 s per run is host-side, and it is not our submit timeout either — wall time is flat across ORK_MM_TIMEOUT 500/1000/2000/4000.

Mechanism: still unknown. Ruled out, each by a measurement rather than an argument:

Per-sub-tile checksums put it in the K=2560 BCHAIN call alone. Something about that geometry makes the NPU produce different results depending on what ran before it, and the remaining suspect is CBUF/warm state — unproven.

What is solid, and what actually justifies the fix: H>4 at K=2560 is not a correctness/speed trade-off, it is worse on both axes. Standalone, 20 timed iterations, repeated:

~2000x slower, with the stall on the host side. So there is no portion of H>4 that remains usable at this K, and lowering it to 4 costs nothing at all — which corrects the "costs standalone throughput" framing from when the fix landed.

The rate falls smoothly with H, the signature of a margin rather than a wrong constant:

K=2560: 7 → 4. Four is the largest fully stable value and is exactly where 2816/3072/3584/4096 already sit — 2560 was the odd one out. The standalone result is unchanged (ea4360d5; H=4 was inside the agreeing cluster), so this costs standalone throughput at that one K and buys a correct wide-K path.

Validate every entry TWICE: standalone AND as a slice remainder (K=8192+rem). These are different tests and standalone is the weaker one. K=2560 agreed six ways standalone at H=7 and was still wrong in the slice path. Repetition matters as much as the checksum compare — a single agreeing run passes H=7 about half the time. All 25 entries are now confirmed stable as remainders (6 runs each), including the seven never previously checked that way (256/512/768/1024/3072/3584/ 6144).

test_slice_rescue covered "i4-wideK (2 K-slice-accum)" but no remainder that triggers this, which is why it went unnoticed. Added case i4-rem2560: K=18944 (2x8192+2560), N=1024, M=32, run 5 times with every run required bit-identical to the first and equal to the golden.

The repetition is the test. With the bug present a single golden compare passes about half the time, so a one-shot case would have been a coin flip; reintroducing H=7 fails the new case 3/3 with run N differs from run 0: … NON-DETERMINISTIC FAIL. N matters too — N=256 does not reproduce at all, N>=512 does — and so does the number of preceding slices (K=10752, one preceding tile, fails only ~1 in 6). It is also implicitly a perf guard, since a reintroduction stalls for seconds per run. Adds ~2 s to make test (48 s total).

Two separate open questions, and they may not share an answer:

1. What corrupts the K=2560 sub-tile at H>4. Everything in the table above is ruled out. It is not the activation budget either: H*K at 2560/7 is 17920 against the 16384 the formula is built on, but 1280/13 (16640), 1536/11 (16896) and 4608/4 (18432) are all over it too and all stable as remainders. The remaining suspect is CBUF/warm state carried from the preceding submit — the slice context differs from standalone in exactly that respect — but it is unproven.
2. Where the ~2 s per run goes. The submit reports done=3/3 in 496 µs and the wall time does not track ORK_MM_TIMEOUT, so the stall is host-side — most likely the O(nout) sentinel seed / de-tile in orki_bch_db_cells, whose cost depends on the (H, NG, Wb) cell pattern. Worth profiling: it is a ~2000x host-side cliff that a geometry change can walk into silently.

Not an open item: making the resubmit idempotent. The int4 branch of orki_mc_recover_resubmit already full-surface-fills the output with ORK_DYN_SENT16 before resubmitting, so it is idempotent today, and no resubmit fires on these runs in any case. The int8/fp16 branch of the same function only seeds sentinel cells rather than clean-filling — a real asymmetry, but it is on the shared int8/fp16 drain (highest blast radius) with no demonstrated failure behind it, so it wants its own change.

Profiling the host-side ~2 s settled both open questions at once, and the answer is a latent silent-corruption bug in the shared doorbell drain, not anything int4-specific.

Phase timers inside orki_i4_run_bchain_db (K=2560, N=1024, M=32):

2 000 392 µs is exactly the i4batch miss window in dyn_ctl.c (miss_to = h->i4batch ? 2000000.0 : 300000.0). The submit is fine — the completion poll is what burns 2 s. It only does so on repeat submits, which is precisely the "needs a preceding submit" property both symptoms have. Seven submits in one process:

K=18944 (slice tiles 8192/8192/2560), per-sub-tile dyn_end against the final checksum — every wrong run has the 2560 tile in the 2 s window, and no fast run is ever wrong:

With ORK_DYN_TRACE, both corrupt runs print:

[dyn_end] S=3 mc=1 done=2/3 last=2 elapsed=2000258us INCOMPLETE(timeout/term)

done=2/3 — one core never landed — yet last=2. Because:

int ork_dyn_progress(ork_dyn_chain *h) { int hi = -1;
    for (int i = 0; i < h->S; i++) if (ork_dyn_done_i(h, i)) hi = i;
    return hi; }                       /* HIGHEST landed index, NOT a count */

and the caller treats it as completion:

last = ork_dyn_progress(h);
if (last >= h->S - 1 || orki_ork_term) break;   /* "all done" — only tests the LAST task */

So a hole at index 0 or 1 is invisible whenever the final task lands: the drain declares success and de-tiles a buffer one core short. That is the silent corruption.

Three consequences, all confirmed by the traces above:

• the recover loop never fires (it is in the else of that same test) — which is why MC-RECOVER read 0 on every corrupt run and sent this investigation down a blind alley twice;
• the auto-dump never fires either — if (last < h->S - 1 && !h->spin_end) has the same flaw;
• it is not int4-specific. ork_dyn_end is the shared int8/fp16/int4 doorbell drain, so any multi-core chain that loses a non-final task silently returns partial output.

The poll loop already maintains edone[] and the count n. Use the count rather than the index:

int ndone = 0; for (int i = 0; i < h->S; i++) ndone += edone[i];
if (ndone >= h->S || orki_ork_term) break;      /* was: last >= h->S - 1 */
...
if (ndone < h->S && !h->spin_end) { /* diag + auto-dump */ }

In the end the cleaner fix was in ork_dyn_progress itself, because every caller reads it as a frontier, not a maximum — ork_dyn_dump names the stuck op prog+1, ork_dyn_remaining computes P-(p+1), the queue halts at prog+1+HEADROOM. So it should stop at the first hole:

for (int i = 0; i < h->S; i++) { if (!ork_dyn_done_i(h, i)) break; hi = i; }   /* was: if (done) hi = i; */

Identical for a single-core chain (tasks retire in order); correct for multi-core, where they do not.

Result — better than "fail loudly": it repairs the corruption. With the K=2560 H=7 bug deliberately reintroduced, test_slice_rescue now PASSES 3/3 with [MC-RECOVER] ... reset + resubmit in the log, where before the fix it failed 3/3 non- deterministically. The recover machinery was never broken — it was unreachable, sitting behind the same faulty test. This is also why MC-RECOVER=0 on corrupt runs misled this investigation twice: the missing recover was a symptom of the bug, not evidence against it.

Validation (AGENTS shared-drain protocol): make test ALL PASSED at default, ORK_SSM_KEEPWARM=0 and ORK_MIXED_NOTHRASH=1 (env propagation confirmed first — 92 [ork-trace] lines); reset count 51, identical to the pre-change baseline of 51; check-registry + check-attest clean.

The int4 H=4 table entry stays — the two are complementary. The table keeps K=2560 out of the 2 s miss window; this makes a miss recoverable instead of silently wrong wherever one happens.

Note the H=4 table fix is still right and still wanted: it keeps K=2560 out of the 2 s window entirely. The two are complementary — one avoids the stall, the other stops a stall from silently corrupting.

AGENTS' mode-transition protocol names mode_probe as a required gate, but it could not be used for the fix above: it hangs on this board with or without the change. Verified by reverting the fix first, specifically so a pre-existing fault would not be attributed to it — the baseline run stalls partway through the transition matrix (0 failures in what it prints), the wrapping timeout 900 does not reap it, the process ends <defunct>, and dmesg shows:

file:card1 fault:rknpu_gem_fault mmap:drm_gem_mmap read_folio:0x0
rknpu_gem_object_destroy+0x150/0x188
rknpu_gem_free_object+0x34/0x4c

It also took the board down hard twice during this session (no ping, not just SSH — so a kernel-level hang, not the usual NPU wedge that leaves SSH alive), each needing a Rock 5B Plug power cycle and a governor re-pin. A make test immediately afterwards also died, but a clean run on a freshly booted board passed in 40 s, so that was residue from the GEM fault rather than a second cause.

This means the mode-transition protocol currently has a broken instrument. Worth its own investigation; until then, changes to the drain have to lean on make test across the keep-warm knobs plus the reset-count diff.

• Unmeasured K are a known risk, and the miscompute mode is silent. Pin any new K with the agreeing-cluster method before trusting it — the floor(12288/K) fallback is an observed margin, not a bound. Twenty-five K are measured (32..8192), which is the whole domain (see below).
• Why the ceiling behaves this way is unexplained. It is not the CBUF data-bank budget (K=1024 H=2 uses 1 KB of an 8 KB budget and still misses), so the small-H doorbell miss in particular has no model.
• loaded 0 / packed 0 on the timed W4A4 run — the i4-native pack is written (102 weights, 153.9 MiB) but not read back into ctx->wcache, so the timed pass re-packs. Separate, pre-existing gap already flagged in I4_GROUPED_WEDGE_WIP.md; native W4A4 prefill remains slow and opt-in.