M-fold schedule RE + the CBUF-bank-capacity mechanism (why a single submit reduces a partial K)

Board work on RK3588 (10.3.0.236). This entry reverse-engineers rkllm's M-folded int8 matmul schedule (the per-program-faster int8 path we captured), reconciles it against the mainline accel/rocket register names, and — most importantly — decodes the mechanism behind a puzzling result (a single submit reducing only part of K) that turns out to name a whole class of registers we had mislabeled.

TL;DR

• rkllm's faster int8 program folds M into the CNA WIDTH dimension and uses an NC1HWC2 feature layout (C2=16). We reverse-engineered the full register delta vs ork's synth_i8 (confirmed at M=2 and M=24) and built synth_i8_mfold, which reproduces rkllm's captured regcmd bit-for-bit.
• Trying to run it as a single task (ork emits one task; rkllm emits a multi-task chain) computes a deterministic but partial result: all-ones A·W at M=16/K=3584 gives a uniform 2400, not 3584. This is not corruption — it is exactly 2400 = 150 of the 224 input channel-surfaces.
• Root cause, via rocket_registers.h: the registers we'd been calling a "schedule" are the CBUF bank/entry allocation. 0x1040 = CNA_CBUF_CON0 (DATA_BANK/WEIGHT_BANK/FC_DATA_BANK); 0x1044 = CNA_CBUF_CON1 (DATA_ENTRIES). A single submit reduces exactly DATA_BANK × bank_capacity contraction elements. mfold allocates DATA_BANK=3 → ~2400.
• Consequence: a single submit is correct only when the bank capacity equals K. capacity < K truncates (completes, partial); capacity > K over-runs (stalls). The capacity steps (~2400/3200/4000 at M=16) never hit K=3584, and raising DATA_BANK steals the ~11 WEIGHT_BANKs the weight needs — the fixed 12-bank trade-off that forces rkllm's K-slice chain. mfold's ~1.6×/program advantage therefore requires the accumulation chain; it is not a standalone kernel.

The M-fold register delta (vs synth_i8, rocket-named)

Confirmed by diffing ork's synth_i8 output against rkllm's captured regcmd at M=2 and M=24 (same K=3584, N=1216), then rebuilt as synth_i8_mfold (a delta on the synth_i8 baseline):

reg	rocket name	ork synth_i8	mfold (rkllm)	meaning
0x1020	CNA_DATA_SIZE0	(1<<16)|M (W=1,H=M)	(M<<16)|1 (W=M,H=1)	M folded into WIDTH
0x1084	CNA_DATA_SIZE0 (mirror)	(1<<16)|M	(M<<16)|1	ditto
0x1028	CNA batch/atomics	1	M
0x100c	CNA_CONV_CON1	0x20000000 (int32-out)	0x20000000 (int32-out)	(int8-out capture had 0)
0x1010	CNA_CONV_CON2	16·rows (grains)	0x20 (FEATURE_GRAINS=2)
0x1040	CNA_CBUF_CON0	0xb1 (DATA_BANK=1)	0x93 (DATA_BANK=3)	CBUF bank split
0x1044	CNA_CBUF_CON1	ceil(K/64)=56	(K/64)·M=896	DATA_ENTRIES
0x107c	CNA_DMA_CON1	K/16=224	4·M=64	input DMA burst/stride
0x1080	CNA_DMA_CON2	0	(W·C2)/8 (8-byte units)	input surface stride
0x4030/0x4034	DPU_DATA_CUBE_WIDTH/HEIGHT	H=M	W=M-1, H=0	output cube (M in width)
0x405c/0x3014	WDMA size / dst	(M-1)<<16	(M-1)	output width
0x4024	DPU_DST_SURF_STRIDE	linear	M·16·elsz	NC1HWC2 out surface stride
0x4038	DPU_DATA_CUBE_NOTCH_ADDR	((s/4-1)<<16)|(N/4-1)	0	NC1HWC2 (no notch)
0x40c0	DPU_SURFACE_ADD	small const	absolute output IOVA	see wedge note

Feature layout (host packer/de-tiler), NVDLA NC1HWC2, FEATURE_ATOMIC=16: A[M,K] → [C1=⌈K/16⌉][H=1][W=M][C2=16], off(m,k) = (k/16)·M·16 + m·16 + (k%16); C output the same with C=N. Weight layout is unchanged — identical to ork's ork_mm_pack_i8 ([⌈N/32⌉][⌈K/32⌉][32] [32], raw signed int8, no zero-point), matching Mesa rkt_coefs.c.

The wild descriptor: 0x40c0 = DPU_SURFACE_ADD is an absolute output IOVA

rocket_job.c confirms addresses are baked absolute into the regcmd — no kernel relocation. Of the four address-bearing regs in the mfold program, our replay harness patched three (0x1070 A, 0x1110 weight, 0x4020 DPU_DST_BASE_ADDR) but left 0x40c0 at a bogus size value → the WDMA wrote to an unmapped low IOVA → AXI bus hang / hard wedge (unfixable by buffer oversizing — the address isn't near any buffer). Fix: treat 0x40c0 as an output IOVA (patch it to the C base). With that fix the mfold submit fails soft (errno-110 timeout + self-heal) instead of hard-wedging — the difference between "needs a plug power-cycle" and "recoverable". See NPU-Quirks and Exp-2026-07-15 NPU Hard-Wedge Circumstances.

The CBUF-bank-capacity mechanism (the real reason for the partial reduction)

All-ones A·W makes the output equal the accumulation count (independent of layout). Measured on silicon at M=16, K=3584 (all values uniform across the output):

0x1040 (CBUF_CON0)	DATA_BANK	capacity vs K	result
0x93 (mfold)	3	~2400 < 3584	truncates → 2400 (completes)
0x75 (override)	5	~4000 > 3584	over-runs → stall (errno-110)
small M (2/8/12)	2–3	capacity > K	stall

So a single submit reduces exactly DATA_BANK × bank_capacity channels. capacity < K → truncate; capacity > K → stall; capacity == K → correct. ork's normal path instead sets DATA_BANK=1, WEIGHT_BANK=11 (weight-resident) and streams the full K through one data bank — which is why it is correct for any K. The 0x1040 "M-tile schedule" base-slope formula in the ISA reference is exactly this bank allocation: each −0x0f step is (DATA_BANK+1, WEIGHT_BANK−1) (bits [3:0]/[7:4]).

Because the bank-capacity steps never equal an arbitrary K, and stealing banks for data evicts the resident weight, a single submit cannot cover a general K — which is precisely why the vendor chains (each task = one bank-full, accumulate). This is the mechanistic reason mfold is a K-slice-chained kernel, not a standalone matmul.

Status / verdict

• synth_i8_mfold is RE-complete and synth-proven (reproduces rkllm bit-for-bit), board-safe (soft-fail) with the 0x40c0 fix, and its partial-reduction behavior is fully explained by the CBUF bank model.
• Not yet a usable matmul: realizing the ~1.6×/program advantage requires implementing the multi-submit K-slice accumulation chain. That is a large build, and the payoff is speculative — ork already beats rkllm on 7B prefill (see Performance). Banked as a documented, chain-required candidate for a future shape→(schedule, tiling, precision) selector.
• Register-naming correction (feeds the ork-driver register-remap refactor): 0x1040=CBUF_CON0, 0x1044=CBUF_CON1 DATA_ENTRIES, 0x107c=DMA_CON1, 0x1080=DMA_CON2, 0x1024=DATAIN_CHANNEL, 0x1088=FC_DATA_SIZE1.DMA_CHANNEL, 0x40c0=DPU_SURFACE_ADD (output IOVA). See the corrected rows in regcmd ISA Reference.

Code: ork-driver src/npu.c synth_i8_mfold + ork_npu_synth_i8_dump (ORK_MFOLD), ork_npu_replay_i8 (now patches 0x40c0); probes tools/re/{synth_dump,validate_mfold,nc1hwc2_test}.c.

CORRECTION (2026-07-30): 0x40c0 is an elem-size CONFIG (not an IOVA); the m8 tile CHAINS bit-exact

The "0x40c0 = absolute output IOVA / patch it to the C base" reading above is SUPERSEDED. 0x40c0 (DPU_SURFACE_ADD) is an element/surface-size CONFIG: 0x20 int8, 0x80 int32, 0x400 for the M-fold (=128*M at M=8). It is not an address — do not patch it. The captured m8 tile carries 0x40c0=0x400 and replays and chains bit-exact with it preserved untouched; patching it to a C base is what pointed the WDMA at an unmapped low IOVA and hard-wedged. (ork_regs.h RK_DPU_SURFACE_ADD documents this; the earlier "IOVA" label was a misread of a magic value.)

The M>8 chain is bit-exact (ork_npu_mfold_chain_cap)

"At M>8 you chain additional M<=8 tasks" — verified on RK3588 (governors=perf): chain P copies of the captured bit-exact width-8 mfold regcmd (mm_regcmd_m8.txt) as ONE task_number=P submit, shared resident weight, PC-chained at words 216-219. 0-mismatch / maxerr=0 at M=8/16/32/64, no wedge. Two regcmd-prep requirements (the real cause of every prior "chain wedges / errno-110"):

1. Preserve 0x40c0 (config, above) — do not patch it.
2. Neutralize the captured PC-chain descriptor keeping the reg-ids — zero only the VALUE halves of the 0x0010/0x0014 block-0x101 entries (exactly like ork_npu_replay_i8). Zeroing whole words 216-219 drops the 0x0014 next-amount terminator and even a single task hangs.

Open (perf): each M=8 tile carries 0x1040=0xb1 (DATA_BANK=1/WEIGHT_BANK=11) and re-streams the full K*N weight (~397us) every tile -> linear timing (P1 444 / P2 836 / P4 1618 / P8 3182 us) -> ~7x slower than ork-normal at M=228 (which amortizes one weight stream over a big M-tile). The ~1.6x needs weight residency across chained tiles (load-once/reuse; rkllm varies 0x1040 per task) — decode the reuse config offline from a captured rkllm chain; do NOT sweep 0x1040 on-board.

Reinforcement (2026-07-30): word count is uniform — M>8 fails standalone because tiles inherit chain state

Two confirmations (decode-tool anchored, tools/re/regcmd_decode + the capture task descriptors) that strengthen "M=8 is the largest STANDALONE full-K tile; M>8 needs the chain":

• Every matmul task is a uniform 232-word / 108-reg regcmd (regcfg_amount=108 for all 4395 matmul tasks in the capture; the 40/56-word blocks are tiny non-matmul setup tasks). So the M>8 standalone failure is NOT a word-count / program-length difference — an M=36 tile is the same size as an M=8 tile. (An earlier 364-word "M=36 tile" was an extraction bug: extract_tile.py over-ran the block boundary; reading exactly the declared header word count fixes it, and the clean M=36 tile decodes to the same 116 reg-writes as M=8.)
• A cleanly-extracted M=36 tile STILL errno-110s as a standalone task_number=1 submit — so the failure is the tile schedule/CBUF config, not truncation. Tellingly, the M=8 tile sets 0x40c0=0x400 (self-contained) while the M=36 tile omits 0x40c0 (decode shows it absent) — i.e. the larger-M tile inherits register state from earlier chain tasks and cannot run in isolation. Concrete evidence that big-M tiles are chain-context-dependent (they rely on state the earlier tasks left in the register/CBUF banks), consistent with the K-slice-accumulate reading above.

Net: the M=8-tile weight-resident chain (ork_npu_mfold_chain_cap) is bit-exact but re-streams the weight per tile; the 1.6x still requires implementing the big-M K-slice-accumulate chain (which supplies the inherited state), not a drop-in of a captured big-M tile.

CRACKED (2026-07-30): 0x1080 is a SIGNED stride — the "0x0fffffe8 sentinel" was never real

Captured both 0x1080 (CNA_DMA_CON2) variants across the rkllm-7B dump (4114 writes) and settled what its w1 upper bits mean: they are two's-complement sign extension, 0xfff iff the 16-bit value is negative and 0x000 iff positive — 0/4114 violations. So:

• 0x1080 is a signed 28-bit feature surface stride, not a stride+mode pair and not a sentinel.
• The infamous 0x0fffffe8 is simply -24 (= -3*M at M=8) in two's complement, read as unsigned. A small negative stride mistaken for a ~268 MB stride is what looked like an "inherit-CBUF / OOB sentinel" and misled this RE for weeks. There is no chain-inherit marker here.
• Per-M values are a planner-chosen set (-3*M is the common small-M value; 2160/M appears at large M), not a closed formula.

Tooling: tools/re/regcmd_decode now decodes register values by width (16-bit = w0>>16 with w1&0xffff a separate field; 32-bit for addresses/packed regs) and marks 0x1080 signed, printing e.g. 0x0fffffe8 (signed -24). The M=8 and M=36 tiles now decode with 0 unknown-value / 0 out-of-mask. This supersedes every "0x1080 sentinel / 0x0fffffe8 = ~2GB stride" note on this page and in NPU-Quirks — the value was always a signed stride.

Why the strides: 0x1080 is derived from the burst 0x107c (verified on the dump)

The per-tile 0x1080 values are not independent planner choices — they are a mechanical function of the feature-DMA burst length and the fold width, verified across the rkllm-7B capture:

• 0x1080 = ±(0x107c − M) — 82%+ of 3887 writes match exactly (residual = per-task multi-write pairing + the burst=4M subcase). It is a surface-stride correction: after a burst of 0x107c, rewind/advance by (burst − M) to land on the next NC1HWC2 fold surface. Sign = traversal direction (rewind − / advance +).
• 0x107c (feature-DMA line burst) = 4·M is the dominant value for every M (top-count at M=1,2,4,6,8,10,12,14,16,24,32), hard-capped at 128 (M=36: 4·36=144 → 96/128; max 0x107c observed = 128, never exceeds; M=20 is a lone anomaly at 128). Minority per-M values are secondary in-task writes / tiling context.
• Therefore the famous −3·M stride is simply the burst=4M case: 0x1080 = M − 4M = −3M (a rewind).

Takeaway: 0x107c/0x1080 are a coupled burst+correction pair keyed to M, not two free planner outputs. The only free-ish knob on this DMA path is the burst length (~4·M, a bandwidth choice bounded by the 128 cap); the stride follows. This meaningfully shrinks the "planner is an opaque multi-variable heuristic" conclusion for the CDMA registers (relevant to the register-naming roadmap, #40).

Burst rule for the anomalies (M=20, M=36): per-output-position, not 4·M

Isolating the clean 108-reg tile body (words 0-215; the earlier per-M "tail" was next-task bleed) gives a single dominant 0x107c per M:

• Matched single-pass tiles: 0x107c = min(4·M, 128) for M∈{1,2,4,6,8,10,12,14,16,24} and M=32 (=128). Valid burst set is {4, 8·k (k≤8), 96, 128} — granularity jumps to 32 above 64 (no 72/80/88/…), hard cap 128.
• Large accumulate tiles (M=20, M=36 — the ones that can't run standalone): the burst is per-sub-block, tracking output position, not 4·M. For M=36, 0x107c correlates exactly with the A/C offset: 40 @ 0xa000, 96 @ 0xa240, 128 @ 0xa480 — offsets stepping by 0x240 = 576 = M·16 (one fold line), the burst growing 40→96→128 across the tile's three positions (a progressive per-position DMA prefetch), capped at 128. M=20 → 128 uniformly (its natural 4·20 = 80 is invalid in the coarse {96,128} region, so it lands at the cap).

So the previously-"opaque multivariable planner" burst is pinned: the free variable for the anomalous tiles is the sub-block output position, and 0x1080 = ±(0x107c − M) still follows per task. Net for the CDMA schedule (0x107c/0x1080): matched tiles are 4·M / −3·M; accumulate tiles are position-driven prefetch bounded by the 128 burst cap.