Experiment — INT4 Multi-M Schedule RE: the 0x1040 K-schedule wall

Date: 2026-07-07 · Status: ⏳ PARTIAL — native multi-M proven bit-exact at the capture K (K=64), but capped at row 0 for production K by an un-cracked K-reduction/CBUF-budget schedule · Path: ork-driver matmul level (RK3588 10.3.0.236) · Tools: tools/i4_multim_probe.c (fixed + extended), examples/i4.c

---

Why (motivation)

W4A4 prefill is submit/weight-DMA-bound: the production int4 path (i4_mcworker, post-Exp-2026-06-20) issues one single-row program per M-row (PC-chained), so the K×N weight is re-streamed from DRAM once per row. The int8 path avoids this with the hardware M-scheduler (reg 0x1040, mg_max*64 rows/submit, weight streamed once) — hence int8 ≈40 t/s vs int4 ≈5 t/s at the same shape. Goal: resurrect the native int4 multi-M (batch mode) that Exp-2026-06-19 claimed, so the weight amortizes over H rows.

Prior art reconciliation (06-19 vs 06-20)

• 06-19 claimed native multi-M works: leave PPU stride 0x405c=0, set M-count mc=2H, read output at stride-2 (logical row m → physical row 2m). Validated only at K=64, N=64 (the captured shape).
• 06-20 reverted to chunk_M=1 calling it "structurally single-row / rows 1..M-1 silently discarded." 06-20 never applied the stride-2 readback — it read contiguous rows, saw the odd physical rows as garbage, and concluded inert. That conclusion was wrong; it just wasn't de-striding.
• NVDLA cross-check (nvdla.org/hw/v1/hwarch.html): batch mode = multiple datasets, one weight load (D_BATCH_NUMBER); per-batch output spacing is a stride reg (D_LINE/SURF/BATCH_STRIDE); CACC accumulates wide and clips to 16-bit (D_CLIP_CFG). So 0x405c=0 selects the HW-default int32-slot step ⇒ stride-2, and the int16 output is a clipped int32 accumulation (exact when it fits; identical per-row vs multi-M).

Method

Fixed the probe's two latent bugs (the same trap 06-20 fell into):

1. ork_npu_probe_i4_mm allocated the output at M*N int16 — stride-2 writes physical rows 0..2(M-1), so it wrote OOB and never read the even rows back. → now allocates/returns 2*M*N.
2. Added an ORK_I4_MREGS bitmask in synth_i4 (which of the int8-ported multi-M regs are applied), plus ORK_I4_1040 / ORK_I4_1010 direct overrides, and a collision-proof per-64-block physical-location map in the probe (finds where each row×block lands, vs. guessing layout formulas).

Gated the whole native path behind ORK_I4_MSCHED (default OFF) in i4_mcworker: one synth_i4(mc=chunk) submit + stride-2 de-tile readback, else the proven per-row PC-chain. Zero regression — ./i4 default is ALL OK; A is tiled per-row-contiguous (ORK_I4_ALAY=1), matching 06-19.

Results

1. 0x1040 (K-reduction schedule) is the decisive register — not an inert set

ORK_I4_MREGS fuzz at M=4/K=64/N=64 (ALAY=1), stride-2 check:

mregs	includes 0x1040?	stride-2
0x05, 0x07, 0x1f	no	MATCH (0 mismatch)
0x2f, 0x37, 0x3f	yes	FAIL (row 0 corrupt too)

0x1040 is bit 0x20. Every config that sets it corrupts; every config without it is bit-exact. The int8 formula value (177 − 15·(K/256−1) ⇒ 188@K64, 72@K2048) is wrong for int4. Captured base = 177 (K-independent-correct for M=1, which is why the M=1 path never overrides it). New default: mregs=0x1f (full int8-ported set minus 0x1040).

2. Native multi-M IS bit-exact — at the capture K only

With mregs=0x1f, ALAY=1, single 64-wide N-block: M=2 and M=4 at K=64 match bit-exact via raw[2m·N + n] (stride-2). Confirms 06-19 and refutes 06-20's "structurally single-row."

3. The wall: row count is capture-K-tuned + multi-block output is 2D

Per-64-block physical-location map (collision-proof), rows actually computed:

K	rows computed (N=64)
32	1 (only row 0)
64	all (4 @ M=4)
128	2
256	1
≥512 (incl. 2048)	1 (only row 0)

Non-monotonic, peaking at K=64 = the captured program's own K. So the batch row-scheduling is baked to the capture and degrades away from it; at production K=2048 only row 0 ever lands. 0x1010 override (→0xffff) does not lift the budget. Multi-N-block (N≥128) output is a 2D surface: at N=128 the block offsets are (b·H + m)·2N with H itself N-dependent (H=2 rows at N=128 vs 4 at N=64).

Verdict

Native int4 multi-M (batch mode) is real and bit-exact at K=64, and the long-standing "structurally single-row" claim is disproven. But it is not yet usable at production K: the 0x1040 K-reduction schedule (and likely a companion CNA/CBUF batch-activation-budget reg) must be re-derived for int4 as a function of (K, mc) — the int8 formula and the captured base both fail for K≠64. This needs the systematic sweep 06-19's (now-deleted) i4_multim_fuzz.c did. Until then, W4A4 stays 1-submit/row (per-row PC-chain, ORK_I4_MSCHED OFF).

Next concrete steps

1. Sweep ORK_I4_1040 (0x1b..0xff) × K ∈ {512,1024,2048} at M=8/N=64, watching the block-map for row count → find the int4 K-schedule law (candidate: it must encode K-tiles and batch rows, unlike int8's mg).
2. Identify the batch activation-cube budget reg (the thing capping rows at K>64) — fuzz the 0x10xx CNA / 0x30xx DPU space with the state-saving/blacklist fuzzer pattern; 0x1010/0x1088 ruled out.
3. Then derive the multi-N-block output surface stride (the (b·H+m)·2N generalization) and wire i4_mcworker to submit wide-N multi-M with a full 2D de-tile.

Artifacts (all committed, gated OFF)

• src/npu.c: synth_i4 mc>1 = ORK_I4_MREGS(def 0x1f)+ORK_I4_1040/ORK_I4_1010 overrides; ork_i4_msched() gate; i4_mcworker native-multi-M branch (stride-2 de-tile) vs per-row; ork_npu_probe_i4_mm 2×-buffer fix.
• tools/i4_multim_probe.c: 2×-buffer, stride-2 + surface layout candidates, per-64-block location map.

Benchmark — batch vs per-row (RK3588, ./i4 ORK_I4_BENCH, warm, 100 reps, DDR governor pinned @ performance)

Matmul-level W4A4 prefill, ORK_I4_MSCHED=1 (batch) vs =0 (per-row PC-chain). CORE batch = whole N-slice fits the weight budget (Ncore·K ≤ 131072), single submit, no sub-slice:

M	K	N	batch (ms)	per-row (ms)	speedup
32	512	256	0.289	0.389	1.35×
48	512	512	0.409	0.645	1.58×
32	2048	64	0.311	0.504	1.62×

Core batch = 1.35–1.62×, governors pinned — the real, validated win (default-on ORK_I4_MSCHED). Below the theoretical 8–16× row-amortization because only weight-DMA shrinks (A re-tile / submit / de-tile costs don't).

N-subslice (wide-N) is NOT a win — a governor cautionary tale

Extending batch to wide-N (Ncore·K > 131072) via N-subslicing (ORK_I4_NSUB) first looked like 1.10–1.19× (K512/K1024, N=2048) — but that bench ran on an unpinned dmc_ondemand boot. Re-run with DDR pinned @ performance, N-subslice is ~neutral (K512 0.948 vs 0.938, K1024 1.483 vs 1.431) and loses at K=2048 (1 block/submit → submit-bound). With fast DDR the weight-DMA saving is small and the extra submits cancel it. So ORK_I4_NSUB ships default OFF (opt-in, correct + bit-exact, just not faster). Lesson: always pin the DDR governor before timing — a parked governor inverted the sign of this result. The real end-to-end prefill lever is elsewhere (CBUF bank re-partition / int4 streaming — see roadmap), not N-subslicing.