Exp 2026 07 12 Matmul Shape Boundaries - oRKLLM/ork-driver GitHub Wiki

Exp-2026-07-12 β€” Matmul shape boundaries: the small-K 0x1040 zero-output

Commit: ork-driver feat/ssm-npu Β· src/npu.c (ork_bmm_fp16_fused sched fix) Β· probe tools/f16_shape_probe.c (sweep + ork_f16_fuzz_* register bisect) Β· reproducer tools/ssd_fusedchain_probe.c. Board RK3588 10.3.0.236.

TL;DR

Bare synth()/synth_i8() with sched=1 zeros (or garbles) the output below a K threshold β€” fp16 at K < 96 (i.e. K∈{32,64}), int8 at K < 256. The single load-bearing register is 0x201:0x1040 (the K-reduction schedule): the sched=1 formula base = 177 βˆ’ 15Β·(K/S βˆ’ 1) (S=256 fp16 / 512 int8) overshoots the template default 0xB1 (177) as K drops (0xBC@K64, 0xBE@K32 fp16), selecting an invalid K-reduction atom β†’ the MAC array emits zeros. The correct paths (ork_mm_run single-core npu.c, mcworker) never hit this because they compute sched per shape and use sched=0 (which leaves the template 0x1040) below the schedule's valid band. The fused batched-GEMM primitive hardcoded sched=1 and so broke at the SSD-scan's K=64 stages. Fix: match the run-path sched predicate. Validated bit-exact across the scan shapes.

The symptom

ork_bmm_fp16_fused (chains nb real fp16 matmuls into one PC-chained submit for the on-NPU SSD/Mamba-2 scan) produced zero output at K=64 where the per-op primitive ork_bmm_fp16 is correct:

shape (M,K,N) ork_bmm_fp16_fused (was) ork_bmm_fp16 (per-op)
64, 128, 64 rel-L2 5.4e-8 βœ… βœ…
64, 64, 128 rel-L2 1.000 ❌ zero βœ…
64, 64, 64 rel-L2 1.000 ❌ zero βœ…

Root cause (code + silicon)

synth()/synth_i8() take a sched flag. It toggles only two registers: 0x1010 (row/CBUF hint) and 0x1040 (K-reduction schedule). At M=64,K=64 0x1010 is identical either way (16Β·(mc+1)), so only 0x1040 differs:

  • sched=1, K=64 fp16: scale=64/256=0.25, base=(int)(177+11.25)=188=0xBC β†’ 0x1040=0xBC.
  • sched=0: leaves the REGCMD template default 0x1040=0xB1 (regcmd_array_4x32x16.h word 0).

The correct fp16 paths compute sched = (K pow2) && K∈[128,2048) (npu.c single-core run + mcworker), so at K=64 they use sched=0 β†’ the template 0xB1 β†’ correct. The broken paths hardcode sched=1: ork_npu_probe_f16_mm, ork_ssd_probe_rawmm_f16, ork_ssd_probe_fusedmm_f16, ork_bmm_fp16_fused.

Register bisect on silicon (f16_shape_probe 1, ork_f16_fuzz_add(0x201,0x1040,Β·)), M=64,K=64,N=64:

0x1040 rel-L2 verdict
0xBC (sched=1 formula) 1.000 ZERO-OUT
0xB1 (template default) 3.8e-8 correct

Overriding only 0x1040 flips zero→bit-exact. Causation confirmed.

Why the formula overshoots: base = 0xB1 βˆ’ 15Β·(2^lg βˆ’ 1) with lg = log2(K/256) is calibrated for the prefill band (Kβ‰₯256 fp16 / Kβ‰₯512 int8, where 2^lg β‰₯ 1). Extrapolated below the reference K, 2^lg < 1 makes the correction term negative, pushing base above the template 0xB1. The K-reduction schedule value is monotonic in K (0xA2@K512 β†’ 0xB1 template β†’ 0xBC@K64); a value too large for the actual K selects a reduction atom the MAC array can't satisfy for that contraction length β†’ zero output.

The boundary matrix (bare-synth, sched=1, RK3588)

fp16 (f16_shape_probe 0, M∈{1,8,64} Γ— K∈{32..512} Γ— N∈{16..256}) β€” the boundary is purely K-driven, N- and M-independent:

K fp16 sched=1 note
32 ZERO-OUT (M=1) / wrong (M>1) 0x1040=0xBE
64 ZERO-OUT / wrong 0x1040=0xBC
96 correct first valid; 0x1040=0xBA
128–512 correct prefill band

(M=1 zeros; M>1 garbles β€” same broken schedule, the M=1 all-zero input just makes it read as pure zero.)

int8 (f16_shape_probe 3, M=64,N=64) β€” same mechanism, wider band (formula scales by K/512):

K int8 sched=1
32, 64, 96, 128 ZERO-OUT
256, 512, 1024 correct

The int8 K=64 case is fully wrong at every tested N (32…512) β€” i.e. it is the K-schedule bug, not an N-width effect. This reconciles the earlier FLOOR_DECOMP note ("raw-synth_i8 at M=64,K=64,N=1024 computes only 512 of 1024 cols"): the primary defect is the small-K 0x1040 schedule; any partial-N symptom at N=1024 is that compounded with output-width/N-tiling, and routing through the run path's config fixes both.

0x1040 is a K-AND-rows schedule, not a constant. Forcing the template 0xB1 at large K breaks the other direction: at M=64 the 0xB1 override goes partial-rows at Kβ‰₯384 (f16_shape_probe 2: K=384 rel-L2 0.85, K=512 0.99). This is why the run path selects 0x1040 per (K, mc-group) rather than pinning one value β€” and why the fix is "use the run-path sched predicate," not "always use the template."

Supported-shape matrix per synth path

path fp16 valid region int8 valid region notes
bare synth/synth_i8 (sched=1) K β‰₯ 96 K β‰₯ 256 raw probes; small-K zeros
ork_mm_run / mcworker all K (per-shape sched + M-tile) all K production; sched=1 only for pow2 K∈[128,2048) fp16 / K∈{512,1024} int8
ork_bmm_fp16 (β†’ ork_mm_run) all K β€” correct at K=64 (goes through run)
ork_bmm_fp16_fused (chain) K β‰₯ 96 (before) β†’ all scan K (after fix) n/a single M-tile (mc=M ≀ K-dependent chunk)

The fix

ork_bmm_fp16_fused now computes sched exactly like the run path instead of hardcoding 1:

int sched = (K&(K-1))==0 && K>=128 && K<2048;   /* K<96 -> sched=0 -> template 0x1040 (correct) */
synth(rc, M, K, N, ..., sched, CBUF);

Validated (f16_fusedchain_check, board, fixed npu.c):

nb, M, K, N rel-L2 verdict
8, 64, 32, 64 0.0 correct (was ZERO)
8, 64, 64, 64 4.4e-8 correct (was 1.0)
8, 64, 64, 128 4.3e-8 correct (was 1.0)
4, 64, 96, 64 5.0e-8 correct
8, 64, 128, 64 5.4e-8 correct (unchanged)
8, 64, 128, 128 5.4e-8 correct (unchanged)

Board healthy throughout (errno=110=0; the dmesg soft reset lines are the probes' deliberate per-call ACT_RESET on dtype entry, not wedges β€” gate wedge-detection on errno=110, not dmesg).

Guard audit (include/ork_npu.h)

The declared shape rules (K%32, N%16 fp16 / N%32 int8) are correct but too loose on the low-K end for any raw-synth caller: they admit K=32/64, which the raw path miscomputes. Production callers are safe (the run path handles low K); the constraint belongs on the raw-synth-based primitives. Documented here + in the regcmd ISA Reference rather than tightening the public guard (which would wrongly reject the valid ork_bmm_fp16/run K=32/64 path).

Method / reproduce

# on board, fresh checkout ~/ork-driver-ssm, branch feat/ssm-npu
cc -O2 -Wall -Iinclude -Isrc -pthread tools/f16_shape_probe.c src/npu.c src/soc.c \
   src/soc/rk3588.c src/soc/rk3576.c src/neon_activations.c -o f16_shape_probe -lm
sudo env ORK_MM_TIMEOUT=3000 ./f16_shape_probe 1   # causation: 0x1040=0xB1 fixes K=64
sudo env ORK_MM_TIMEOUT=3000 ./f16_shape_probe 0   # fp16 KΓ—NΓ—M sweep
sudo env ORK_MM_TIMEOUT=3000 ./f16_shape_probe 3   # int8 K sweep
⚠️ **GitHub.com Fallback** ⚠️