Experiment — per-channel int4 (full-K single submit) vs grouped vs int8

Date: 2026-06-13 · Status: ✅ MEASURED — per-channel kills the K/G explosion, but int4 still can't reach int8 speed · Path: ork-driver matmul level (RK3588 10.3.0.236) · Tool: tools/int4_bench.c

Hypothesis

Tier 4b: the grouped W4A4 path is ~30× slower than int8 because per-group scales force K/G submits/matmul (~16 for K=2048, G=128) on a submit-floor-bound NPU. Hypothesis: switching to per-channel int4 (one scale per output channel, like int8 — ork_mm_run_i4, already in the driver as a full-K single submit at the 10752 ceiling) kills that explosion and brings int4 to ~int8 speed.

Method

tools/int4_bench.c times three paths on the same shapes, warm (30 iters), governors pinned (CPU performance 2.4 GHz / DMC performance 2112 MHz):

• pc-i4 — per-channel int4, ork_mm_pack_i4 + ork_mm_run_i4 (full-K single submit, host int32 accumulate, one submit per M-row per the captured M=1 regcmd).
• grp-i4 — grouped int4, ork_mm_run_i4_grouped (G=128 → K/G submits).
• i8 — int8 ork_mm_run_i8 (the shipped path, with the full-K prefill M-scheduler).

It also dequant-checks pc-i4 vs the fp32 reference (the int4 MAC is exact; the RMS is just the coarse per-channel int4 quant — the 4a accuracy question).

Benchmark

M,K,N	pc-i4 ms	grp-i4 ms	i8 ms	i4/i8	grp/pc-i4	pc-i4 RMS
512,2048,512	23.16	181.10	1.74	13.3×	7.8×	10.1%
128,2048,512	6.83	47.14	0.45	15.1×	6.9×	10.2%
16,2048,512	0.82	5.77	0.15	5.7×	7.0×	10.1%
1,2048,512 (decode)	0.148	0.489	0.072	2.05×	3.3×	11.0%

(librkllmrt has no per-matmul knob; its end-to-end reference is prefill 183.9 / decode 11.71 tok/s — see Open vs Closed. int8 is the open-stack matmul that already reaches parity there.)

Results

1. Per-channel beats grouped 7–8× — the K/G submit explosion is real and per-channel removes it. ✅ hypothesis confirmed on that axis.
2. But int4 is still 2–13× slower than int8, and the shape of the gap reveals two distinct causes:
   • M-submit wall. pc-i4 scales linearly with M (0.148 / 0.82 / 6.83 / 23.16 ms for M=1/16/128/512 ≈ 0.05 ms/row) — it does one submit per row (the captured int4 regcmd is M=1; the M-count regs 0x4034/0x4038/0x1010 that drive int8's multi-row scheduler are left at 0). int8 batches rows, so it's sublinear.
   • Intrinsic per-submit penalty. At M=1 (one submit each), int4 is already 2× int8 (0.148 vs 0.072). That's not an M problem — it's the int4 regcmd's ceil(K/128) K-pass scheme + int16-output accumulate. So even a perfectly M-batched int4 would land at ~2× int8, not parity.

Verdict

int4 cannot reach int8 speed on this NPU — best case ~2× slower. Two independent reasons: (a) the NPU is submit-bound, so int4's half-size weights buy nothing (same submit count as int8 once M-batched); (b) int4's per-submit cost is intrinsically 2× int8. Porting the int8 M-scheduler to the int4 regcmd would cut prefill from 13× → ~2× (worth ~a day of regcmd RE + multi-M output-layout validation), but the result is still strictly slower and less accurate (needs Hadamard) than int8.

So int4's only real value is memory footprint, and that splits into two paths:

Goal	Path	Cost	Speed	Accuracy
Smaller download/disk	int4-stored / int8-computed (dequant int4→int8 at load, run int8)	trivial	= int8	= int8
Fit bigger model in NPU RAM	full int4 matmul (M-scheduler port + Hadamard)	~days RE	~2× int8	~int8 with Hadamard
(speed)	— none —	int8 wins outright	—	—

The pragmatic recommendation: int4-stored / int8-computed captures the download benefit for free; the full int4-matmul path is only worth it if fitting a larger model in constrained NPU RAM is a hard requirement. There is no int4 path that improves speed.

Reproduce

make int4_bench && sudo ./int4_bench 30   # governors pinned to performance first