Experiment — int4 W4A4 grouped path

Date: 2026-06-12 · Status: 🧪 EXPERIMENTAL (incoherent — needs Hadamard) · Path: ggml-ork

Hypothesis

The NPU MAC is same-precision, so there is no w4a16 (int4 weight × fp16 activation) — int4 weights require int4 activations. Hypothesis: per-group int4 quant (group size G along K) for both weights and activations, with per-group dequant in fp32, halves bytes again vs int8 and runs on the NPU's native int4 MAC.

Implementation

• Commit: 81533e27 (oRKLLM/llama.cpp-rockchip, ork-backend); selected by ORK_QUANT=4.
• Symbol: ggml-ork.cpp : ggml_backend_ork_mul_mat_i4 → ork-driver ork_mm_pack_i4_grouped / ork_mm_run_i4_grouped.

const int G  = (K % 128 == 0) ? 128 : (K % 64 == 0) ? 64 : 32;  // largest std group dividing K
const int NG = K / G;
// ...
ctx->as.resize((size_t) M * NG);            // per-row, per-group activation scale
ork_weight ow; ow.gsize = G; ow.bscale.resize((size_t) NG * N);   // per-group, per-channel
for (int g = 0; g < NG; g++)
    /* ... per-group int4 quant of weights into bi, scale into ow.bscale ... */

ork-driver run (per-group W4A4 dequant into fp32 dst — C_real = aScale·bScale·C):

int ork_mm_run_i4_grouped(ork_npu *ctx, ork_w *w, int M, const int8_t *A,
                          const float *aScale, const float *bScale, float *C);
// cost: K/G submits per core — larger G = fewer submits, coarser scale

Benchmark (standardized — 2026-06-13)

Qwen3-1.7B-w8a8 GGUF, llama-bench -p 512 -n 128 -r 2, all-NPU (ORK_MINM=1), DDR 2112 / CPU 2.4 GHz.

Config	pp512	tg128
int4 W4A4 (ORK_QUANT=4)	5.73 ± 0.02	2.73 ± 0.02
int8 W8A8 (ORK_MINM=1)	131.42 ± 2.29	5.03 ± 0.15
librkllmrt W8A8 (closed ref)	183.9	11.71

(int4 is the post-affinity build; CPU-core pinning lifted int4 prefill 3.66 → 5.73, since the int4 multi-core workers were on the little cores.)

rkllm 4-bit: rkllm's own 4-bit is w4a16 (weight-only 4-bit — a different scheme from our W4A4), and the 1.7B w4a16 .rkllm on this board fails to load (E rkllm: the model's target_platform does not match the real platform), so no rkllm 4-bit data point is available here. The W8A8 reference stands in.

int4 is ~23× slower on prefill and ~1.8× slower on decode than int8 — the per-group W4A4 layout costs K/G submits per matmul, multiplying the submit floor. Combined with the incoherent output, int4 is strictly worse here.

Results

• Runs without error, but output is fluent-yet-incoherent.
• ~9.5% matmul-error floor (W4A4), compounding across layers.
• Much slower than int8 (above) — K/G submits/core multiply the submit floor.

Verdict

🧪 Experimental only. W4A4's ~9.5% error needs a Hadamard rotation for quality (not implemented). int8 is the practical path. Left runnable behind ORK_QUANT=4 per the "keep experiments runnable" rule — useful if a Hadamard pre-rotation is added later. Confirms the hardware reality: w4a16 and mixed-precision are ruled out by the MAC array (same-precision only).

Open work → Optimization Roadmap, Tier 4: the two fixes are (a) Hadamard rotation (accuracy) and (b) submit-count reduction (speed — the per-K-group scheme is ~16 submits/matmul vs int8's 1; fix via per-channel int4 + full-K single submit, and/or graph fusion which — unlike int8 — does help int4 because int4 is submit-bound, not work-bound). Note: you cannot pack two int4 matmuls into one int8 MAC (cross-term contamination), and it wouldn't help anyway (this NPU is submit-bound, not MAC-bound).