NPU Tiling Autotuner + K-Tiling — and the proof that prefill is host-bound, not matmul-bound

Commit: ork-driver 33905ff (branch re/rkllm-mtile-decode, local, not pushed) · tools/autotune.c · tools/run_autotune.sh · ORK_KTILE knob in src/npu.c · AUTOTUNE_WIP.md. RK3588 10.3.0.236, governors max, isolated checkout.

TL;DR

Built an NPU matmul tiling autotuner (per-shape config search, bit-exact-gated, cached) and ran it on the Qwen2.5-7B prefill shapes. It found a real matmul-level win — CBUF K-tiling lifts GOPS +36–60% by raising the CBUF row-ceiling R. But end-to-end pp256 was flat (11.96 → 11.46 t/s, ~16% of librkllmrt's 73.6). The decisive result: speeding the NPU matmul +36–60% moved prefill 0% → prefill is host-bound (ggml-ork CPU-side quant/dequant/scheduling), not NPU-matmul-bound. This resolves a long prefill-optimization arc: there is no NPU-side prefill lever left; the K-tiling win is banked for the M>1 (batched-verify / spec-decode) regime, where the matmul actually binds.

The autotuner (Phase A — works)

tools/autotune.c: for each shape (M,K,N) — pack once, compute a CPU int8 reference once, then sweep the knobs ork exposes (core count 1–3, M/N-subtile via ORK_SMALLTILE, K-tile via ORK_KTILE), bit-exact-gate every config against the reference, time warm ork_mm_run_i8 (median), report effective GOPS, and cache the per-shape min-µs winner. Reusable across models/shapes and re-runnable per SoC.

Finding: the existing knobs are saturated — M-tile is already clamped to R−1 (the CBUF row ceiling), and core-count (1→3 ≈ 2.7×) is the dominant lever and already the default. The only new lever is K-tiling.

K-tiling (Phase B — real matmul win, ORK_KTILE, bit-exact, gated)

ork's prefill row-ceiling is R = pow2_floor(2·cbuf/K) — it shrinks as K grows (R=16 at K=3584). Smaller K-slices → smaller per-row CBUF working set → bigger R → larger M-tile → more M-amortization of the ~80–100 µs per-task submit floor. ORK_KTILE overrides the int8 K-slice (suppressing the full-K Bf, reusing the already-validated wide-K accumulate — no new regcmd, wedge-safe). KT=1024 (R 16→64) was the sweet spot (M=256, 3 cores):

shape	K×N	default GOPS	KT=1024	gain
q_proj / o_proj	3584×3584	700	1118 / 1104	+60%
gate_up	3584×18944	731	994	+36%
kv_proj	3584×512	479	479	0 (N-bound)
down	18944×3584	1413	(already K-split)	—

Bit-exact vs the CPU reference at every config; no wedges.

End-to-end — the decisive negative

Warm pp256, 7B Q8_0, GGML_NO_REPACK=1, -t4, r3, governors max:

	pp256 t/s	% of rkllm 73.6
default	11.96	16.2%
ORK_KTILE=1024	11.46	15.6% (flat/−4%)

A +36–60% matmul-GOPS win produced 0% end-to-end prefill gain. run_i8 is ~9.9 ms/call, but prefill is dominated by host-side work — activation quant/dequant, scheduling, and the per-slice A re-tile + extra accumulation passes K-tiling itself adds. So the matmul compute K-tiling accelerates is not the binding constraint; the host/ggml-ork path is.

Verdict

✅ The autotuner works and K-tiling is a genuine matmul-level lever (+36–60% GOPS, bit-exact). ⚖️ But it does not move prefill: the 12→73.6 prefill gap is host/ggml-ork-side (CPU quant/dequant/scheduling), not NPU matmul throughput. This closes the NPU-side prefill-optimization arc (submit-floor → quant → tile-size → K-grouping → multi-task-M-offset [doesn't exist] → autotuner): the matmul is already adequate, the knobs are saturated, and faster matmul doesn't help. ORK_KTILE stays gated-off; the matmul-GOPS win is banked for the M>1 batched-verify path (spec-decode), where the matmul binds. The autotuner + K-tiling are durable, reusable assets (re-tune any model/SoC; pay off at M>1).

NEXT: pivot to spec-decode — decode→M>1 (the throughput win the multi-domain residence breakthrough unblocked) and the regime where the banked K-tiling matmul win finally binds.