Experiment — Full-K prefill (close the prefill kernel gap)

Date: 2026-06-13 · Status: ✅ KEPT (default on; prefill → 97% of librkllmrt) · Path: ork-driver

Hypothesis (Tier 1c-ii)

The single-core large-M (prefill) gap to RKNN was diagnosed as: NPU submit 64% / host accumulate 21% / copy 10% (roadmap 1c). The 21% host accumulate is caused by the K-split — for K=2048, int8 uses KS=1024 → Sk=2 K-slices, and every output element is written twice (host int32 +=), plus each submit cache-syncs its full result. K=2048 ≤ the 10752 single-submit cap, so one submit over the full K (Sk=1) should remove the second slice's accumulate and halve the result cache-sync.

Implementation

• Commit: c1bb9cc (oRKLLM/ork-driver) · knob ORK_FULLK_PREFILL (default on, =0 disables).
• The full-K weight layout (Bf) already existed for M=1 decode. Extended it to M>1 prefill: one submit per M-tile over the full K using the NPU's internal M-scheduler (synth_i8(..., sched=1)), result written directly (no +=). Added to both the single-core (run) and multi-core (mcworker) paths.
• Gated to power-of-2 K (K & (K-1) == 0). The M-scheduler is only valid for power-of-2 Kp; K=6144 (down-proj) produced wrong output, so non-pow2 K falls back to the K-split path (still correct). Caught by the CPU-reference check below.

Correctness

examples/quant (random fp32 → per-channel int8, ork output vs the fp32 reference, RMS rel err < 3% = pass). With full-K on, RMS err unchanged (~0.55%) for M=1/64/128/512 at K=2048/4096; K=6144 falls back and passes. (Before the power-of-2 gate, K=6144 failed — the harness caught it.)

Benchmark (standardized — 2026-06-13)

Qwen3-1.7B-w8a8, warm, DDR 2112 / CPU 2.4 GHz.

Matmul (mc_prof, M=512 2048×2048, µs/matmul):

	full-K off	full-K on
1-core	8567	4763 (1.8×)
3-core	4376	3340

End-to-end (llama-bench -t 4, tok/s):

	full-K off	full-K on	librkllmrt
pp512 (prefill)	156.0	178.5	183.9
tg128 (decode)	12.82	12.80	11.71

Verdict

✅ Kept, default on. Full-K prefill lifts single-core matmul 1.8× and end-to-end prefill +14% (156 → 178). Combined with big-core thread count, the open stack now reaches prefill 178 = 97% of librkllmrt (184) and decode 12.8 = 109% (beats it) — at parity on prefill, ahead on decode, on the same model. The residual prefill gap is the deeper NPU-MAC-efficiency / native-B-layout work (1c-i).