Prefill Parity Grind — lever-by-lever from 31 → 40 t/s, and the diagnostic that says the rest is integration (not the matmul)

Commits: ork-driver main — per-N-slice chain prefill (af5bf78) + test_sn3 regression (aceffb5), multi-domain dom_activate Af fix (cb99c1c), chain-prefill/ksplit Sn==1 guard, build/test-suite repair. ggml-ork — lever D (attn_output routing), B′ (NEON act-quant, both paths), C (CPU‖NPU overlap, gated ORK_OVERLAP) — consolidated on a fork branch. RK3588 10.3.0.236, governors max, complete .orkpack resident across 4 IOMMU domains, -t4, warm.

TL;DR

Ground-up grind on the Qwen2.5-7B-Q8_0 prefill, each lever measured and bit-exact-gated, against the closed librkllmrt reference (pp256 73.89 / decode 4.07 tok/s). Net: pp256 ~31 → ~40 t/s (54% of rkllm), decode unchanged (M=1, residence-churn-bound). Two of the levers were the only big ones; three more were measured null/neutral/tiny — and the negative results, plus a CPU-operator profile and a per-op rknn_vs_ork diagnostic, are the real payload: they prove ork's NPU matmul is competitive with Rockchip's closed kernel (~1.3× at the 3-core operating point), so the remaining gap is host-side integration + a per-core kernel residual — not a matmul wall.

Prerequisite: a trustworthy baseline (and two push-blocker bug fixes)

Cross-agent pp256 had swung 11.5/15.7/26/31/36.5 — all config artifacts. Reconciled: it's a residence continuum (IOMMU-domain count × wcache budget), and the "fast" 36.5 was an artifact of a broken chain-prefill path skipping work. Establishing a clean number surfaced two real bugs (both fixed, both validated bit-exact, both were the board-wedge cause):

• Multi-domain residence SIGSEGV at the first IOMMU domain transition — dom_activate re-seeded only regcmd+task for a fresh domain, leaving Af NULL. Fix: allocate Af at first-use too (cb99c1c). The "breakthrough" feature now actually runs.
• chain-prefill wedge for Sn>1 — chaining N-slices across distinct weight buffers emitted a malformed cross-buffer regcmd → RKNPU_SUBMIT errno 110 / cdma address wild / 60 s job timeout (the hang that wedged the board all session). Fix: gate chaining to Sn==1, route wide-N to the proven per-tile path.

Clean baseline (complete 168-tensor .orkpack, fixed multi-domain, ORK_MAXN=20000, zero eviction): pp256 26→31 t/s, decomposition NPU-run ~54% + activation-quant 6% + dequant 5% + cold .orkpack load (one-time) + ggml CPU ops.

The levers (measured, ranked by what actually moved)

lever	what	result
1 — per-N-slice chain prefill	chain M-tiles within one N-slice buffer (single-buffer-legal) so Sn>1 weights (gate/up N=18944) stop going per-tile	ioctls/matmul 55→5.17 (10.7×), run_i8 −23%, pp256 30.94→36.11 (+16.7%) — the headline. bit-exact, wedge-free (test_sn3 Sn=3 regression added)
D — route attn_output to NPU	device_supports_op guard strstr(name,"output") over-matched attn_output.weight, forcing all 28 attention-output [3584,3584] matmuls to CPU. Drop the substring; lm_head still excluded by N>max_n	pp256 35.79→40.2 (+12.6%), bit-exact, no cont-copy needed (src1 was contiguous). One line. Gain ~half the 25% CPU-slice because NPU submit/quant recaptures only part
B′ — NEON the chain act-quant	lever-2 NEON'd only the single-node path; the dense 7B uses the chain path (still scalar)	null (−0.2%) — chain act-quant isn't hot; arithmetic is only ~4% of wall. Landed anyway (bit-identical, lower CPU energy)
A — fuse dequant→requant	skip the f32 round-trip on matmul→matmul edges	structural no-op — 0 such edges (100% of NPU matmuls are fed by GGML_OP_MUL/RMSNorm; a norm always sits between matmuls)
C — CPU‖NPU overlap	async-submit + overlap dequant of the next chained matmul (ORK_OVERLAP, default-off)	~3% ceiling (only dequant is movable; quant is shared up-front). Gated, low wedge-risk (single-in-flight)

The CPU-operator profile (what the host actually spends)

ORK_PROFILE only timed the ork-backend phases; instrumenting the ggml CPU nodes showed the rest. Per pass (pre-D): NPU matmul 54% + CPU MUL_MAT ~25% (the 28 attn_output, mis-routed — that's lever D) + act-quant 6% + dequant 5% + FlashAttn/softmax 4.6% + genuine activations (RMSNorm/RoPE/SwiGLU/add) only ~7%, already NEON + threaded 3.4–3.6× across 4 cores. So faster activation kernels are not the lever (no headroom, and ork's neon_activations.c is dead code here — ggml's CPU kernels run them). The ~25% "CPU ops" hotspot was a matmul on the wrong backend — fixed by D.

The diagnostic — rknn_vs_ork: is our matmul the problem? (No.)

Same int8 (M,K,N) through ork vs the closed RKNN matmul API (librknnrt), M=256, median-of-5:

shape	ork-3core	RKNN-1core	ork-3c / RKNN
q/o/attn_out 3584×3584	9459 µs	7299	1.30×
gate/up 3584×18944	47509 µs	37436	1.27×
down 18944×3584	24991 µs	1,552,220	0.02× (RKNN pathological large-K — excluded)

Stable 1.27–1.31× across M=64/128/256/512. AC-layout probe: RKNN's "native" layout is 1.28× slower than normal → ork isn't disadvantaged by layout; the comparison is fair.

Verdict: ork's matmul at the real (3-core) operating point is within ~30% of Rockchip's closed kernel — same efficiency class, no hard frontier. At 1.3× and ~65% of the wall, the matmul residual cannot account for the 40→74 full-model gap → the bulk is host-side integration/scheduling/engine (the closed rkllm's advantage is how it orchestrates the matmuls, not a faster matmul). The nuance: ork-1core is 3.5× slower than RKNN-1core — ork uses all 3 cores to reach 1.3×, so the per-core kernel has real (hard) headroom (matching a hand-tuned vendor kernel via regcmd RE). Because matmul dominates the wall and the same kernel runs at decode/M=1, closing the per-core residual is a both-regimes win — the next target.

Verdict & next

✅ pp256 31→40 (54% of rkllm), bit-exact, two push-blocker wedges fixed, build/test-suite repaired. The grind's real value is the map: levers 1 + D were the only big prefill wins; activation kernels / dequant-fusion / chain-quant are spent; the matmul is competitive, so the remaining gap is per-core-kernel efficiency (deep regcmd RE) + host integration. ⏭️ NEXT: go after the per-core matmul kernel via regcmd RE (close the 3.5×/core gap toward RKNN) — it lifts prefill and decode and the future M>1 verify path. Methodological note recorded for the community: every "fast" number this session was a config artifact until pinned — measure the exact config (ioctls/call fingerprint) before trusting a t/s.