2026-07-10 — ✅ Fused matmul + output-stage activation: the no-crossing chain (softmax exp / RMSNorm rsqrt / SwiGLU)

Commit: ork-driver main · src/npu.c (ork_mm_run_f16_act), include/ork_npu.h, examples/test_bmm.c (test_fused_act, in make test).

Realizes the fusion that M4.6 / Exp-2026-07-10 On-NPU Attention/Norm identified as the win: a non-matmul op (softmax exp, RMSNorm rsqrt, SwiGLU silu) that rides an adjacent matmul's DPU output stage in ONE submit — no separate op submit, no CPU↔NPU crossing, no activation→matmul mode-switch. This is why a standalone on-NPU op loses (submit-floor-bound, ~200 µs–2 ms for something CPU does in µs) but a fused one wins: the activation is free — it's the matmul's own submit.

What shipped

ork_mm_run_f16_act(c, K,N,B, M,A, C, fn, fnctx, in_lo, in_hi) → C[M,N] = fn(A·B) in one submit. Mechanism (fp16, PPU-fuse-gated, rk3588): build fn's PWL LUT over the matmul-output range [in_lo,in_hi] (ork_mm_build_f16_lut), pack the weight as ±S·B so acc lands in the fp16 index-spread band, run the fused-LUT matmul (ork_mm_run_f16_silu is fn-agnostic — it applies whatever LUT it's handed), recover fn(x)=C·out_scale. Validated in make test:

fused op	use	accuracy (maxrel vs CPU)
exp(A·B)	softmax	0.0041
silu(A·B)	SwiGLU FFN	0.0017
gelu(A·B)	GELU FFN	0.0063
rsqrt-into-reduce	RMSNorm/L2Norm	0.0006 (shipped earlier, same mechanism)

The one real constraint: single-signed output

The fp16 SDP output-stage index only spreads for one sign of acc, so the matmul output must be single-signed across the tile (one LUT covers one sign). The primitive handles both signs by construction:

• positive-input fns (RMSNorm rsqrt of ss>0): pack −S·B → acc=−S·x<0.
• negative-input fns (softmax exp(scores−max), input ≤ 0): build g(u)=fn(−u) over positive u, pack +S·B → acc=+S·x<0. ✅ softmax's exp fits naturally (post-max-subtract is ≤ 0).
• mixed-sign output (e.g. a raw SwiGLU gate spanning ±): returns -2 — needs two LUTs (or CPU). This is the actual meaning of the earlier "fp16 silu WIP" note; it's a single-signed-index limitation, not a bug.

Why this is the answer (vs the kernel-sequenced replay)

The alternative — replaying the vendor's standalone 9-task softmax graph — works bit-exact (see NPU-Quirks) but pays 9× the submit floor (~2 ms) as separate task_number=1 submits, because the rknpu vendor driver needs hardware PC-chaining for a single-submit multi-task job and that chain stalls at the LUT-load task. Fusion sidesteps all of it: the transcendental is the matmul's output stage, zero extra submits. For softmax the full op still needs the CPU max-subtract + the Σ reduction (a second matmul) — fusion delivers the expensive exp for free on the score matmul; the reduction rides its own matmul and the ÷Σ is a cheap CPU scalar. RMSNorm is fully covered (reduction = matmul, rsqrt = fused output stage).

Verdict

The no-crossing chain is real and validated. ork_mm_run_f16_act is the reusable primitive; wiring it into ggml-ork (fold exp onto the attention score matmul, silu/gelu onto the FFN up-proj, rsqrt onto the norm reduce) is the integration follow-up. RE box + tooling unchanged (tools/re/).