Exp 2026-07-13 — Linear-Attention Scans on the NPU (Mamba-2 SSD ✅ at scale, Gated-DeltaNet ❌)

Porting the two big linear-attention scan families to the RK3588 NPU as chunked matmul + CPU-elementwise: Mamba-2 / SSD (GGML_OP_SSM_SCAN) and Gated DeltaNet (GGML_OP_GATED_DELTA_NET; Qwen3-Next / Qwen3.5 / Ornith-1.0-9B). Both were built end-to-end, wired into the ggml-ork backend, and validated coherent on silicon. SSM wins ~2× at scale; GDN loses — and the why is a clean architectural result.

Knobs: ORK_SSM_NPU (size-gated default-on), ORK_SSM_KEEPWARM (default on), ORK_SSM_CHAIN (default on), ORK_SSM_CS (default 128), ORK_GDN_NPU (opt-in, off), ORK_SSM_PROF / ORK_GDN_PROF (per-stage). ork-driver main @ 6ecb168 (perf defaults); GDN parked on feat/gdn @ 3ce735e.

---

1. Design — chunked scan as matmul + CPU elementwise

Both scans use the standard chunked (CS=64) reformulation so the heavy contractions become NPU matmuls, with the sequential inter-chunk carry + gate/decay elementwise on CPU. The kernels mirror each other:

• SSM (ork_ssm_scan_f32): 4 matmul stages/chunk (pS gram C·Bᵀ, pD intra-chunk output, pC chunk-state, pO inter-chunk) on a pooled fused-multicore fp16 stream; softplus/cumsum/decay on CPU.
• GDN (ork_gdn_scan_f32): 6 matmul stages/chunk (Sk/Sq state-reads, KK/KQ grams, O_intra, Sdelta) plus a CPU forward-substitution UT-transform — the delta rule's (I+A)⁻¹, the one non-matmul piece.

Both are validated by pure-CPU references (test_ssd_chunk, test_gdn_chunk) that prove the chunked form is bit-exact to the definitional sequential recurrence (~1e-11) at the real d=128/CS=64, then NPU variants (test_ssd_chunk_npu, test_gdn_chunk_npu) confirm on-hardware coherence within fp16 tolerance.

---

2. The general unlock — keep-warm (int8↔fp16 no-thrash) ✅ DEFAULT ON

Both scans interleave fp16 scan matmuls between the int8 projection matmuls, and the NPU regcmd mode is stateful: each transition was doing an ACT_RESET + re-warm + scratch-realloc per layer. That churn dominated (28 "soft resets" / prefill; matmul cost tripled). The fix — extend the existing ORK_MIXED_NOTHRASH keep-warm to the int8/fp16 family (ORK_KW_DT = I8_LIVE || F16) at every transition site — is safe because the regcmd is re-synth'd every call regardless, and the reset was a conservative software choice, not a HW law (NVDLA/rocket reset only at init).

Result: on-NPU SSM prefill 20.7 → 103.5 tok/s (5×), soft-resets 28 → 1, byte-coherent. This is a general win for any fp16 op interleaved with int8 matmuls (scan, softmax, chained attention). Now default-on (ORK_SSM_KEEPWARM=0 to disable).

---

3. Mamba-2 / SSD — ✅ crossover is REAL, ~2× at Q8 2.7B

Earlier 130M-scale work concluded "no crossover, param count isn't the lever." That was a small-scale + Q5_K dilution artifact. The clean picture, NPU-scan vs CPU-scan (projections int8-on-NPU both sides), warm:

model	quant	NPU-scan	CPU-scan	ratio
mamba2-130m	f16	103.5	147.6	0.70× (loses)
mamba2-2.7b	Q5_K_M	9.06	10.05	0.90× (diluted — Q5_K projections dequant-bound both sides)
mamba2-2.7b	Q8_0	9.72	4.37	2.22× (WINS) — P=128; 2.02× at P=512

Warm P=128 (Q8 2.7B): 29.65 vs 6.51 tok/s = 4.55× (removing the one-time weight-pack un-dilutes the ratio).

Why it crosses over: 130M→2.7B, the CPU scan degrades ~33× (Mamba-2's per-token recurrence is memory-bound and craters under cache pressure) while the NPU degrades only ~11× (it amortizes the submit floor across the growing count of fixed-size 64×128×64 matmuls via chaining). Output byte-identical NPU vs CPU. Enablers all default-on: keep-warm + fused-multicore fp16 stream (ORK_SSM_CHAIN) + CS=128 + persistent pool.

Size-gated default (ORK_SSM_NPU unset = auto): on-NPU only when the model is large enough to win — n_head ≥ 64 (≈ ≥1.5B Mamba-2; head_dim/d_state are fixed in the family so nh is the size proxy). Small Mamba-2 (130m, nh=24) and decode stay on CPU. ORK_SSM_NPU=1/=0 forces on/off for A/B.

---

4. Gated DeltaNet — ❌ built + coherent, but not a win

Full stack built following the SSM pattern and validated coherent end-to-end in a real Qwen3.5 model (byte-identical output vs CPU, no wedges): the ork_gdn_scan_f32 kernel, a ggml-ork marshaller (densify non-contiguous q/k/v/g/β + state key/val transpose + GQA hv%Hk head replication), the GATED_DELTA_NET supports_op/graph_compute cases, and a weightless-op offload path in ggml-backend.cpp (GDN has no USAGE_WEIGHTS src, so the sched wouldn't route it otherwise — scoped to the one op).

But the per-stage timer (ORK_GDN_PROF) shows the scan loses to ggml's CPU scan at both scales, gap widening:

model	quant	NPU-GDN scan	CPU-GDN scan	ratio
Qwen3.5-0.8B	Q8	~262 ms	~135 ms	~1.9× slower
Qwen3.5-9B	Q4	~748 ms	~156 ms	~4.8× slower

NPU-scan breakdown (both scales): NPU matmul-submit 44%, CPU UT-solve 38%, prep (fp16 cast) 14%, post 4%. (An earlier "0.8B 1.14× win" was single-run noise — the CPU number jumped 112→150 between runs; the isolated per-stage measurement is the honest read.)

Why GDN loses where SSM won (the real result):

1. The chunked form's UT-solve is pure overhead here. ggml's CPU GDN op is a solve-free per-token recurrence — it never does the (I+A)⁻¹ transform. The chunked reformulation (mandatory to make the scan into NPU matmuls) adds the CPU forward-substitution (38% of the scan) that the CPU path simply skips.
2. NPU submit overhead on many small 128×128 matmuls × heads × layers (44%).
3. No scale crossover. SSM won because Mamba-2's CPU recurrence craters at scale. GDN's state is a fixed 128×128/head that fits in cache, so ggml's vectorized recurrence stays fast at any size — there is no cratering baseline for the NPU to overtake.

Verdict: not-yet-viable. Would need the UT-solve on-NPU (kill the 38%) and far cheaper submit, and even then it fights a cache-resident CPU recurrence. Parked on feat/gdn (coherent, ORK_GDN_NPU-gated, no regression). Reusable if a breakthrough or a different linear-attention variant with a crater-prone CPU baseline appears.

---

5. Reusable machinery (survives regardless of the GDN verdict)

• Keep-warm (§2) — general int8↔fp16 no-thrash; default on.
• Fused-multicore fp16 stream (ork_mm_run_stream_f16_chain) — PC-chaining + 3-core, amortizes the submit floor.
• Weightless-op offload (ggml-backend.cpp) — routes activation-only ops (no weight src) to the NPU.
• densify marshaller — feeds non-contiguous (permuted-view) in-graph operands to the NPU.

These three (weightless-offload + densify + fused-multicore fp16 + keep-warm) are exactly the machinery attention-on-NPU needs — the top re-evaluation target now that GDN proved they work end-to-end.

6. Method note — verify, don't recall

The GDN "win" I nearly reported was a false positive from a coarse end-to-end prefill number; the isolated ORK_GDN_PROF measurement overturned it. With linear-attention scans + activations now running on the NPU, prior "not viable / closed / CPU-only" verdicts predate the capability and should be re-derived on hardware, not trusted (e.g. the 130M "no SSM crossover" conclusion, which was wrong at scale).