Decode: M=1 was single-core — ORK_DECODE_MC recovers 1.62×, then hits the same sync wall

The decode companion to Prefill NPU Utilization and the Sync Wall. That entry showed the prefill gap to rkllm is the synchronous ggml execution model, not the matmul. This one applies the same per-core-utilization method to decode (M=1), finds one real, recoverable win — the decode was needlessly single-core — lands it, and then bottoms out on the identical wall. RK3588 10.3.0.236, governors max, complete .orkpack resident across 4 IOMMU domains, -t4, warm, measured on the committed build.

The starting gap

	decode tok/s	vs rkllm
ork (open stack, resident)	0.92	—
rkllm (librkllmrt 1.2.3)	4.09	0.22×

rkllm's decode is NPU-bound — sampled rk_bench decode phase: NPU Core0/1/2 = 91/91/90%, CPU idle (cpu7≈20%). So the honest question is not "ork-CPU vs rkllm-NPU" but "why is ork's NPU decode 4.4× slower per token."

The measurement: the resident decode was running single-core

Sampling /sys/kernel/debug/rknpu/load (per-core) + per-CPU /proc/stat at 150 ms during a clean, resident llama-server decode (/completion, churn 0 — not the churny llama-bench -n path, which masked this with a re-loading artifact):

	decode tok/s	NPU Core0/1/2	hot CPU
default	0.92	55 / 1 / 1 %	cpu7 = 42 %

Two of the three NPU cores were idle. Traced to one line in src/npu.c: budget() hard-returned 1 core whenever M==1, so every decode matmul — including the big FFN projections (gate/up/down, N/K = 18944) that dominate the decode wall — ran on a single core. (The earlier "decode is 3-core at 26%" reading came from the churny llama-bench decode, where the per-token re-load dominated; the clean resident decode is unambiguously single-core.)

The fix: ORK_DECODE_MC=1 → 1.62×

Gated the M==1 short-circuit behind an env flag so decode falls through to the normal multi-core budget and the big FFN matmuls split N-tiles across all 3 cores (the existing run_multicore path):

	decode tok/s	NPU Core0/1/2	hot CPU
default (single-core)	0.92	55 / 1 / 1 %	cpu7 = 42 %
ORK_DECODE_MC=1	1.49 (1.62×)	33 / 33 / 33 %	cpu7 = 62 %

All three cores now light up, decode rises 0.92 → 1.49 (1.62×), bit-exact vs the CPU reference (test_matmul/decode/layer all ALL OK with the flag on). Committed to ork-driver main as a95a655, default-off: at M=1 a small matmul (e.g. QKV, N=512) loses the multi-core barrier to the single-core dispatch floor, so this only pays on the big-N decode matmuls — opt-in keeps the safe default.

Why run_rr_chain is not the next lever (scoped and rejected by analysis)

The natural follow-on idea is a no-barrier round-robin of chained submits (run_rr_chain) — combining run_chain's PC-chaining with run_stream's per-core round-robin. For the decode's matmul shape it would be worse than the N-split just landed:

• Decode is dominated by the big FFN matmuls (N/K = 18944). The barrier N-split puts all 3 cores on each one (≈ T/3).
• A whole-matmul-per-core round-robin assigns one full matmul per core — but a decode layer exposes only 2–3 independent matmuls per group (QKV = 3, gate/up = 2). So it would leave a core idle and make each core run the un-split full matmul (≈ T each) — strictly worse than ≈ T/3, and it touches neither the per-submit dispatch floor nor the CPU-side stalls.

The 3-core win is already captured by the one-line budget change; run_rr_chain would add a substantial regcmd/scheduler path to land behind where we now are.

Why it stops at 1.49: the same synchronous-execution wall

With all 3 cores engaged the NPU still sits at only 33 % while cpu7 runs at 62 % — the CPU is now the busier resource. Compare rkllm's decode (NPU 91 %, CPU ≈20 %): rkllm's integrated executor slides the CPU feed underneath NPU compute (wall ≈ max(CPU, NPU)), while ggml-ork runs the graph one node at a time and blocks on each — quant (CPU) → run_i8 (NPU, CPU idle) → dequant (CPU) → RMSNorm/attention/RoPE/softmax (CPU) → next matmul — so wall ≈ CPU + NPU, NPU idle ~67 %. This is the exact wall the prefill entry traced: ggml-sched is synchronous; ork-driver even has an async submit primitive, but the layer above can't issue work without blocking. Closing it needs cross-node async execution in the graph runner (upstream meta-backend territory), not an ork-driver change.

There is also an irreducible piece: the decode chain is data-dependent (matmul B needs A's output, with a norm between), so even a perfect async backend couldn't fully hide the CPU behind the NPU within one token — part of rkllm's edge is engine integration ggml structurally lacks.

Verdict & next

✅ Banked a real 1.62× decode win (0.92 → 1.49, all 3 cores, bit-exact, ORK_DECODE_MC=1, default-off, ork-driver a95a655) from a single-core-at-M=1 default that per-core sampling exposed. ⚖️ run_rr_chain ruled out by analysis for the FFN-dominated decode shape. The residual 1.49 → 4.09 gap is the synchronous-execution wall (NPU 33 % / cpu7 62 %), identical to prefill — architecturally upstream, not an ork-driver tweak. ⏭️ The clean path to NPU-bound decode parity remains M>1 batched / speculative verify (EAGLE-3 / DFlash): each NPU submit then does 8–16 rows of real compute, the dispatch floor amortizes, and decode behaves like prefill (NPU busy, not starved) — sidestepping this wall instead of fighting it.

Follow-up: does async CPU‖NPU overlap recover the sync wall? (prototype, validated but modest)

To test the overlap lever directly — before anyone invests in upstream ggml-sched async work — tools/async_decode_probe.c (ork-driver 72b82e9) runs the real 7 decode projections (M=1, resident) two ways: serial (cpu_prep; run_i8) vs async-pipelined (run_i8_async; cpu_prep(next); wait), with a real representative CPU-prep workload (activation requant + RMSNorm), sweeping the CPU/NPU ratio up to decode's measured ~1.9. Bit-exact gate throughout.

Two configurations: the stock per-call async API, and a persistent worker pinned to a free big core (the production-shape fix — worker on cpu5, caller on cpu7, both big):

CPU/NPU ratio	sync (µs)	stock per-call	persistent pinned
0.09	~30,000	0.91×	1.08× (100% of ceiling)
0.20	33,000	0.96×	1.17×
0.45	40,039	1.20×	1.32×
0.79	49,339	1.32×	1.31×
1.9 (decode point)	80,761	1.21×	1.22× (80% of the 1.53× ceiling)

The mechanism works, but only with the right thread management — and even then it's incremental:

1. Default thread affinity masks it to 1.00× (an ork-driver defect). The first run showed a flat 1.00×: ork_mm_run_i8_async is a bare pthread_create that inherits the caller's CPU-affinity mask, so ork's big-core pinning lands the worker on the caller's own core — they time-share instead of overlapping. ORK_NO_AFFINITY=1 frees it but is the wrong fix — it unpins everything, so the worker can drift onto an A55 little core and lose more than it gains. Correct fix: pin the async worker to a free big core (or the big-core set via a multi-bit mask), off both the caller's core and the A55s.
2. Per-matmul spawn caps the stock path below the ceiling. ~200 matmuls/token each pay a fresh pthread_create (negative at low ratios). Fix: a persistent worker (created once, condvar-fed — ork already has exactly this in the run_multicore pool, src/npu.c:2458). Validated above: the persistent pinned worker eliminates the spawn penalty (reaches 100% of ceiling at low ratio) and holds ~1.22–1.32× across the decode range, big-cores-only. Single-core and 3-core submit overlap near-identically at the decode ratio (the NPU isn't the bottleneck there), so overlap composes with ORK_DECODE_MC rather than competing.

Verdict: ✅ async overlap is a genuine ~1.22× lever at the decode operating point (up to ~1.32× at balanced ratios), and the production fix is pinned-down: a persistent worker bound to the big-core set. But even the ceiling (~1.53× at this ratio) takes decode 1.49 → ~1.8–2.2, still ~0.5× of rkllm 4.09, and it's inert until ggml-ork is restructured to call the async API + supply independent work (today it's fully synchronous). The M=1 residual is weight-streaming bandwidth + dispatch floor, which overlap can't touch (consistent with the prefill overlap_probe ~1.15× ceiling). So async-overlap stacks on the parity path rather than being it — that path remains M>1 batched/speculative verify.

Artifacts: ork-driver a95a655 (ORK_DECODE_MC knob in src/npu.c budget()), 72b82e9 (tools/async_decode_probe.c); per-core sampler ~/npu_samp2.sh + ~/summ.awk (board); clean decode measured via llama-server /completion (the churn-free decode tool).