Exp-2026-09-05 — Heterogeneous dispatch: the bus ceiling is ~30 GB/s, and two engines already reach it

Question: decode is weight-streaming work. If CPU + NPU together beat either alone, does adding the Mali GPU as a third engine beat that?

Answer: it depends entirely on whether the third engine's work is memory-bound or compute-bound.

• Memory-bound third engine: NO GAIN. The DDR controller saturates at ~30 GB/s / ~88–90% load, CPU+NPU already reach 29.3 GB/s of it, and a third memory client only redistributes the same total.
• Compute-bound third engine: GENUINELY ADDITIVE. Mali running a compute-bound kernel (DMC 0%) beside a CPU+NPU pair saturating the bus at 88% cost 0.7% of their throughput and lost 0.1% of its own.

The two resources are separate, so heterogeneous work overlaps for free. The homogeneous result below is the special case, not the rule.

Kernel 6.1.115-vendor-rk35xx-fence, board 10.3.0.236, DMC 2112 MHz. Probes: tools/engine_mix_probe.c, tools/vk_bw_probe.c (+ tools/vkbw.comp).

---

First, a retraction

An earlier probe reported "37.4 GB/s aggregate, ~1.5× over the best single engine." That number was wrong. It added two different currencies — the NPU's useful weight bytes and the CPU's memcpy read+write traffic — and the sum exceeded the board's theoretical LPDDR peak (2112 MHz × 2 × 8 B = 33.8 GB/s), which is by itself proof that it over-counted. It was never a bus-utilisation measurement and should not have been quoted as one.

Two fixes make the replacement trustworthy:

1. One currency. Every engine's work is a pure weight-read stream (decode writes almost nothing), so "bytes consumed" is unambiguous — no read+write doubling, no write-allocate guesswork.
2. Ground truth. /sys/class/devfreq/dmc/load — the DDR controller's own utilisation counter — is sampled during every measurement, so the summed GB/s is checked rather than trusted.

The check passes, repeatedly and independently: 27.40/0.81 = 33.8, 30.17/0.90 = 33.5, 20.88/0.63 = 33.1, 22.66/0.70 = 32.4 — every configuration implies the same ~33-34 GB/s at 100%, which is the theoretical peak. The accounting and the hardware counter agree within ~2%.

What each engine pulls alone

engine	GB/s	DMC
1 A76 core, pure stream	18.58	53%
3 A76 cores, pure stream	27.41	81%
1 A76 core, decode-shaped GEMV	6.91	21%
3 A76 cores, decode-shaped GEMV	20.78	61%
NPU (chained int8 matmul)	20.73	63%
Mali-G610 (Vulkan compute, coalesced)	22.66	70%

The GEMV arm is calibrated: 6.91 GB/s on one A76 matches the independently known ~8 GB/s for a real M=1 decode loop (test_nf4_decode), against the ~21.9 GB/s that core can stream. Decode is latency-bound, not bandwidth-bound, at the single-engine level — which is precisely why adding an engine can help at all.

Two engines

CPU arm	CPU alone	+NPU (CPU / NPU / sum)	DMC	gain vs best single
stream	27.41	14.43 / 15.69 / 30.12	90%	1.10×
GEMV (decode-shaped)	20.78	12.25 / 17.05 / 29.30	88%	1.41×

The dual-engine gain depends entirely on how badly the engines saturate the bus on their own. A perfectly streaming CPU already holds 81% of the bus, so the NPU can only add 10%. A decode-shaped CPU holds 61%, and the NPU fills most of the rest — 1.41×.

The ceiling shows itself directly: CPU2+NPU = 30.08 and CPU3+NPU = 30.12. Adding a whole extra core of demand raised the total by 0.04 GB/s and merely shifted the split. That is a saturated resource.

Three engines — the decisive test

Mali was run concurrently (Vulkan compute streaming) while the CPU+NPU sweep was repeated:

GEMV, CPU3+NPU	CPU	NPU	CPU+NPU	DMC
without Mali	12.25	17.05	29.30	88%
with Mali	7.38	9.54	16.92	85%

Total bus traffic did not increase — it fell slightly (88% → 85%). Mali took ~12 GB/s directly out of CPU and NPU. Every configuration in the sweep was depressed proportionally (a single CPU core dropped 6.91 → 4.33 GB/s) while DMC stayed pinned near saturation regardless of how many engines were demanding.

So a third engine doing MEMORY work does not beat two. The bus is the binding constraint and two engines already reach ~98% of what it will deliver.

But a compute-bound third engine IS additive

The test above is homogeneous — every engine streaming. Repeat it with Mali running a compute-bound kernel instead (four independent dependent-integer chains per invocation, one uint written; measured DMC avg 0%, so it is genuinely memory-free):

	solo	concurrent	retention
Mali compute	7978.1 Mit/s	7970.2 Mit/s	99.9%
CPU+NPU memory (GEMV)	29.30 GB/s	29.10 GB/s	99.3%

Both sides keep essentially everything, against a bus already at 88%. Compare the same GPU doing memory work, which cut CPU+NPU to 16.92 GB/s (58%). Same engine, same board — the only variable is which resource its work consumes.

Additivity is proportional to how memory-free the "compute" work really is

The NPU can be made compute-bound too, by raising M so one weight stream is amortised over many rows (ORK_MIX_NPU_M=256, K=1024, N=160 → 97.9 GMAC/s at DMC 16%, versus 63% at M=1). But it is only partially memory-free: at M=256 each task still reads A (256 KiB) and writes C (160 KiB) against 160 KiB of weight. The result tracks that exactly:

config	CPU retention	NPU retention	DMC
CPU1 + NPU(M=256)	95%	96%	35%
CPU2 + NPU(M=256)	96%	79%	52%
CPU3 + NPU(M=256)	92%	61%	67%

Additivity holds while the bus has headroom and erodes as it fills. Mali's truly memory-free kernel stayed additive even at 88%; the NPU's partially-memory-free one starts losing past ~50%.

The rule: engines are additive when they contend for different resources and zero-sum when they contend for the same one. "How many engines" is the wrong question; "what does each engine's work consume" is the right one. Any dispatch decision should be made on the resource profile of the op, not on engine count.

Caveat on the M>1 accounting

At M>1 the GB/s column counts weight bytes only, while the NPU also moves A and C — at M=256 that is 256 + 160 KiB against 160 KiB of weight, so the column undercounts NPU DRAM traffic ~3.6×. DMC is the arbiter there, not the sum. (This is the same class of error as the retracted 37.4 GB/s figure; the metric is kept because it is the right one at M=1, which is the decode shape.)

Why the NPU cannot be a pure compute engine here (and it is not an RE gap)

Measured NPU compute, routed multi-core, M at/above the M-tile cap:

shape	GMAC/s	GOP/s	DMC	vs 6 TOPS
K=4096 N=160 M=256	130.9	262	21%	4.4%
K=4096 N=1024 M=256	322.3	645	26%	10.7%
K=4096 N=1024 M=64	298.4	597	24%	10.0%

Raising M does not raise compute density. M=64 and M=256 give the same rate at the same DMC, because at K=4096 the M-tile cap is 64 rows and M=256 simply runs four tiles — re-streaming the 4 MB weight four times. (The GB/s column counts the weight once, which is why it appears to fall from 4.66 to 1.26 while DMC does not move. DMC is the arbiter.)

So arithmetic intensity is rows per weight stream = the M-tile cap, and the NPU stays a ~25%-DMC memory client no matter how much compute you give it. It cannot approach 6 TOPS: sustaining that at ~64 MAC per weight byte would need ~94 GB/s of weight bandwidth, and the bus delivers ~30.

The obvious fix — the hardware WEIGHT_REUSE bit — is closed and unavailable. CNA_CBUF_CON0 (0x1040) bit 13 does mechanically skip the weight re-DMA (M=128: 1639 → 1034 µs, 1.6×) but computes wrong: the weight is not retained, there is no retention register, +DATA_REUSE is faster and still wrong, and +FC_SKIP_EN errors the submit. A capture of librknnrt's own int8 matmul (B set once, four runs on one context) is byte-identical every run with 0x1040=0x84, i.e. WEIGHT_REUSE=0 and the weight IOVA re-fetched each time — the vendor's own matmul never uses it either. WEIGHT_REUSE/FC_SKIP are conv-only (a ~36 KB filter fits WEIGHT_BANK and stays resident across spatial tiles); an LLM matmul weight is ~76× the ~57 KB CBUF and can never be resident. See Experiment Log #39 and Optimization-Roadmap Tier 13 (weight-stationary modelled + measured at break-even, ×0.98–1.03).

Consequence for dispatch: the NPU is the best compute-per-bandwidth engine available (~37 MAC per DRAM byte against the CPU's ~1 on decode GEMV), but it is a hybrid client, not an off-bus one. Mali's compute kernel was the only arm that measured genuinely off-bus (DMC 0%). Assign the NPU compute-heavy work because it is bandwidth-efficient per unit of compute — not because it stays off the bus, which it does not.

Mali is NOT a compute peer for int8 GEMM — measured, bit-exact

The open question was whether Mali's absolute compute rivals the NPU's, given the NPU is bandwidth-capped at ~645 GOP/s. Measured on the same shape (M=256, K=4096, N=1024) with a real tiled int8 GEMM (tools/vkgemm.comp + tools/vk_gemm_probe.c), verified bit-exact against a CPU reference (7085/7085 sampled outputs) before any number was reported:

engine	int8 GEMM	vs
NPU	322 GMAC/s (645 GOP/s)	—
Mali-G610 (Vulkan, HW dot product, register-blocked)	17.5 GMAC/s (35 GOP/s)	18× slower

Both the primitive and the shader were checked before blaming the hardware:

• integerDotProduct4x8BitPackedSignedAccelerated = **true** — the 4×int8 dot product is genuinely hardware-accelerated on Mali, not emulated.
• The signed overload is selected by the int container type; uint containers silently resolve to the unsigned overload — a correctness trap that happens to surface here only as a type error.
• Widening the K tile 4× (16 → 64 K-values per barrier pair) changed nothing (7.3 → 7.5 GMAC/s), ruling out barrier overhead. Register blocking did: the naive loop issued two shared-memory loads per dot product, and hoisting the 4 A and 4 B values per k gave 2.3× (7.5 → 17.5).

A production kernel would beat 17.5, but not by 18×. The conclusion is robust to shader quality: for int8 GEMM the NPU is roughly an order of magnitude ahead of Mali on this SoC.

Accumulator retention — chased, and CLOSED negative (no RE needed)

The reasoning that led here: the M-tile cap is CBUF-capacity-driven (cap×K ≈ const — measured 704@K512, 320@K1024, 128@K2048, 64@K4096), so weight traffic for M rows is M·K²·N/const — quadratic in K. Splitting K into S slices should cut weight traffic by S, and the only thing standing in the way is that ork host-sums the S partials (npu.c:123, chain.c:400/614). Hence: is on-chip accumulator retention the missing mechanism worth reverse-engineering?

Measured first, before touching the output stage (tools/ksplit_accum_probe.c, M=704 K=4096 N=1024, every arm verified bit-exact against the single-task result):

S	NPU-only time	GMAC/s	vs single
1 (single K=4096 task)	5.31 ms	555.8	1.00×
2	5.58 ms	529.6	0.96×
4	8.13 ms	363.3	0.65×
8	13.69 ms	215.7	0.39×

K-slicing is monotonically WORSE, and that is the NPU time alone — before any accumulation cost. Adding the host sum (5.07 ms at S=8) takes it to 0.28×. So even if accumulator retention were completely free it would recover at most 0.39×, i.e. it cannot make K-slicing win. The mechanism is not worth reverse-engineering: there is nothing behind it.

Why the premise was wrong. Weight traffic does fall with S, but the output stage cost scales with S: every slice produces a full M×N int32 cube, so S=8 writes 23 MB of partials instead of 2.9 MB and does 8× the output-stage work. That swamps the weight saving. The cost model is not weight-DMA alone — it is weight-DMA plus per-output-element work that K-slicing multiplies. This is the concrete reason behind Tier 13's break-even result and behind the vendor never K-slicing either.

Correction: the NPU is faster at compute than earlier numbers in this page implied

The single-task path reaches 555.8 GMAC/s = 1.11 TOPS = 18.5% of the 6 TOPS nameplate (2 TOPS/core × 3), against the 322 GMAC/s measured earlier through ork_i8_mm_run_chain with T chained tasks. The chained path understates NPU compute by ~1.7×; large-M ork_i8_mm_run is the right shape to quote. Consequently the NPU-vs-Mali int8 GEMM gap is 32× (555.8 vs 17.5 GMAC/s), not the 18× stated above.

L3 cache for weight residency — no, for three independent reasons

• The NPU is a non-coherent DMA master. /proc/device-tree/npu@fdab0000 has no dma-coherent property, and the driver requires explicit RKNPU_MEM_SYNC cache maintenance. NPU reads do not snoop the CPU cache hierarchy, so they cannot hit in L3 however the page is mapped. Making them hit would require a coherent (snooping) port into the DSU — a silicon property, not something a kernel change or a fixed-page assignment can confer. This is exactly why the driver must flush before every submit.
• L3 has no physical address to map. An IOMMU maps IOVA→PA; a cache is not in the physical address space, so there is nothing to put in a page table entry. Contrast syssram, which is a real addressable region at its own PA — which is precisely why TRY_ALLOC_SRAM can map it.
• No mechanism to pin it, and it is too small anyway. No /sys/fs/resctrl; MPAM is ARMv8.4 and RK3588 is A76/A55 (v8.2). L3 is 3 MiB shared across all 8 cores — smaller than the single 4 MB weight tile used in these tests, and ~0.03% of a 9.67 GB weight set. The same size wall that killed SRAM, one level up.

What this means for heterogeneous decode dispatch

• The hard ceiling is ~30 GB/s of weight streaming, so max decode ≈ 30 GB/s ÷ (weight bytes per token). A 1.7B Q8_0 (~1.7 GB/token) tops out near ~17 tok/s; the measured 12.8 is already ~73% of the wall. A 7B Q8_0 (~7 GB/token) tops out near ~4.3 tok/s, and the recorded 0.89 is ~20% — so the headroom is model-dependent, and largest exactly where decode is slowest.
• For memory-bound work the lever is memory-level parallelism, not engine count. Single-engine decode leaves the bus 40–80% idle because it is latency-bound; CPU+NPU together close that to ~98%.
• For a mixed workload, a third engine is real throughput. A compute-bound op — speculative/draft verify at M>1, or anything whose working set stays on-chip — runs on a third engine essentially free while the memory-bound decode stream continues at full rate. This is the configuration the speculative-pipeline direction already implies, and it is now measured rather than assumed.
• Dispatch on resource profile, not engine count. The scheduling question is "what does this op consume", and the payoff comes from pairing ops that consume different resources.
• Pair this with Exp-2026-09-05 Fixed Per-Task Cost: getting to the ceiling also requires the NPU's host-side per-task cost not to inflate under CPU load, which is what the precompiled regcmd cache fixes.

Caveats

• Mali's arm is a synthetic coalesced stream, not real model work. It establishes Mali as a first-class memory client (22.66 GB/s solo) and settles the bandwidth question; it does not evaluate Mali for any particular op.
• engine_mix_probe's CPU GEMV arm is a dequant-and-MAC emulation of decode, calibrated against test_nf4_decode's known per-core rate, not the production kernel.
• DMC load occasionally reports >100% on a first sample (a 122% peak was observed once); the averages are consistent and are what the conclusions rest on.
• These are throughput probes and do not check correctness.