Exp-2026-09-05 — Memory hierarchy, engine dispatch, and weight reuse (session record)

One session, one thread of reasoning: why is the RK3588 NPU stuck at ~18% of its 6 TOPS, and can any memory-hierarchy trick fix it? Four candidate levers were chased to a measured conclusion, three closed negative and one reopened a verdict that had been closed for weeks. Detail pages: SRAM aggregate bandwidth · fixed per-task cost · heterogeneous dispatch ceiling.

Kernel 6.1.115-vendor-rk35xx-fence, board 10.3.0.236, DMC 2112 MHz. Results from other kernels are not comparable.

---

The findings

#	Question	Answer
1	Is SRAM a separate memory port?	Directional. Confirmed for CPU access (104% retention under DRAM saturation); null for NPU weight reads (67.7% vs 67.5%, 3 sizes, sign flips).
2	Does the fixed per-task cost live on the NPU?	No — it is HOST work. Kernel hardware time for the fixed part is flat under DRAM load; 113% of its 2.23× inflation is host-side. ork_pc_* removes 72–80% of it.
3	Does a 3rd engine add throughput?	Only if its work is compute-bound. Memory-bound: no (bus saturates ~30 GB/s at 2 engines). Compute-bound: yes, essentially free (Mali 99.9% + CPU/NPU 99.3% simultaneously).
4	Is Mali a compute peer?	No. 17.5 GMAC/s vs the NPU's 555.8 — 32×.
5	Can L3 hold weights?	No, three independent reasons (non-coherent master, no PA to map, 3 MiB).
6	Is on-chip accumulator retention the missing mechanism?	No. K-slicing is monotonically worse (0.96/0.65/0.39× at S=2/4/8) before accumulation cost, so nothing to unlock.
7	Does HW WEIGHT_REUSE work?	YES on the colsplit M-tile path — bit-exact and up to 1.52×, reopening #39.

The one that paid: WEIGHT_REUSE (#39 reopened)

#39 closed cross-tile weight reuse as unachievable — the bit skipped the weight re-DMA but computed wrong. That was measured entirely on the fold path, whose 0x1040/0x1030 are baked from a K=3584 N=1216 capture, so K and N are not patchable and the weight was always 4.36 MB.

On the colsplit path 0x1040 is synthesised from mc, and every M-tile already re-streams the same K×segw weight from one address. Setting WEIGHT_REUSE there (ORK_MTILE_WR=1) is bit-exact against an exact int32 CPU reference and up to 34% faster (1.52×) at N=8192, with the gain growing monotonically in N. The saving equals the skipped weight bytes at 10.5 GB/s — the known single-core weight-stream rate.

This is the largest single win measured in the session, and it lands on wide-N prefill, which is where colsplit runs in production. It is off by default pending make test and a real end-to-end prefill.

The corrections this session forced

• "37.4 GB/s aggregate / 1.5×" was wrong — it summed two different currencies (NPU useful weight bytes
  • CPU read+write traffic) and exceeded the theoretical LPDDR peak. Real dual-engine gain: 1.41% in the decode-shaped regime, 1.10× when the CPU already streams well.
• NPU compute was understated ~1.7× — ork_i8_mm_run_chain with chained tasks gives 322 GMAC/s; large-M ork_i8_mm_run gives 555.8 GMAC/s (1.11 TOPS, 18.5% of 6 TOPS). Quote the latter.
• "SRAM is not a separate port" — invented to explain a flat result, contradicted a measurement already on disk in another project's history. Retracted.
• "Weight traffic scales K²/S so K-slicing wins S×" — the output stage scales with S and swamps it.
• "WEIGHT_REUSE doesn't work" — flattened "fails on the fold path at 4.36 MB" into a false general claim.

Method lessons (each one nearly produced a false result)

1. Verify the knob fires before believing a null. A stale ccache build faked five bit-exact rows from a binary that had no knob compiled in. touch + rebuild, and count the writes.
2. A contention probe must verify its own contender. At -O3 the compiler deleted the saturator's memcpy; the hog "reached" 7,056,858 GB/s while generating no traffic, and both arms dutifully reported 99.9% retention of nothing.
3. Never explain a null before proving the experiment could have shown a positive. A flat A/B where the manipulated variable governs ~7% of runtime is not evidence about that variable.
4. "Matches the baseline" is not "is correct". Both arms were upgraded to check against an exact CPU reference on a strided sample covering every output tile.
5. Use one currency, and check it against a hardware counter. /sys/class/devfreq/dmc/load reconciled every configuration to ~33 GB/s at 100% — the composite figure never did.
6. Positive controls are worth their cost, but bound the cost. Clobbering 0x1040 proved the write reaches hardware — via a 60 s job timeout and six soft resets. Kept in-source, not run by default.
7. Cross-project findings need an anchor in both indexes. The SRAM record lived in the llama.cpp session history, which is why this ground was re-covered.
8. A physically implausible component of a delta is a noise signature, not a mechanism.

What is now closed, and what is open

Closed with measurements: SRAM for NPU weights · L3 for weights · accumulator retention / K-slicing · Mali as an int8 GEMM peer · a 3rd memory-bound engine.

Open and worth pursuing:

• WEIGHT_REUSE on colsplit: why it tolerates reuse far above the nominal WEIGHT_BANK size, and whether it can be defaulted on (needs make test + end-to-end prefill).
• The 2.19× host-side per-task contention cost — ork_pc_* removes most of it and is worth more under CPU load than any quiet-system benchmark shows.
• Raising M (speculative/draft verify) remains the only lever that changes the class of the decode problem: at M=1 the work is 1 MAC/byte and every engine is equally bus-bound.