NPU on-chip SRAM (the "second memory interface")

The RK3588 SoC has a small on-chip system SRAM (sram@ff001000, 956 KB = 0xef000) that the NPU can use as a DMA-able buffer alongside DRAM. This page records how to enable it, that it is usable, and the (negative) result of the first lever we tried against it.

Enabling it

Stock DietPi/Armbian vendor kernels ship it disabled for the NPU two ways at once:

1. # CONFIG_ROCKCHIP_RKNPU_SRAM is not set — the driver's SRAM support is compiled out.
2. The device tree hands the system SRAM to the video decoder (rkvdec), not the NPU.

So out of the box the driver reports 0 bytes of NPU SRAM. Enabling it requires rebuilding the board's kernel with two changes:

• Kernel config: CONFIG_ROCKCHIP_RKNPU_SRAM=y (the gate also needs CONFIG_NO_GKI=y, which vendor kernels set, and the NPU in IOMMU mode — the default).
• Device tree: give an SRAM region under the mmio-sram controller to the rknpu node via a rockchip,sram = <&…> phandle, and drop the conflicting rkvdec claim. Simplest (if you don't need hardware video decode): expand rkvdec0_sram to the whole 0xef000 region, remove rkvdec1_sram and both rkvdec rockchip,sram refs, and point rknpu at it.

Step-by-step (source pin: armbian/linux-rockchip branch rk-6.1-rkr5.1 = 6.1.115; DietPi ships this exact vendor kernel) is in the repo README, section "Enabling the NPU on-chip SRAM". Verify with tools/sram_probe.c (queries RKNPU_GET_TOTAL_SRAM_SIZE): 0 before, 978944 bytes (956 KB) after.

It is allocatable / usable

tools/sram_alloc_probe.c proves a real allocation lands on-chip: a 256 KB MEM_CREATE with RKNPU_MEM_TRY_ALLOC_SRAM (flags 0x503 = NON_CONTIGUOUS|CACHEABLE|IOMMU_LIMIT|TRY_ALLOC_SRAM) reports sram_size = 256 KB, the free pool drops by exactly 256 KB, a real IOVA (0xfff80000-range) is assigned, and destroy restores the pool.

Gotcha: iommu_domain_id must be a real domain (e.g. 0); the driver rejects -1 from a fresh context (invalid iommu domain id: -1 … failed to switch iommu domain, ret: -22). ork-driver's own packs work because they resolve -1 → ork_dom_default() = 0. The driver maps cache(SRAM)+DDR into one IOVA range (SRAM low, DDR high) via rknpu_gem_alloc_buf_with_cache(RKNPU_CACHE_SRAM); a SRAM alloc is visible in dmesg as RKNPU: allocate iova start: 0xffc4xxxx (the high SRAM IOVA range).

Lever tried: stage int8 matmul weights in SRAM → NO speedup

Hypothesis: the single-core int8 matmul is weight-DMA-bound (each M-tile submit re-streams the whole K×N weight from DRAM ~11 GB/s), so a weight tile resident in SRAM should cut the per-submit weight read.

Measured (governors pinned perf@2112 MHz, mc_prof A/B, ORK_WEIGHT_SRAM OR-ing TRY_ALLOC_SRAM into the weight buffer alloc):

shape	note	DRAM 1-core	SRAM 1-core
M=256, K=512, N=1024	single M-tile (no re-stream)	1100 µs	1099 µs
M=4096, K=512, N=1024	~6 M-tiles, weight re-streamed 6×	17373 µs	17411 µs

Identical. Why: submit ≈ 1783 µs for a 512 KB weight, but a DRAM read of 512 KB @ 11 GB/s is only ~47 µs — <3 % of the submit. The submit is bounded by fixed per-submit cost (ioctl/regcmd/bsync) and the NPU's on-chip CDMA→CBUF weight-load path, whose rate is the same regardless of source memory — not DRAM bandwidth. Fundamental tension: weights big enough to be genuinely DRAM-bandwidth-bound (real LLM layers, MBs) don't fit in 956 KB; weights that fit are submit-overhead-bound. (Also plausible: RK3588 syssram is not higher-bandwidth than LPDDR5.) The code hook was reverted; the enablement + probes stayed (real capability).

Where SRAM may still earn its keep

• SDP/PPU LUTs + coefficient tables — the SDP (regcmd domain 1001) does LUT-based nonlinear activation; a custom LUT implements an arbitrary elementwise f(x) the NPU has no native op for. That is a small, hot, resident buffer — exactly SRAM's shape (residency/capability, not bandwidth). Blocked on the PPU compute-replay RE (see the PPU activation notes) — SRAM gives that effort an on-chip home.
• Chained-pass scratch — intermediates between composed regcmd passes.
• RK3576 has a real 1 MB nbuf (RK3588's nbuf_size = 0); RKNPU_MEM_TRY_ALLOC_NBUF may behave differently there.

Not a lever for the int8 weight-DMA path.

---

Re-verified 2026-09-04 — SRAM still works; it is just SLOWER than DRAM

Two questions came up: did our kernel work (patches #patch64…#patch74, the IOMMU rework) break SRAM, and why did we ever measure only ~3 GB/s from something on-chip? sram_bw_probe on kernel 6.1.115 #72:

SRAM total=956KiB free=764KiB          (weight tile=128KiB, ORK_WEIGHT_SRAM=1)
after pack:  SRAM free=636KiB          consumed exactly 128KiB -> weight IS in SRAM
[1] NPU solo:       3049 submits/s
[2] CPU DRAM solo:  10.5 GB/s
[3] concurrent:     NPU 2821 (93% of solo) | CPU 8.8 GB/s (85% of solo)
==> SHARED memory controller (partition limited — both degrade under contention)

NOT broken. The allocation lands, the free pool drops by exactly the request, and destroy restores it. Nothing in the kernel work regressed it. (A separate observation that day — c->mrc carrying TRY_ALLOC_SRAM and not landing — is one allocation failing over, not the capability being gone. Do not conflate the two.)

Why on-chip SRAM is SLOWER than DRAM here

This is the counterintuitive part, and it is the reason every SRAM lever has come out flat:

path	bandwidth
CPU ↔ DRAM	10.5 GB/s (measured 2026-09-04)
CPU ↔ SRAM, cacheable	~3 GB/s
CPU ↔ SRAM, uncached	~1.2 GB/s

• syssram is a system SRAM block, not a cache — a narrow port on an interconnect path sized for firmware/control buffers.
• RK3588's DRAM is a wide multi-channel LPDDR4X/5 interface. Width and channel parallelism beat proximity for anything streamed.
• The mapping dominates: uncached kills burst coalescing (1.2), cacheable triples it (3), and it is still 3.5× under DRAM.

So on-chip buys LATENCY and freedom from DRAM contention — never throughput. Corollaries, all measured:

• Weights in SRAM: flat. They are streamed, and 956 KiB cannot hold a real layer anyway.
• regcmd in SRAM: no measurable effect (A/B 2026-08-26; an apparent win was a false positive caught by an A-B-A control). The NPU reads a regcmd once per submit — 108 registers, ~432 bytes — latency-bound and invisible against a ~150 µs job, while any CPU writes to it get slower.
• Task descriptors in SRAM: HARD WEDGE (2026-09-04, plug power-cycle to recover) and counterproductive by construction — the CPU rewrites descriptors every call, and CPU→SRAM is the slow direction. See the SRAM rule in Optimization-Roadmap: 0x403 NPU-read is SRAM-safe, 0x40b kernel-read is not.
• Same phenomenon from the other side: the precompiled-chain work needs its outputs in DMA memory, which is uncached. Reading them with a scalar loop cost 6144 uncached accesses — host-silu 45 → 311 µs — fixed by one burst memcpy into a cached mirror. Never scalar-read uncached memory in a hot loop.

And per row [3]: even with the weight in SRAM the NPU and CPU still degrade each other (93% / 85%), so SRAM is not a separate memory path in practice. That closes the "partition the NPU onto SRAM while the CPU uses DRAM" idea on measurement rather than theory.

⚠ ORK_SRAM_REGCMD false positive — reproduced a SECOND time, 2026-09-04

The 2026-08-26 entry says an apparent win was "a false positive caught by the A-B-A control". That trap caught a second person, so it is worth spelling out.

Re-tested at M=1 decode (down shape, K=3072 N=1024) with the doorbell phase split, which the original A/B did not have. The single-shot A/B looked like a clear win:

	DRAM	SRAM
total	0.461 ms	0.403 ms (1.14×)
begin	249.4 µs	220.3
end (post)	134.6 µs	105.6 (1.27×)
kernel hw/job	148.6 µs	129.6 (1.15×)

Three-trial replication reversed it completely:

trial	DRAM	SRAM
1	0.408	0.458
2	0.421	0.432
3	0.401	0.431

Ranges overlap entirely (DRAM 0.401–0.461, SRAM 0.403–0.458). Verdict unchanged: no measurable effect.

The tell that should have stopped it earlier: the single-shot showed kernel hardware time improving 15%. Where the regcmd lives cannot plausibly speed the NPU's compute by 15% — a physically implausible component of a delta is a noise signature, not a mechanism.

Placement is NOT the doubt. ORK_SRAM_REGCMD=1 genuinely places it — the driver prints 956 KB of the regcmd landed in on-chip SRAM. The capability works; it just does not pay.

Why it does not pay — corrected twice, so take the third version.

Wrong argument #1 (bandwidth): syssram is only 1.2–3 GB/s. True, but irrelevant — a descriptor ring is not bandwidth-bound, and syssram is designed for descriptor rings. Bandwidth kills weights-in-SRAM, not this.

Wrong argument #2 (read once): "the NPU reads a regcmd once per submit, so there is nothing to win." False, and the code proves it. ork_dyn_halt writes 0x0014 = amount 0 into a not-yet-executed program's regcmd after the submit, flushes it, and the sequencer stops there. An in-flight halt cannot work if the chain were slurped at submit time. Per AGENTS.md the kernel programs the PC from the FIRST task only and the hardware walks the chain per task, fetching each program's 0x0010 (next regcmd addr) and 0x0014 (next register-amount) as it reaches it. So regcmd reads are per task, at task-load time, on the sequencer's path.

The actual reason — overlap: ping-pong (RKNPU_JOB_PINGPONG, 1<<2) double-buffers the register banks and "swaps them the instant a task's register config is done", so task N+1's registers load while task N computes. The fetch is on the sequencer's path but OFF the critical path, so where it lives is invisible. That is consistent with the flat measurements and, unlike the first two arguments, does not contradict how the chain demonstrably behaves.

Wrong argument #3 (overlap) — TESTED AND FALSIFIED 2026-09-04. The overlap story predicted that with ping-pong off the register load would be exposed and SRAM placement would then pay. It does not.

The first attempt at this test had no power and is worth recording as a trap: run at the down shape it produced 3.0 jobs/call — one program per core, so there is no chain to walk, nothing for ping-pong to overlap, and no per-task regcmd fetch beyond the PC's first. All four cells were flat because neither variable could act. Check the program count before believing a chain-walk measurement.

Redone on a genuine 6-program chain forced onto one core (ORK_CHAIN_NC=1, K=1024 N=1024; the S-scaling 0.375→0.693 ms confirms serial execution), ms per 6-task batch, 3 trials each:

	regcmd DRAM	regcmd SRAM
ping-pong ON	0.695 0.693 0.693 (0.694)	0.706 0.695 0.712 (0.704)
ping-pong OFF	0.694 0.695 0.712 (0.700)	0.684 0.684 0.694 (0.687)

Total spread across all four cells is 2.5%, within-cell ~2%, and SRAM's sign flips between the ping-pong rows — the signature of noise, not a mechanism.

The more telling result: ping-pong itself is also flat (0.694 vs 0.700). If removing the register-bank overlap costs nothing measurable, the register load was never on the critical path in either mode, so there was never anything for SRAM to hide.

What survives all three attempts is the magnitude argument. A program's regcmd is ~1 KB against 115 µs of execution per program (0.693 ms / 6). Even at full DRAM latency that fetch is well under 1%. The timing model matters and is worth getting right — reads are per-task at walk time, proven by the in-flight halt — but the count and the overlap are both irrelevant because the quantity is negligible.

ORK_SRAM_REGCMD is closed. Three independent mechanisms proposed, all wrong or immaterial; two false-positive "wins", both killed by replication. The knob works, places 956 KB, and buys nothing.

Measurement rule for this board: these probe timings and decode tok/s carry roughly 10% run-to-run spread. A single A/B here is worthless. Use ≥3 trials, or A-B-A. Two separate investigations have now produced an apparent SRAM win that vanished on replication.

Aggregate bandwidth: SRAM IS a separate port, but the NPU-weight case is flat — 2026-08-11 / 2026-09-05

A recurring hope is that running the NPU out of SRAM while the CPU streams DRAM raises aggregate bandwidth, since they would use different memories. sram_bw_probe, 3 trials each, concurrent throughput as a percentage of each engine's solo rate:

weight in	t1	t2	t3
DRAM	90% / 78%	90% / 85%	90% / 78%
SRAM	90% / 77%	89% / 84%	97% / 76%

Identical within noise. Both configurations degrade the same under contention (~90% NPU, ~80% CPU), so the ~1.7× aggregate comes from having two engines, not from SRAM. A single trial had suggested SRAM was worse (98/84 → 90/77); replication showed that was noise as well.

Why it is flat here is NOT "SRAM shares the DRAM controller." That explanation was written on 2026-09-05 and is wrong — it was directly falsified on 2026-08-11:

SRAM IS a separate port (confirmed, A76 core 4): CPU→SRAM cacheable+flush held at 104% (2.89 → 2.99 GB/s) while three big cores saturated DRAM — completely unaffected.

So SRAM traffic genuinely does not spend DRAM's budget, and additivity is real. What the 2026-09-05 table above shows is narrower: placing the NPU's weight in SRAM does not help, and the ~90%/~80% mutual degradation is therefore not DRAM-bus contention being relieved. Two candidate explanations, both open:

• the NPU side at this size is submit-overhead-bound, not weight-bandwidth-bound (measured 2026-08-11: ~331 µs for what should be ~18 µs of streaming — ~95% overhead), so weight placement cannot move it; and
• the residual mutual slowdown may be fabric/interconnect or scheduler contention that is shared even when the memories are not.

RESOLVED — measured, not asserted (tools/sram_port_probe.c, 2026-09-05, kernel 6.1.115-vendor-rk35xx-fence; full record: [[Exp-2026-09-05 SRAM Aggregate Bandwidth|Exp-2026-09-05-SRAM-Aggregate-Bandwidth]]). Re-run with the NPU driven into a demonstrably weight-bandwidth-bound regime (16 tasks chained over a 640 KiB weight, 73% streaming fraction, verified by an in-run fit that voids the run below 70%) and with a SPIN contender to divide out scheduling:

place	solo	under SPIN	under LOAD	memory-attributable
DRAM weight	20.54 GB/s	100.0%	13.85 GB/s	67.5%
SRAM weight	20.64 GB/s	100.0%	13.97 GB/s	67.7%

Δ +0.2 pp, replicated at three weight sizes with the sign flipping. So the answer is the first bullet above, and it is now directional: SRAM is a separate port for CPU access and NOT for NPU weight reads. Never write the unqualified sentence again — it has now been wrong in each direction.

The residual contention also localises: the weight stream slows 1.22× under a 23.5 GB/s CPU load while the fixed DRAM-resident per-task remainder (A/C/descriptors) slows 2.19×. Most of the NPU's loss was never the weight stream — and that remainder cannot move to SRAM, since task descriptors in SRAM is a hard wedge.

And the CPU-side numbers are about the mapping, not the memory. Through the default uncached (pgprot_noncached) mapping, SRAM and DRAM are identical — 1.19 GB/s write both. The "~3 GB/s" figure is the true CPU→SRAM feed ceiling once the data actually lands (write-combine 3.17; cacheable + dc cvac flush 2.77), and it is affinity-insensitive (A55 ≈ A76), unlike the ~21 GB/s cacheable write-only number, which is a cache illusion (writes sitting in L1/L2). Any claim that "on-chip SRAM is slower than DRAM because it is a narrow port" is unfounded — see Performance and the test_sram_bw / test_sram_npubw / test_sram_pipe probes (2026-08-11).

The separate CPU+GPU+NPU tensor-split aggregate result (Performance) is a different lever again — that one is engine count.

So what IS on-chip SRAM good for here?

Being straight: for the weight and regcmd cases, nothing measurable. The one role that measured real and is still open is an activation pipe: separate port (confirmed), contention-free, but it feeds at only ~3 GB/s and holds ~700 KiB usable. It pays only if the activation traffic routed through it is worth ≤3 GB/s and the DRAM contention avoided is worth more than the slow feed. Weights are out on size, not bandwidth (700 KiB usable vs 9.67 GiB resident = 0.007%). Its other properties are

1. Memory that costs no DRAM and no IOVA. The one deliberate use in the tree — ork_pc_compile parks its regcmd pool there because it is "otherwise-unused memory, so residing there frees DRAM/IOVA for the rest of the pipeline". Real, but 956 KB against the 4 GiB IOVA window is 0.02%; it matters only at the edge.
2. Low latency for small always-available objects — which is what syssram is designed for, and we have no latency bottleneck: the M=1 per-call cost is CPU work (synth, bsync, descriptors), not memory latency.

Every lever tried against it has measured flat or worse — weights (streamed, and 956 KB holds no real layer), regcmd (negligible against 115 µs/program), descriptors (hard wedge; kernel-read), and the partition above. Do not spend more on it without a mechanism that predicts a win in latency of a small hot object, which is the only thing it is actually better at.