⚠️ SUPERSEDED for models ≤ physical RAM (2026-06-28): the "only ~4 GiB DMA-mappable → must stream" premise of this page is now known to be per IOMMU-domain, not per-device. The rknpu driver exposes up to 8 domains (~4 GiB each) → ~32 GiB residable without streaming. See Multi-Domain Residence. Streaming remains relevant only for models that exceed physical RAM; for anything that fits RAM, multi-domain residence (no streaming, zero per-token churn) is the path.

Tools: src/npu.c (ork_mm_free, ork_w_bytes, ork_w_dump, ork_mm_load_i8, chunked weight arena), ggml-ork.cpp (budgeted-LRU wcache, .orkpack read/write, ORK_EVICT_SRC), tools/dma_probe.c · tools/stream_probe.c · tools/persist_probe.c (RK3588, governors pinned)

The RK3588 NPU's IOMMU can address only ~4 GiB at once (a hardware limit, not software). That makes a full int4 27B (~13.5 GB) or int8 9B (~9 GB) impossible to keep resident. We added (1) layer-streaming — cycle weights through the 4 GiB window with an LRU that reclaims IOVA on eviction — so a model larger than the window runs on the NPU, and (2) .orkpack persist — a self-populating on-disk cache of pre-tiled weights, so the expensive dequant→quant→tile conversion is paid once and every subsequent load is a plain DMA copy. Validated end-to-end: a forced-streaming 1.7B reloads 5.2× faster from .orkpack (160 s → 31 s) with byte-identical output.

Repeated bcreate (DRM_IOCTL_RKNPU_MEM_CREATE) failures (EFAULT / kernel failed to allocate IOVA: -12) on big models turned out not to be system RAM. tools/dma_probe.c allocates 64 MB DMA buffers in a loop with no model loaded:

mode	result
CREATE-only (raw IOVA)	FAIL at buffer #63, 4.23 GB (errno 14 EFAULT)
CREATE+mmap (real path)	FAIL at buffer #63, 4.23 GB (identical)

Both modes fail at the same point → it's the IOMMU input address space (32-bit = 4 GiB), not host RAM (31 GB) or mmap space. Confirmed in the device tree: the NPU sits behind rk_iommu (rockchip,iommu-v2). The kernel drivers/iommu/rockchip-iommu.c shows v1 and v2 use the same 32-bit virtual-address decode (rk_iova_dte_index/pte_index); v2's "40-bit" only widens the physical page address in PTEs (to reach RAM >4 GB), not the device-generated IOVA. Others hit the same wall in the closed stack (rknn-llm #335: a 3 GB model → errno 14 Bad address).

Consequence: "resident on the NPU" = "IOMMU-mapped," and ≤4 GiB can be mapped at any instant. A dma_buf can be 17 GB in system RAM, but mapping it for the NPU consumes IOVA equal to its size — so you can map ≤4 GiB and offset-address within it; beyond that you must swap (remap). This is also why rkllm streams: it keeps weights in CPU RAM and DMAs each layer's working set through the window per compute step (which is why its decode is bandwidth-bound).

ork_w_free historically only freed the Bb array; the device buffers leaked until fd close (the resident-forever model). Streaming needs the window back:

• ork_mm_free(c, w) — frees a weight AND bdestroys its DMA buffers, reclaiming IOVA. Gated on ork_w.owns (=1 for per-tile pack/pack_i4/pack_i8; arena-view weights are owns=0 and left to teardown).
• tools/stream_probe.c proves it: 100 × 268 MB packs/frees = 26.8 GB cycled through the 4 GiB window, ≤256 MB resident (it sails past the iter-16 / 4 GB point where the old leak died).
• ggml-ork budgeted-LRU wcache (ORK_WCACHE_BUDGET_MB, default 3072): tag each weight last_use; before a pack that would exceed budget, evict the LRU weight via ork_mm_free. 9B w8a8: 0 IOVA-wall hits, 352 evictions, resident bounded — a 9 GB model ran on the NPU.

In one prefill pass each weight is packed once and reused for the whole token batch before aging out, so the DMA amortizes over M — the verify/prefill shape. Decode (M=1) would re-stream every layer per token (bandwidth-bound), so the big model is verify/prefill-only; decode runs on a small resident draft.

Streaming alone still re-does dequant→quant→tile every time a layer cycles in. The packed bytes are deterministic, so cache them:

• ork_w_dump / ork_mm_load_i8 (ork-driver): serialize a weight's tiled Bb bytes, and reload them straight into DMA (bcreate + memcpy + bsync) with no dequant/quant/tile. tools/persist_probe.c: byte-identical roundtrip, load 8.3× faster than pack (16 MB weight: 105 ms → 13 ms) — and that's only the tile→memcpy delta; reload also skips the upstream Q4_K dequant + int8 quant.
• .orkpack format (ggml-ork): magic ORKPK01, a name-keyed index (tensor name → K,N,dtype,blob_off/size,bscale) + tiled-bytes/bscale blobs + a footer. Self-populating via ORK_PERSIST=<path>: reads (mmap → load) if present, else builds a .tmp and atomically renames on finalize. Per-weight (K,N,dtype) is re-checked on load, so a stale file can never feed wrong weights (mismatch → repack). Wired into the LRU at the int8 site: load-on-hit, pack+dump-on-miss, evict via ork_mm_free.

Packing held both the source GGUF (~mmap) and the packed copy resident (~30 GB peak → OOM). ORK_EVICT_SRC=1 madvise(MADV_DONTNEED)s each source plane once it's packed (clean, file-backed → re-faultable). Peak RAM 5.7 GB vs 30 GB. (Fixes RAM, not the IOVA cap — those are independent.)

End-to-end, forced streaming (1.7B, ORK_WCACHE_BUDGET_MB=1000 < 1.7 GB so eviction engages on both passes):

pass	wall	weights	output
CONVERT (stream-by-pack, builds .orkpack)	160 s	loaded 0 / packed 336	"one word. The"
RELOAD (stream-by-load from .orkpack)	31 s (5.2×)	loaded 336 / packed 0	"one word. The" (identical)

0.6B roundtrip likewise byte-identical (loaded 112 / packed 0 on reload).

✅ A model larger than the NPU's 4 GiB address window runs on the NPU (streaming), and the conversion cost is paid once then skipped (.orkpack). This is the foundation for a streamed 27B verify in a speculative-decode prefill path (draft stays resident). The 4 GiB cap is a permanent design constraint — full-model residency for 27B/MoE is off the table; stream the verifier instead.

ORK_PERSIST=<path> (read if present, else build) · ORK_WCACHE_BUDGET_MB (default 3072) · ORK_EVICT_SRC=1 · always --no-repack (keeps weights host so the offload fires). Don't set ORK_VERBOSE when timing (per-op fprintf inflates it).

• pack→dump→free convert path (no caching) — the one-time CONVERT is slow partly because it streams and runs a wasteful decode; a dedicated convert that packs→dumps→frees each weight once would be faster.
• A ggml-ork DMA-buffer type so input zero-copy reaches end-to-end.
• DMA/compute overlap (prefetch layer N+1 while N computes) to keep streamed prefill compute-bound.
• Wire the streamed 27B verifier into the EAGLE-3 spec-decode path.

DDR headroom (measured 2026-06-25, RK3588, governors pinned). Sampled dmc/load + rknpu/load through a sustained 1.7B w8a8 prefill: during NPU-busy samples DDR load was median 2%, max 13% (/sys/class/devfreq/dmc/load), and NPU cores sat at 2–3%. So the NPU is submit/latency-bound, not bandwidth- or compute-bound — there is ~90%+ spare LPDDR bandwidth during operation. (Method: tools-free — a shell sampler around llama-completion, correlating DDR% with non-zero core load.)

Implication: a prefetch DMA hides trivially. Streaming weights through the 4 GiB window can run a prefetch concurrent with NPU compute and the transfer disappears into the idle bus → streamed prefill ≈ resident prefill (the transfer leaves the critical path). The pessimistic "2/3 floor" only applies if the bus were saturated; with 90%+ headroom the effective floor is near 100%.

Design — (N_cores + 1)-slice double/triple buffer. Split the resident budget into slots; a prefetch thread fills slot i+1 from the .orkpack mmap while the NPU computes slot i. With 3 cores, 4 slots keeps all cores fed (3 ready) + 1 always loading → ~0 NPU downtime, provided T_load(slot) ≤ T_compute (holds for prefill, large M; not decode M=1 — decode stays on the resident draft). The 3 cores need independent units: within-matmul K/N tiles (ork-driver already tiles) or MoE experts.

MoE refinement — slot = expert native size. For sparse MoE, size slots to the expert (the independent unit), not 4 GiB/(n+1). When expert_size ≤ 4 GiB/(cores+1) the expert is the divisor → 4 GiB/expert_size slots (≥ cores+1; many more for small experts → deep prefetch queue, no fragmentation). The router (run for the prefill batch) names the selected experts → prefetch exactly those. This hides the per-expert transfer (the current MoE-offload wall) under compute — potentially flipping MoE-on-NPU viable. Fall back to tiling only if an expert exceeds 4 GiB/(cores+1).

Large experts (e.g. 1.5 G → only ~2 slots fit): keep the expert as the divisor and recover overlap by pipelining within the transfer. (a) Prefill/verify needs no prediction — the batch router is known, so double-buffer at expert granularity and stream the next expert's tiles into a slot as the current expert's tiles are consumed/freed (ork-driver already K/N-tiles each expert) → the transfer starts before the slot fully frees. (b) Decode/online: the next expert isn't known until the router fires, so speculatively prefetch the predicted expert (MoE routing is skewed — hot experts + cross-layer locality give a usable hit rate) and pipe its first tiles early; a wrong guess costs only spare bandwidth (abundant — 90%+), so speculate aggressively. Decode runs on the draft, so (b) is a secondary win.