Exp 2026 06 26 DMA Buf Fill Bandwidth - oRKLLM/ork-driver GitHub Wiki

dma_buf Fill Bandwidth β€” the write-combine wall, the cacheable fix, and direct int4β†’int8 inflate

Tools: tools/prefetch_headroom.c (f959e49) Β· tools/disk_stream_bench.c (f3ef81a) Β· tools/dmabuf_fill_probe.c (c70f71d) Β· src/npu.c (ork_dma_alloc flags, b53c59b; expand_chan_i4_i8/inflate_chan_nf4_i8/tile_i8_to_tiles, ORK_DIRECT_I4, 570cb68). RK3588 10.3.0.236, governors pinned.

TL;DR

Streamed prefill (the path for models bigger than the 4 GiB NPU window β€” see Layer-Streaming and Persist) is per-slice-prep-bound, not matmul-bound, at M≀128. The prep was pinned at a flat ~1 GB/s β€” and that turned out to be an artifact of benching a write-combine (non-cacheable) buffer, not the RAM. The production weight tiles already use cacheable memory; benched correctly, fill is ~10 GB/s (8.3–8.8Γ—). A second lever β€” a direct int4β†’int8 tiled inflate that skips the f32 round-trip β€” adds 2.2–4.9Γ— on the inflate, bit-identically. The optimization chain compounds: cacheable buffer (8Γ—) β†’ direct inflate (2–5Γ—) β†’ clean-only bsync (the identified NEXT lever).

1. Prefetch headroom: prep is 6–26Γ— the matmul, and it's transfer-bound (prefetch_headroom.c, f959e49)

Per-slice prep (int4 inflate + tile into the resident DMA buffer) is 6–26Γ— the NPU matmul at M≀128, and is TILE/transfer-bound, not inflate-bound. The structural reason: t_tile is M-independent (it moves the same weight bytes regardless of batch), while t_npu ∝ M. So:

The prefetch-viable crossover is large-M (prefill) β€” NOT decode (M=1) or small-M verify. At small M the prep dwarfs the compute and can't be hidden behind it.

This focuses streaming on the prefill/verify path and pinpoints the tile transfer into the DMA buffer (not the nibble inflate) as the thing to optimize.

2. disk / mmap: only matters when the model exceeds RAM AND the page is cold (disk_stream_bench.c, f3ef81a)

source vs RAM
mmap-warm == RAM (page cache fully hides disk, 0.5% delta)
cold-disk via mmap (readahead) ~1.4Γ—
cold-disk via blocking pread ~2.2Γ—
NVMe cold-read ceiling ~0.5 GB/s

So disk is irrelevant to a .orkpack that fits in the page cache; it only bites when the model exceeds RAM and the page is cold. (Reinforces that the .orkpack mmap path is the right design.)

3. THE WRITE-COMBINE WALL + the fix (dmabuf_fill_probe.c, c70f71d)

The dma_buf fill measured a flat ~1.06–1.14 GB/s regardless of method. Root cause: NOT slow RAM β€” the buffer was mapped non-cacheable / write-combine, so CPU writes bypass the cache.

Correction (important): that ~1 GB/s was an artifact of benching ork_dma_alloc's zero-copy path (flag 0x401, deliberately set 0x403β†’0x401 in b53c59b "test 0x401 dma flag"). The production weight tiles (pack / tile_f32_i8 / ork_mm_load_i8) already use CACHEABLE 0x403 β€” they were never on the WC wall.

Benched on the cacheable flag:

fill path bandwidth
write-combine 0x401 (the artifact) ~1.06–1.14 GB/s
cacheable 0x403 + clean-before-submit bsync ~10 GB/s (8.3–8.8Γ—) β€” NPU read verified correct
reference: plain cached memcpy ~12 GB/s

The clean flush (the bsync TO_DEVICE before submit) is ~15–18% of the combined cost. NEON non-temporal (stnp) stores do NOT help the WC path β€” the uncached write itself is the wall, not store ordering. The cacheable flag is RKNPU_MEM_CACHEABLE in the uABI.

Candidate change: revert ork_dma_alloc's input path 0x401 β†’ 0x403 (cacheable + clean-before). The output ORK_ZC_OUT case is different β€” it needs invalidate-after and is the known unsafe-under-concurrent-multicore path (Tier 7) β€” leave it as-is.

4. Direct int4β†’int8 tiled inflate (ORK_DIRECT_I4, 570cb68)

The old path inflated nibble β†’ f32 β†’ tile_f32_i8 β€” reading 4 B/elem and re-quantizing. The direct path skips the f32 round-trip:

  • expand_chan_i4_i8 (UNIFORM, sign-extend) / inflate_chan_nf4_i8 (NF4, vqtbl LUT) β†’ a linear int8 buffer;
  • tile_i8_to_tiles rearranges it by 32-byte k-tile memcpy.

Bit-identical to the f32 path: tiled bytes memcmp == 0, NPU output exact vs both the f32 path and the CPU ref, for UNIFORM and NF4 (because tile_f32_i8 with inv=1.0 is the identity on these in-range int codes).

inflate+tile speedup vs f32 path
UNIFORM 2.2–2.7Γ—
NF4 3.6–4.9Γ—

The inflate is now near-free; the residual is the tile copy + a DOUBLE full bsync (TO|FROM then TO).

Verdict

βœ… Two committed levers compound on the streamed-prefill fill: cacheable buffer (8Γ—) Γ— direct int4β†’int8 inflate (2–5Γ—). The ~1 GB/s "wall" was a benchmarking artifact of the zero-copy WC flag, not a hardware limit β€” production cacheable tiles run at ~10 GB/s. Streamed prefill is the right place for this (large-M crossover); decode stays on the resident draft.

NEXT levers (identified, not built)

  • Clean-only bsync(TO_DEVICE) for steady-state re-fill of an already-initialized buffer (vs the current double full sync) β€” should drop the fill floor toward the ~7 ms/14B-slice cacheable number. (The buffer was already invalidated once; steady re-fill only needs the clean.)
  • SRAM / NBUF tier (RKNPU_MEM_TRY_ALLOC_SRAM + RKNPU_GET_FREE_SRAM_SIZE): the uABI exposes on-chip NPU SRAM (small, ~MB) + an NBUF pool β€” a fourth memory tier below DRAM for the hottest small tensors / scratch. Not yet probed.

Knobs

ORK_DIRECT_I4=1 (direct int4β†’int8 inflate) Β· ork_dma_alloc flag 0x403 (cacheable, production) vs 0x401 (WC zero-copy, the artifact).