Goal: an open full-model NPU runtime for RK3576/RK3588 that drives the NPU directly via raw register-command (regcmd) submission to /dev/dri/renderD129 (the RKNPU DRM driver), with all model weights staged resident in NPU memory. This is the only path that closes the prefill gap (librkllmrt ~134 tok/s vs CPU ~95) in an open stack — and it unblocks multi-layer hidden states (EAGLE-3, #462).

rknn_matmul_api (closed librknnrt.so) allocates per-matmul contexts, each with its own NPU buffers + IOMMU mappings, and dies at ~150 contexts — far fewer than a model's ~200 weight matrices. That ceiling lives in the closed library / vendor kernel; patching it is fragile and doesn't help (you still re-upload per context, never staging weights once). regcmd structurally bypasses it: one large resident weight arena (RKNPU_MEM_CREATE, bounded only by the IOMMU domain — multi-GB), with the command stream referencing weights by offset. That's exactly how librkllmrt avoids the wall.

The RK3588 NPU is an NVDLA-derived fixed-function INT8/FP16 accelerator. A regcmd is a list of (register-offset, value) writes programming its blocks (CNA conv, core/MAC array, DPU, PPU) plus DMA descriptors for input/weight/output buffers. We drive it via RKNPU_SUBMIT (struct rknpu_submit → rknpu_task[], each task carrying regcmd_addr, regcfg_amount). Decode = batch-1 GEMV (bandwidth-bound); prefill = batched GEMM (the NPU's strength — the target).

These give us a known-good NPU result to diff against and the memory layer we need.

The keystone RE. We have a validated matmul (mmtest); capture the regcmd it makes, then learn the format by perturbation.

• M1.1 — ioctl trace shim. ✅ LD_PRELOAD an ioctl() interposer that logs every RKNPU ioctl during mmtest: MEM_CREATE (size/handle/dma_addr/flags), MEM_MAP, SUBMIT (regcfg_amount, task count, core_mask, task_obj_addr). Verify: full call sequence + buffer inventory printed for the known matmul.
• M1.2 — regcmd dump. ✅ Track dma_addr ↔ mmap'd CPU pointer (hook MEM_MAP/mmap); on SUBMIT, dump the regcfg_amount words at each task's regcmd_addr, plus the input/weight/output buffer bytes. Verify: a complete hex dump of one real regcmd.
• M1.3 — register dictionary. ✅ Sweep matmul dims (M,K,N), quant, addresses; diff the regcmds to localize which words encode dims, strides, DMA addresses, quant scales, and the opcode/block-enable bits. Verify: annotated regcmd template with every field's meaning; documented in RKLLM-Runtime-Internals.
• Decision gate: if the ISA proves too opaque to generalize, fall back to replaying captured regcmds for fixed shapes (limited but real), or stop here.

• M2.1 — verbatim replay. ✅ Re-issue a captured regcmd via our own RKNPU_SUBMIT (our MEM_CREATE buffers, patched addresses) → reproduce the result without librknnrt. Verify: matches the oracle bit-for-bit.
• M2.2 — parameterized fp16 GEMM. ✅ Generate the regcmd from scratch for arbitrary (M,K,N). Verify: matches CPU across a shape sweep. ← make-or-break milestone.
• M2.3 — int8 / w8a8 GEMM. ✅ The quant format librkllmrt uses (per-channel scales). Verify: matches a quantized CPU reference within tolerance.

Captured a librknnrt RKNN_INT8_MM_INT8_TO_INT32 matmul (works; constraint N%32 vs fp16's N%16). Same 108-entry regcmd structure; diffing int8 vs fp16 (4×32×32) isolates the precision/element-size deltas:

• Element-size-scaled strides (int8 = 1 byte vs fp16 = 2): 0x1030 = K*N (was K*N*2), 0x1034 = K (was K*2), 0x107c = K/16 (was K/8).
• Precision-mode registers (fixed int8 values): 801:3010 = 0x1 (fp16 0x201), 1001:4010 = 0x80000000 (fp16 0xa8000002), 201:100c, 1001:4050, 1001:40c0 differ — to confirm shape-(in)dependence.
• Data: int8 operands, int32 output; weights layout expected to follow the same [Ntile][Ktile][·][·] nesting (N-tile likely 32 for int8). Remaining: derive the int8 tile dims via the encoded probe, apply the precision registers, validate vs CPU. This is w8a8 — the LLM-relevant path.

From-scratch int8 (RKNN_INT8_MM_INT8_TO_INT32) matmul synthesis validated vs CPU for arbitrary M/K/N (4×32×32 … 32×512×256, 0 mismatches). rknpu_synth_i8.c. Deltas from fp16:

• precision-mode registers baked into the int8 template (801:3010=0x1, 1001:4010=0x80000000, 100c, 4050, 40c0);
• element-size-1 strides: 0x1030=K*N, 0x1034=K, 0x107c=K/16;
• 0x1044=ceil(K/64) — int8 reads 64 K-channels per hardware pass;
• weights layout [Ntile(32)][Ktile(32)][32][32] — N tiles by 32 (vs fp16's 16), K tiles by 32; feature stays flat [M][K];
• constraints K%32, N%32.

Phase 2 is complete: from-scratch GEMM synthesis works for both fp16 and int8 (w8a8 — the LLM precision) at arbitrary shapes. Next: Phase 3 (resident weights — reuse one uploaded weight arena across calls).

• M3.1 — single resident arena. ✅ Upload all weights once into one MEM_CREATE region; matmuls reference by offset. Verify: hundreds of matmuls, zero re-upload, no 150-context wall.
• M3.2 — prefill GEMM. ✅ (arbitrary M via software M-tiling) NPU-expected weight tiling + batched M. Verify: a single layer's matmul hits librkllmrt-class throughput.
• M3.3 — decode GEMV (M=1). ✅ Verify: confirm bandwidth-bound (≈ CPU); decide NPU-vs-CPU routing for decode (likely keep decode on CPU).

rknpu_resident.c: uploads NW weight matrices into one NPU arena (MEM_SYNC once), then runs a matmul against each by patching the regcmd's weight address to arena + w*wbytes. 512/512 matmuls correct against a single uploaded arena — i.e. the ~150-context wall is structurally gone: no per-weight rknn_matmul context, just offsets into resident memory. This is exactly the property the per-op ggml backend (Milestones 1–16) could never achieve, and the reason librkllmrt wins prefill. (A one-shot warm-up submit handles a cold-cache first-iteration artifact.) Next: M3.2 prefill-throughput benchmark, M3.3 decode GEMV.

The "multi-M-tile NPU-state wall" was two ordinary bugs (found via the CBUF/fresh-feature research hint):

1. single-M-tile row cap = 16384/K (not 32768/K) — my chunk was 2× too big, so M>cap silently overran the tile (e.g. K=512 cap is 32, not 64; verified M=32 ✓, M=48 ✗ rows 32-47).
2. unique LIVE buffer per tile — reusing one feature buffer (or destroy+recreate, which reuses freed IOVAs) leaves the NPU reading a stale CBUF; giving each tile a fresh, still-allocated feature+output buffer fixes it. Software M-tiling (chunk = 16384/K rows, fresh A/C per tile over resident weights, warm-up submit) → arbitrary M/K/N correct, validated vs CPU incl. 256×4096×512 (K=4096, 64 tiles), 512×512×512, 256×1024×1024 — 0 mismatches. Phase 3 complete. (Perf note: per-tile buffer alloc + separate submits is slow; a real runtime should use a buffer pool or multi-task-per-submit — correctness first.)

• int8 arbitrary-M ✅ — software M-tiling ported to int8; cap = 32768/K (2× fp16; int8 packs 2× rows in CBUF — verified K=512→64, K=256→128). Validated vs CPU incl 256×4096×512, 512×512×512. Both precisions now do arbitrary M/K/N.
• M-tiling perf 🔬 — attempted multi-task-per-submit (N tiles as N rknpu_task in one SUBMIT, rknpu_mt.c) to collapse N submits→1. Times out, task counter 0 — even task 0 doesn't run. Tried task_counter 0/N and completion-interrupt on the last task only; no change. The NPU does not auto-sequence a regcmd task-list; task-chaining must be encoded in the regcmd PC-control entries (blocks 0x0101/0x0041/0x0081, "continue vs stop") — the part mtx512 also left unimplemented. Bounded RE for a future session. Until then the correct per-tile-submit GEMM stands (slower but right); a buffer pool + keeping the NPU warm are partial speedups not requiring the chaining RE.

The fix is to replicate librknnrt's internal single-task M-scheduler (one SUBMIT, the NPU iterates M-tiles), not N submits. Per-submit overhead is ~32ms steady-state (measured 8× warm), so collapsing N→1 is the real win. rknpu_sched.c, two scheduler regs as closed-form:

• 0x1010 = 16*min(M+1, 32768/K) (rows per tile)
• 0x1040 = max(0x1b, base − slope*(M/64−1)), base=0xb1−15*(2^lg−1), slope=15*2^lg, lg=log2(K/256) — valid only for K in the prefill band; it OVERSHOOTS 0xb1 below K=96 fp16 / K=256 int8 → zero/garbage output (use sched=0/template 0xb1 there). See Exp-2026-07-12-Matmul-Shape-Boundaries.

Validated single-submit, 0 mismatches, K≤1024 moderate-M: 64×256 … 256×512×128, 128×1024×64. This is the M-tiling perf fix for the common regime.

Remaining (precisely traced): K≥2048 and many-tile (>~5) have irregular 0x1040 and need the PC-sequencer data buffer — 0x101:0x10 (PC_DATA_ADDR) + 0x101:0x14 (PC_DATA_AMOUNT), with scaled M-fields (seen in librknnrt's 512×512 = 8 tiles). That's the deepest scheduler layer (RKNPU_OFFSET_PC_DATA_*), bounded RE. Until then the correct software M-tiling (per-tile submit, cap=16384/K fp16 / 32768/K int8) is the universal fallback.

rknpu_hybrid.c: software-chunk M into 4-tile pieces (chunk = 4·32768/K rows), each ONE single-submit scheduler call (the NPU iterates 4 internal M-tiles), fresh feature/output buffer per chunk. ~8× fewer submits than per-tile software tiling (per-submit ≈32ms, so this is the real speedup). Validated vs CPU, any M, K≤2048, 0 mismatches: 1024×512 (4 submits vs 32), 2048×1024×16, 512×256×256 (1 submit), 256×2048, 1024×2048. K≥4096 — hard boundary (confirmed): librknnrt sets the PC_DATA buffer (0x101:0x10 PC_DATA_ADDR) even for small M at K=4096, with R=24 (0x1010=0x180) and 0x1040 floored at 0x48. So the scheduler cannot tile K≥4096 without the PC_DATA program — the scheduler-reg-only fast path is fundamentally limited to K≤2048. K≥4096 fast tiling needs decoding the PC_DATA iteration program (deepest scheduler layer, bounded RE); the correct per-tile software path covers K≥4096 (slower). Net: M-tiling perf is fast (≈8× fewer submits) and validated for K≤2048 any M — the common LLM contraction-dim range — and correct for all K.

Final solution (rknpu_hybrid.c), two-level tiling, no PC_DATA RE needed:

1. K-split: split the contraction dim K into ≤2048 slices (the scheduler-fast range), accumulate partial C's. Sidesteps the K≥4096 PC_DATA program entirely.
2. M-tile: within each slice, chunk M into 4-tile pieces, each ONE single-submit scheduler call (NPU iterates 4 internal M-tiles), fresh feature/output buffer per piece.

Validated vs CPU for arbitrary M/K/N (fp16), 0 mismatches, incl. LLM-scale: 256×4096×512, 256×8192×16, 512×8192×128. ~8× fewer submits than naive per-tile tiling (per-submit ≈32ms). The M-tiling perf is finished — fast and correct for any shape. (int8 uses the same structure; K-split bound is wider since int8 packs 2× rows/CBUF.) The PC_DATA single-pass path remains a documented optimization (fewer K-slices) but is no longer needed for correctness or the perf win.

rknpu_hybrid_i8.c. Same two-level hybrid as fp16, with the int8 scheduler registers reverse-engineered from librknnrt:

• rows/tile 0x1010 = 16·min(mc+1, 65536/Kp) — int8 packs 2× rows per CBUF pass vs fp16 (65536/K vs 32768/K).
• 0x1040 = the fp16 closed form evaluated at effective K = Kp/2 (int8 contracts 2× per CBUF pass). Verified exactly against captures: Kp=512 → base 0xb1/slope 15 (= fp16 @256); Kp=1024 → base 0xa2/slope 30 (= fp16 @512).

K is split into ≤1024 clean slices (where R=65536/K is exact); each slice is M-tiled into 4-tile single-submit scheduler chunks; int32 partials accumulated. Odd remainder slices (non-power-of-2 Kp, e.g. 11008 % 1024 = 768, where R/0x1040 go anomalous — same effect seen at fp16 K=1536/2048) fall back to the proven per-M-tile submit path, which is validated for any Kp. Net: fast scheduler on the bulk, always-correct fallback on the tail.

Validated vs CPU, arbitrary M/K/N, 0 mismatches, incl. LLM-scale: 512×8192×128, 512×4096×512, 256×11008, 256×14336, 128×1280×64. Both precisions (fp16 + int8/w8a8) now have a fast, correct, arbitrary-shape GEMM. Phase 2–3 — the complete open NPU matmul engine — is done.

Buffer pool — done. The hybrid now allocates ONE feature+output buffer pair and reuses it (no per-tile MEM_CREATE/mmap/DESTROY). Counterintuitive finding: a single STABLE reused buffer is correct AND fastest; rotating through >1 pooled buffer corrupts results (corruption grows with depth) — the NPU caches feature state by address across sequential submits, so a changing feature IOVA desyncs it (opposite of the multi-task-in-one-submit case). Surfaced a latent bug while testing: non-power-of-2 K (11008, 5120, 768, 1536) hit the scheduler anomaly and were silently wrong in the old hybrid (only power-of-2 K was ever tested); both fp16 and int8 now gate the fast scheduler on clean Kp and fall back to the proven per-M-tile path for odd remainder slices, so "arbitrary K" is genuinely true.

"PC_DATA single-pass" — there is NO PC_DATA accumulation buffer. The original note assumed K≥4096 single-pass needed a PC_DATA partial-sum buffer (0x101:0x10/0x14). A complete regcmd diff of librknnrt's two chained ops at K=4096 disproves it: the ops differ only in 0x1070 (feature slice) and 0x4020 (output slice) with identical weights — they are M-tiles each doing the FULL K in one op, not K-slices that accumulate. The 0x2000 bit in 0x1040 is a "subsequent M-tile" scheduler flag, not a DRAM accumulate (verified: separate-submit accumulate gives last-slice-only; chained op1 overwrites).

What is actually true:

• PC-chaining works. One submit runs S chained regcmd blocks via the word-216 trailer: (reg 0x10, blk 0x0101)=next regcmd addr, (reg 0x14)=0x37/0. This is exactly what the old rknpu_mt.c was missing (it listed tasks but never set the chain pointer → sequencer never advanced → timeout). Pointing 0x101:0x10 anywhere else → timeout (confirms it's the chain pointer).
• Single-op full-K for M≤R. The NPU iterates K internally in one op. Decode (M=1) does ANY K in ONE submit (0x1010=0x20, 0x1040=0xb1) — validated K=2048/4096/8192 × N up to 512, all correct, baked into rknpu_sched.c. This is the practically important single-pass: token generation issues one submit per matmul regardless of K depth.
• General-M large-K single-op is gated by a nonlinear CBUF scheduler heuristic (rows/tile R per K: 1024→48, 2048→32, 4096→24, 8192→9) — not cleanly formularizable from a few captures. The K-split hybrid sidesteps it entirely and is already correct + fast (sub-ms submits), so this remains the production path for prefill; the single-op path is a future submit-count optimization.

The open NPU matmul primitive set is complete and production-shaped — both precisions, both regimes:

• int8 decode single-submit — int8 M=1 uses the same scheduler regs as fp16 (0x1010=0x20, 0x1040=0xb1); any power-of-2 K in ONE submit. Validated M=1 K=2048/4096/8192 × N≤512 + small-batch M<R. int8 is ~2× faster than fp16 (half the weight bandwidth: 8192×512 = 444ms vs 814ms). Non-power-of-2 K (11008/14336) route to the K-split hybrid.
• Path selection rule (for the M5 backend): M==1 && K is power-of-2 → single-submit full-K; otherwise → K-split hybrid (pooled, arbitrary K, both precisions).

General-M large-K single submit — DONE (M-tile PC-chaining). Built for completeness and as a portable foundation (rknpu_pcchain.c). It is the model librknnrt actually uses: S=ceil(M/R) M-tiles, each computing the FULL K in one op, all chained in ONE submit via the word-216 PC trailer (0x101:0x10=next regcmd, 0x14=0x37/0). M-tiles write disjoint C rows, so no 0x2000 accumulate bit is needed (confirmed). Per-tile (R, base 0x1040) come from a per-K CBUF calibration (nonlinear, captured from librknnrt): 512:(64,0x84), 1024:(48,0x48), 2048:(32,0x2a), 4096:(24,0x48), 8192:(8,0x84). M is padded to S*R (extra rows ignored). Validated vs CPU, one submit each, incl. 512×8192×512 (64 tiles chained) — all 0 mismatches; in the regression suite.

RK3588 performance caveat (why it is NOT the default prefill path). Each full-K M-tile re-reads the entire weight matrix (~22× for M=512,K=4096), vs the K-split hybrid's ~per-K-slice (2×). Matmul is weight-bandwidth-bound, so on RK3588 the hybrid is faster for prefill despite issuing more submits. The PC-chain primitive exists as the single-submit reference and for future Rockchip NPUs where more cores / higher bandwidth may flip that tradeoff. (It is calibrated for power-of-2 K; other K use the hybrid.)

Bottom line: Phases 1–3 + perf + polish are done. The matmul layer is correct, fast, pooled, arbitrary-shape, both precisions, with optimal decode. Everything remaining is the transformer runtime (M4–M6).

• M4.1 — reusable matmul library (rknpu_mm) ✅ — keystone: one device handle, weights packed once and kept resident, many matmuls streamed against them.
• M4.2 — non-matmul CPU ops ✅ — RMSNorm, RoPE, attention/softmax, SiLU/SwiGLU.
• M4.5 — non-matmul ops ON the NPU 🟢 (2026-07-04, Exp 2026-07-04 PPU Element-wise / Activation Ops) — supersedes the "PPU activations don't run standalone" finding. Standalone SDP ops (NVDLA-derived, enable_mask=0x18) now run bit-exact on silicon, in make test: EW-mul (int8/fp16/int16), SiLU/GELU (int8 bit-exact, int16 RKNN-class), residual Add (int8/fp16), exp/rsqrt LUT blocks. The activation is a LUT op (contents define the fn). RMSNorm reduction = matmul (no dedicated reduce op); RMSNorm/Softmax/RoPE compose from these blocks. Outstanding (tracked): (a) fp16 activations — NVDLA LE-table exponential mode + multi-task submit; (b) int16 residual Add — SRDMA/X1 operand halves negatives (X1/X2 sub-module scale decode); shared unlock = multi-task submit (also enables the fully-fused exNorm/exSoftmax). RE toolkit: ork-driver tools/re/.
• M4.6 — llama.cpp integration of the on-NPU ops: standalone offload is a DEAD END; the needed op is FUSION 🔴 (2026-07-04) — Wiring the M4.5 standalone ops into the ggml-ork backend as per-op offload paths (ORK_PPU_OPS/_ADD/_SILU/_GELU, branch feature/ggml-ork-ppu-ops) surfaced three independent reasons standalone element-wise/activation offload cannot win on this hardware, so the router correctly keeps them on CPU/NEON:
  1. Op-shape mismatch. Modern llama.cpp does NOT emit separate SiLU + MUL nodes for the FFN — it fuses them into one GGML_OP_GLU op (ggml_swiglu_split = silu(gate)⊙up; GELU FFN → ggml_geglu_split). So standalone SiLU/MUL handlers never even fire on Qwen/Llama.
  2. No arithmetic intensity. The NPU wins matmul because each weight load is reused across M·K MACs; an element-wise op is 1 op/element, so a fp32→(quant)→DMA→submit→DMA→(dequant) round-trip can't beat inline NEON (matches the long-standing "activations are memory-bound / NEON-competitive" finding). Measured (residual ADD, Qwen3-1.7B, pp128): 157 → 19 t/s (~8× slower). Root cause (empirically isolated): NOT buffer churn and NOT the op itself — a persistent-scratch build that eliminated the per-op MEM_CREATE/MEM_DESTROY churn moved it only 19.45 → 18.83 t/s (null result, feature/ppu-scratch-reuse), and the SDP ops run in ~18–118 µs standalone. The 8× comes from INTERLEAVING the SDP ops into the int8-matmul stream: the NPU regcmd mode is stateful, so a matmul→ADD→matmul sequence forces a mode switch + re-warm around every ADD (the "switching modes triggers a one-time NPU re-warm" behaviour), and it breaks matmul PC-chaining (consecutive matmuls can no longer be chained into one submit when an ADD splits them). Both scale with the number of interleaved ops (~56/forward) and neither is fixable by optimizing the standalone op. Only FUSION avoids it — applying the activation/add in the matmul's own output stage keeps a single mode, adds no submit, and preserves chaining (data already resident in NPU SRAM). This is a stronger reason for fusion than arithmetic-intensity alone.
  3. IOVA instability. Each standalone op allocates a fresh NPU DMA buffer; doing that per-residual-add/-activation per-layer churns the 4 GiB IOVA window already consumed by the resident matmul weights (NPU IOVA 4 GiB cap) → measured a hard kernel fault in rknpu_iommu_dma_map_sg (ORK_PPU_ADD=1, Qwen3-1.7B) that wedged the NPU. Verdict: routing standalone element-wise ops to the NPU while weights are resident is not just slow, it's unstable.
  • ★ FUSED GATE-MATMUL + SiLU: DONE, BIT-EXACT ON SILICON (2026-07-04). ork_mm_run_i8_silu (ork-driver feature/ffn-fusion) runs a resident full-K int8 matmul with SiLU fused in the SDP output stage (LUT streamed to PPU SRAM once, then matmul+silu per N/M-tile) → int8 silu(gate). Validated == the silicon probe (tools/fused_silu_test.c: mism 0/512, max|e|=0). KEY BUG resolved: a hand-rolled submit to a fresh output buffer produced bias-only output — the conv accumulator never reached the SDP. The fix is to submit via submit1() into the WARMED c->Cc (+ dom_activate at entry, act(RESET) right before the submits). This is the gate half of on-NPU SwiGLU and the first activation that rides a matmul's own submit.
  • ✗ FUSED UP-MATMUL + EW-mul (×silu(gate)): still WEDGES. ork_mm_run_i8_ewmul uses the set_i8_ewmul DPU_RDMA-graft-onto-synth_i8 output stage; it wedges the NPU (job timeout, task counter 0x0) even on a clean NPU as the first op — the same 2nd-operand-fetch wall the standalone EW-mul RE hit (only the standalone ork_npu_ewmul_i8 cracked it, via the full NVDLA cube layout, not the matmul-fused graft). So a fully-fused SwiGLU needs either that graft fixed OR compose: fused-SiLU gate + up matmul + the multiply via standalone ork_npu_ewmul_i8 / CPU. Gated behind ORK_TEST_EWMUL (won't wedge make test).
  • ★ FUSED-EWMUL GRAFT — web-lead search exhausted, CLOSED as a hard wall (2026-07-04). Chased the NVDLA docs (programming_guide + unit_description) for a config that unwedges the online-mode (conv×memory) EW-mul: (a) confirmed the EW/X1 sub-module is the only per-element-multiply path — the BS sub-module's per-element mode is bias-ADDITION only ("Multiplier will be bypassed during bias operations"), so BS can't do the SwiGLU multiply; (b) the online-mode 2nd-operand fetch (BRDMA/ERDMA + flying D_FEATURE_MODE_CFG) is exactly what the prior ~40-cycle campaign tried (all wedge, task counter 0x0 = program rejected); (c) the docs name no specific working RDMA/ordering/size config that wasn't already tried, and submit1 (which fixed the SiLU acc-not-reaching-SDP) does NOT change the ewmul's program-validity wedge. The two ork sub-paths both fail: the graft (RKNN EW output stage on ork's dense conv geometry) WEDGES; RKNN's verbatim op executes but needs RKNN's own A/B tile layout (zero output on ork data). No new viable path from web search → the fused-ewmul-into-matmul graft is CLOSED. Refs: nvdla.org programming_guide.html, unit_description.html (SDP X1/X2/Y + BS).
  • ★★ CHAINED on-NPU FFN inner BUILT + a 97× ACT_RESET fix + PIPELINE overlap 1.25× (2026-07-04). tools/fused_ffn_probe.c runs the composition int8-throughout (gate+SiLU → up-out8 → ewmul → down, ork_mm_run_i8_out8 added). KEY GENERAL FIX: the fused SDP ops (silu/out8/ewmul) were doing a per-call act(RKNPU_ACT_RESET,0) — an NPU soft-reset costs tens of ms/op — leftover debug scaffolding (submit1 was the real correctness fix, not the reset). Removing it: chained FFN inner 320 ms → 3.2 ms/ffn (97×), and standalone ewmul_i8 ~107 ms → 73 µs, BOTH re-validated bit-exact (test_ewmul_i8 mism 0/8192; fused-silu mism 0/512). This is why standalone SDP offload measured so badly before (the 8× ADD regression was largely this reset, not only arithmetic intensity — worth re-measuring the other SDP ops after dropping their resets too). PIPELINE (probe pipe mode): running the NPU FFN inner concurrently with representative CPU work overlaps them — SERIAL 1621 ms → OVERLAPPED 1297 ms = 1.25× (ideal 1.68×); monitor harness shows NPU + CPU both active at once. Gap to ideal = thread contention (no pinning / 1 NPU stream + 1 CPU thread). Confirms the architecture: keep the FFN on the NPU (frees the CPU, 66%→13%), overlap the freed CPU with the NPU → throughput win. NEXT: wire into ggml-ork so REAL per-token CPU work (attention/quant/norm) overlaps the NPU FFN + pin threads + multi-core the FFN ops → push 1.25× toward the ideal and into an end-to-end model number.
  • ★ PIPELINE pinning 1.25×→1.57× + GGML_OP_GLU wired end-to-end (2026-07-04). (a) Pinning the NPU thread to core 4 and the CPU work to cores 5–7 cut the overlap contention: 1.25× → 1.57× (overlapped 1066 ms vs 1002 ms NPU-alone; ideal 1.67×) — nearly all CPU work now hidden behind the NPU FFN. (b) Wired GGML_OP_GLU (SwiGLU) into the ggml-ork backend (supports_op + graph_compute + ggml_backend_ork_glu, fork feature/ggml-ork-ppu-ops): quantize gate/up per-tensor int8 → ork_npu_silu_i8 → ork_npu_ewmul_i8 (gain 1/128) → dequant, ORK_PPU_GLU-gated, fp32 CPU fallback. Runs a real model end-to-end (Qwen3-1.7B): engages on-NPU (not the fallback), no wedge, COHERENT output — but int8-degraded ("capital of the world"), the expected loss of the standalone fp32→int8 path. This is the wiring foothold; the perf win needs the round-trip-free int8-CHAIN (gate/up matmuls emit int8-out feeding the GLU + down reads int8, no fp32 boundary) which requires FFN-pattern detection in graph_compute + per-tensor act-quant scale plumbing.
  • ★★ ROUND-TRIP-FREE int8 FFN CHAIN — BUILT END-TO-END, measured NET-LOSS (2026-07-04, decisive). Implemented the full chain in ggml-ork (ORK_FFN_CHAIN, off by default; fork feature/ggml-ork-ppu-ops commit 3457ca744): FFN-pattern detection in graph_compute (validated: [gate,up,GLU,down] arrives as ONE 4-node ork subgraph at prefill M≥32; at M<32 GLU→CPU so it fragments and no-match) → ggml_backend_ork_ffn_swiglu_chain runs gate matmul+fused-SiLU (per-layer LUT via ork_mm_silu_build_lut) → up int8-out → ork_npu_ewmul_i8 → down, int8 intermediates never touching fp32; per-tensor scales (the fused SiLU stage's scalar R can't carry the model's per-row/per-channel — the accuracy concession), CPU fp32 fallback. In-model (Qwen3-1.7B-Q8_0, RK3588): CHAIN pp128 = 140.14 t/s vs per-node baseline 147.43 → ~5% SLOWER, plus intermittent NPU soft-reset. Root cause (structural, matches the weight-DMA/submit-floor findings): the fused chain needs MORE submits (LUT-load + silu + out8 + ewmul + down, and the fused SiLU path is M-capped at 64/submit) than the baseline's 3 larger matmuls (M-scheduler to mg_max*64) + a cheap CPU GLU; the fp32 round-trip the chain removes is NOT the prefill bottleneck. NOT YET VIABLE — to reach even parity needs a breakthrough: (1) large-M-tiles for the fused output-stage path (mg_max*64, like the int8 matmul cap fix) so silu/out8 aren't capped at 64/submit, (2) load the silu LUT ONCE not per-tile, (3) fix the fused-op intermittent wedge at Nff=6144/down K=6144. Decode can't use it (M=1 → CPU GLU → no match, and the serialization wall loses at M=1 anyway). Code kept, gated OFF.
  • ★★★ FFN CHAIN reaches SPEED PARITY+ (2026-07-04, flips the net-loss above). Four levers turned the ~5%-slower incoherent chain into a faster, coherent one (ork-driver commit 2ef2f0a, fork 3533ac922): (1) fused output-stage M-tile cap 64→mg_max*64 (bit-exact, tools/fused_mtile_check); (2) SiLU LUT loaded once/call not per-tile (bit-exact vs probe); (3) ewmul moved to CPU int8 — the elementwise multiply has no arithmetic-intensity so it loses on the NPU (per-call IOMMU churn + cube-marshaling ~6ms/ffn
    • it wedged); SiLU still rides the gate matmul output stage for FREE, so this is the arithmetic-intensity split (matmuls NPU, elementwise CPU); (4) SmoothQuant per-layer (migrate x's per-channel outliers into gate/up weights x'=x/s W'=s*W, and glu's outliers into the down weight; static per-tensor scales calibrated on the first batch) → coherence: PPL 23041 → ~22. Gated OFF (ORK_FFN_CHAIN).
  • ★★★★ CORRECTION + HYBRID chain (2026-07-04). (1) The earlier "pp512 6×" was a benchmarking error — that run's baseline pp512=32 was anomalous (cold / parked DDR governor). Re-measured with governors verified at performance: baseline pp128 150 / pp512 191; the all-int8 chain is pp128 174 (+16%) / pp512 193 (~parity, NOT 6×). (2) Per-channel SDP RE finding: RKNN does int8 matmul per-channel requant on the HOST, not in the SDP regcmd (per-layer vs per-channel captures byte-identical bar an int_status task-desc bit; tools/re/percapture.c) — so no SDP recipe to copy. (3) Pivot → HYBRID chain: only the GATE stays per-tensor (its fused SiLU has a scalar SDP output scale); up + down use the STANDARD per-channel host-dequant path and the ewmul is fp32, so silu_i8 is the ONLY int8 intermediate. PPL ~22 → ~16 (vs baseline ~9); speed pp128 162 (+8%) / pp512 160 (−16%, fp32 intermediates cost at large M). Quality floor ~16 = the gate's per-tensor + int8-silu-output (fused scalar-R). True parity (PPL 9) needs per-row GATE activation — provably impossible in the fused stage (silu(a·x)≠a·silu(x); SDP scale is per-channel/per-N not per-row/per-M). The only exact-output route left is drafter-verify (degraded chain drafts, exact baseline verifies). Also an intermittent NPU soft-reset persists on the fast path (ACT_RESET removed for the 97× win). Gated OFF.
  • RESOLUTION for on-NPU SwiGLU (no graft needed): COMPOSITION of validated pieces — gate matmul + fused SiLU (ork_mm_run_i8_silu, bit-exact) → silu(gate) int8; up matmul + int8-out (set_i8_out8) → up int8; then silu(gate) ⊙ up via the STANDALONE ork_npu_ewmul_i8 (mem×mem SDP, validated bit-exact) or CPU. 3 submits, all validated-working, entirely on the NPU in int8/SDP mode (no per-op interleaving with CPU ops, so it avoids the standalone-offload 8× penalty). This is the productionizable on-NPU SwiGLU; the fully-fused 2-submit form is blocked on the closed graft.
  • NEEDED OP (theoretically possible, high-value): fused matmul→activation output-stage. The only way the NPU adds activation value is to apply the activation IN the matmul's output stage, riding its existing submit + DMA buffers (no new IOVA, no new submit, no round-trip). This maps exactly to GGML_OP_GLU: swiglu(gate,up) = a gate matmul with SiLU fused in its output stage (set_i8_silu, already RE'd + validated on silicon) followed by an up matmul with the per-element EW-mul fused in its output stage reading silu(gate) as the 2nd operand (the per-element SDP EW-mul, also RE'd + bit-exact). Both building blocks exist; the missing pieces are (a) a public ork-driver fused-matmul API (ork_mm_run_i8_silu / _ewmul, output int8 or dequantized) and (b) a ggml-ork graph-pattern matcher that detects GLU(gate=Wg·x, up=Wu·x) and emits the fused NPU sequence instead of 2 plain matmuls + a CPU GLU. This is the productionizable on-NPU-activation path.
  • Other needed-but-unsupported ops (theoretically possible): fp16 SiLU/GELU (NVDLA LE-exp LUT mode, path known — see M4.5(a)); softmax max-reduce (the one non-matmul reduction not expressible as a matmul — needed for a fully-on-NPU softmax; exp LUT block already exists); RoPE (cos/sin LUT + EW-mul + Add — composable from existing blocks, not yet built standalone). RMSNorm needs no new op (reduction = matmul + rsqrt LUT), just fusion if profiling ever justifies it (memory-bound → NEON-competitive today).
• M4.3 — full decoder-layer hybrid forward ✅ — matmul on NPU, ops on CPU; validated vs a pure-CPU reference.
• M4.4 — KV-cache incremental decode ✅ — token-by-token, M=1 projections, RoPE at the running position, attention over cached K/V; validated vs CPU and consistent with prefill.
• Multi-layer / real-model wiring ⏳ → Phase 5 (M5).

Phase 4 is complete. The open NPU stack now runs a full transformer decoder layer end to end — prefill and KV-cache decode — with matmul on the NPU and the non-matmul ops on the CPU, all validated to fp16 precision and in the regression suite (43 cases).

The matmul kernels are now a library (rknpu_mm.{h,c}): rknpu_mm_init → rknpu_mm_pack(K,N,B) (pack + upload weights once, resident) → rknpu_mm_run(w,M,A,C) (stream activations; K-split hybrid internally, arbitrary M/K/N). On top of it, rknpu_layer.c runs a complete Llama/Qwen-style decoder layer — h = x + Wo·Attn(RoPE(Q,K),V), y = h + Wdown·(SiLU(Wgate·hn) ⊙ Wup·hn) — with the seven big projections on the NPU and the non-matmul ops (RMSNorm / RoPE-NeoX / causal softmax / SwiGLU) on the CPU (the pragmatic hybrid graph).

Validated against a pure-CPU reference (identical ops, CPU fp16 matmul, so the matmul engine is the only variable): H=512, NH=8, HD=64, FFN=2048, SEQ=16 → maxabs 0.044 vs |ref|max 276 (~0.016 %, pure fp16 rounding). Both in the regression suite.

Two bugs found while turning the standalone kernels into a reusable library:

1. setr must patch ALL matching regcmd entries, not the first. The template carries some regs (0x1040) more than once and the NPU uses a later copy; first-match-only left a stale schedule value — wrong for multi-M-tile shapes only (single-tile masked it).
2. A freshly (re)allocated output buffer is cold — its first submit returns stale data. The one-time warmup must re-fire whenever the output buffer is resized (here when N grows 512→2048 between projections), not just once per context.

Both are NPU-state subtleties (not algorithm errors) — the kind of thing the standalone one-shot kernels never exercised.

rknpu_decode.c runs the same layer one token at a time: per step, M=1 NPU projections, RoPE at the running position, append K/V to a per-layer cache, attend over the cached keys/values. Validated vs a pure-CPU reference over a 16-token sequence — maxabs 0.044 vs |ref|max 276, identical to the prefill layer's error, confirming prefill ↔ incremental-decode causal consistency. This is the token-generation path (M=1 matmuls, the decode regime). Phase 4 done. Next: stack N layers + load a real model's weights / tokenizer / embeddings / LM-head (M5).

Works across a wide range — pure large-M (512×32×16 ✓), large-M + moderate K/N (128×128×64 ✓), large-K/N at modest M (32×512×128 ✓), GEMV (M=1 ✓). The raw NPU output C buffer byte-matches librknnrt (verified at 128×32×16), and feature/weights/regcmd all match — so the matmul is correct over this range. It diverges only when M>32 AND K≥256 together (e.g. 128×256×256, 128×512×512) — a feature M×K co-blocking interaction (the feature, flat [M][K] at smaller sizes, needs M×K tiling past that threshold). This is the last GEMM edge. Feature confirmed flat [M][K] even at 128×256 (row0 = k0..255 contiguous), so the divergence is on the weights-at-large-(KT×NT) or output side — same byte-compare→decode method applies. M-tile = 64 (M>64 ⇒ multiple M-tiles): two registers then change — 0x1010 = min(16*(M+1),0x800) (saturates at M≥127, FIXED) and 0x1040 (packed M-tile-count field: 0xb1/0xa2/0x84 for M/64=1/2/4, partially decoded) — plus likely an M-tile data layout. GEMM is now correct for any M at single/moderate-N (any K) + GEMV: M-tiling solved via 0x1010=min(16(M+1),0x800) and 0x1040=0xb1-15*(ceil(M/64)-1) (both occurrences set). Remaining edge: large-M AND large-K together (e.g. 128×512×16, K≥512 at M>64). Root cause precisely characterized: the NPU auto-tiles M based on an SRAM budget that depends on K — 0x1010 and 0x1040 are K-dependent (128×256: 0x1010=0x800,0x1040=0xa2; 128×512: 0x1010=0x400,0x1040=0x84). The budget is ≈ 32768 elements: M-tile-rows ≈ min(M, 32768/K) (K=256→128 rows, K=512→64 rows), so 0x1010 = 16·min(M, 32768/K). Verified correct now for any M with K≤256 (128×256×256, 256×256×256 ✓), large-N at small K (128×64×512 ✓), and GEMV. Completing the K-driven M-tiling scheduler (0x1010 + 0x1040 + M-tile count) is the last GEMM piece — a bounded scheduler model, not a new layout.

[superseded] Earlier note (large-M>8 diverges) was wrong: it conflated pure large-M (which works) with the large-M+large-K shapes. Verified for M=128: the regcmd byte-matches librknnrt (no M-tile registers), and with uniform per-row probes the feature is flat [M][K] (rows in order) and the output C is flat [M][N] (row m = K·m). Yet random-data M=128 gives C[0] correct, rest wrong — the same "uniform-matches, random-diverges" signature that the nested K×N case had, i.e. a subtle M-blocking layout the constant-row probe can't expose. Resolution is the same proven method (full byte-compare synth-vs-librknnrt buffers with identical random data → localize → decode), not yet done. Decode GEMV (M=1, M3.3) and small-M prefill already work, so this is the large-batch prefill edge.

• M4.7 — batched-GEMM (attention/GDN) + normalization ON the NPU ⚖️ (2026-07-10, Exp 2026-07-10 On-NPU Attention Batched-GEMM and Norm, ork-driver main 32db14c) — the two remaining non-matmul op classes, built + bit-exact-validated + gated, all in make test. ork_bmm_i8/i4/fp16: a batched dynamic GEMM (both operands dynamic) — the attention QKᵀ/AV and Gated-Delta-Net chunked-matmul primitive the static-weight path can't serve (i8/i4 exact, fp16 4e-4). ork_npu_rmsnorm_f16/l2norm_f16 (gated ORK_NORM_NPU): the full-channel Σx² reduction runs on the NPU as a matmul contraction (any n via K-split — the M4.5 "reduction = matmul" now realized), and the rsqrt via ork_mm_build_f16_rsqrt_lut = the fp16 fused-into-the-reduce twin of the M4.5 standalone ork_npu_rsqrt (fused K=n≤2048 one submit 6e-4 / decoupled K=512 any-n 5e-4; K=32 is degenerate for the fp16 fused-silu tiling). Plus ork_l2norm_f32 (NEON CPU). Corroborates M4.6 (standalone offload loses; fusion is the need): ORK_ATTN on Ornith-9B routes the GDN chunked matmuls but NPU stays 1.6% / prefill unchanged (the CPU wall is the sequential GDN recurrence — gated_delta_net/solve_tri/cumsum/exp — not matmuls); pure-attention Qwen3-1.7B doesn't offload at all (real QKᵀ/AV are non-contiguous KV-cache/permuted-Q views → declined). Standalone norm is submit-floor-bound vs the ~µs fused NEON pass; the win needs fusing into an adjacent matmul's output stage. RE landed: the DPU output stage is NAMED against the mainline drivers/accel/rocket/rocket_registers.h (0x4040 DPU_BS_CFG, 0x4060 DPU_BN_CFG, 0x4070 DPU_EW_CFG+LUT, 0x4030/34/3c DATA_CUBE_WIDTH/HEIGHT/CHANNEL geometry), and NVDLA's CDP LRN reduction window is HW-capped at n∈{3,5,7,9} — proving a full-channel (d_model) norm reduction cannot use the fixed-function LRN op and must ride a matmul contraction (which is what shipped). RE box: rknn-toolkit2 2.3.2 on the .239 Colima VM.

• M4.8 — FUSED matmul + output-stage activation: the no-crossing chain ✅ (2026-07-10, Exp 2026-07-10 Fused Output Activation No-Crossing Chain) — the payoff M4.6/M4.7 kept pointing at. ork_mm_run_f16_act(K,N,B,M,A,C,fn,…,in_lo,in_hi) = C=fn(A·B) in ONE submit, the activation riding the matmul's DPU output stage — no separate op submit, no CPU↔NPU crossing, no activation→matmul mode-switch. Validated bit-accurate in make test: fused exp (softmax) 4e-3, silu (SwiGLU) 2e-3, gelu 6e-3, rsqrt (RMSNorm, shipped earlier) 6e-4. Mechanism: build the fn PWL LUT over the output range, pack the weight ±S·B so acc hits the fp16 index-spread band, run the fn-agnostic fused-LUT matmul (ork_mm_run_f16_silu), recover fn=C·out_scale. One real constraint: the fp16 SDP index spreads for a SINGLE sign of acc, so the matmul output must be single-signed per tile — handled by construction (positive-input rsqrt→pack −S·B; negative-input exp(scores−max)≤0→build fn(−u), pack +S·B); mixed-sign returns −2 (needs two LUTs — the real meaning of the old "fp16 silu WIP"). Contrast the kernel-sequenced standalone softmax replay (NPU-Quirks): bit-exact but 9× the submit floor (~2 ms); fusion makes the transcendental free. Follow-up = wire into ggml-ork (exp on the score matmul, silu/gelu on FFN up-proj, rsqrt on the norm reduce).

• M5.1 — resident-regcmd GEMM in the ggml-rknpu backend, replacing the rknn_matmul path. Verify: prefill re-benchmark vs librkllmrt (target: close 134).
• M5.2 — end-to-end forward for one architecture (Qwen3) via the oRKLLM/llama.cpp-rockchip fork. Verify: llama-bench prefill ≈ librkllmrt; correct output.
• M5.3 — productionize: 3-core scheduling, error handling, per-arch graphs.

Took the standalone-harness route first (prove the whole stack with a real model before wiring the ggml fork). Three increments, all in the regression suite:

• M5.1 multi-layer body (rknpu_model.c): N stacked decoder layers, each with its own weights; validated NPU-hybrid vs CPU to 24 layers — error grows gracefully (1L 0.0015 → 24L 0.046), no divergence.
• M5.2 real weights + M5.3 end-to-end forward (rknpu_llama2.c): loads Karpathy's llama2.c stories15M (real trained Llama: dim 288, 6 layers, 32000 vocab), mmaps the fp32 .bin, packs projections + LM head into resident fp16 NPU weights (transposed [out][in]→[in][out]), and runs the full decoder — embed → 6 layers w/ KV cache → final RMSNorm → 32000-vocab logits — matmul on NPU, ops (RMSNorm, interleaved GPT-J RoPE, causal softmax, SwiGLU) on CPU. Greedy-generates; NPU-hybrid logits match a pure-CPU-fp16 reference to worst |Δ| = 0.01 over 12 steps. A real LLM now runs on the regcmd NPU stack.

Bugs found + regression-locked this milestone:

1. NPU output-width cap = 8192 (N=16384 silently wrong, N≤8192 exact). The 32000-vocab LM head exposed it. Fixed with N-tiling in rknpu_mm (weights now pack into Sk×Sn = K-split × N-split≤8192 buffers; run accumulates K-slices into each N-slice's columns). test_mm now covers N=16384 and the 1×288×32000 LM-head shape.
2. Unsigned-arithmetic in synthetic norm-weight init ((i+seed)%5-2 underflowed to ~4.3e9) → RMSNorm explosion → fp16 overflow → NaN; the check then silently passed (NaN compares false). Hardened to fail on NaN/inf/dead output.
3. maxout over-allocation (ignored the actual chunk → a 58 MB LM-head output buffer DMA-synced every submit). Sized to the real chunk.

The reusable library (rknpu_mm) + ops are the substance the ggml/llama.cpp-fork integration (M5.1-fork above) will call; that wiring + multi-arch + 3-core scheduling remain.

The remaining GEMM edge is the NPU's internal M-tiling scheduler (0x1010, 0x1040 depend on BOTH M and K). Captured table (value = high16 of the reg word):

0x1010 = 16·(rows-per-M-tile): K=256→128, K=512→64, K=1024→48 rows (not a clean 1/K — there's a floor/rounding in the SRAM heuristic). 0x1040 is a packed bitfield encoding the M-tile schedule. Modeling this scheduler is the last GEMM piece — bounded RE, but genuinely the NPU's tiling logic (what rknn_matmul computes internally), not a single formula. The GEMM is correct without it for the common range: any M at K≤256, large-N at small K, GEMV, and all int8/fp16 K%32/N%(16|32) shapes that fit one M-tile.

Tried to finish the large-M+large-K edge two ways; both hit a genuine NPU multi-M-tile state behavior I could not crack this session:

• NPU internal M-scheduler — 0x1010/0x1040 are K-dependent (data table above); even with the right values for moderate cases, the 2nd M-tile's output lands wrong for large-M+large-K.
• Software M-tiling (split M into ≤cap row-chunks, loop submits over resident weights) — fails on the 2nd chunk in both sub-variants: offset feature addressing corrupts the result, and same-address re-upload reads a stale NPU feature cache (M3.1's loop worked only because its feature was static and only the weight address changed). Non-cacheable buffers + warm-up submit didn't resolve it.

So consecutive submits with changing feature data is the unsolved bit — the NPU appears to reuse feature state across jobs in a way that needs an invalidation/handshake I haven't found, or a fresh feature buffer per chunk. The single-M-tile GEMM is solid (any M at K≤256, large-N at small K, GEMV, all int8/fp16); large-M+large-K prefill batching is the remaining edge and is genuinely harder than a register formula.

• M6.1 — multi-layer hidden states → resolves #462, enables open EAGLE-3.
• M6.2 — open NPU prefill engine selectable in oRKLLM (the two-engine model: open-NPU/llama.cpp for everything, no closed librkllmrt dependency).

Spun out of [Exp-2026-07-07-INT4-Multi-M-Schedule-RE]. We landed int4 native multi-M batch mode (0x107c=K/16 → CBUF activation budget 16384 int4 elems; rows = min(M/2, 16384/K) = 8/submit @ K=2048, 32 @ K=512; stride-2 in/out, physrow = 4j + 4H·b; default-on, bit-exact). Batch is CBUF-residency-limited (the 8-row ceiling at K=2048). int8/fp16 instead stream (int8 M-tile = mg_max*64 = 128 @ K=2048, not CBUF-limited) via the 0x1040 K-reduction schedule — which poisons int4 with its int8 value. The 0x107c = D_ENTRY_PER_SLICE result (elements/entry: fp16 8, int8 16, int4 16 — int4 rides int8's 16-element/64-bit entry, not a dense 32/128-bit one) ties these together and generates the leads below, ordered by value. All share the CNA/CSC batch recipe (0x405c=0 + mc_phys stride-2 + 0x107c + de-tile), so they compose.

1. ★★ int4 STREAMING (biggest win: ~16×, → int8-class W4A4 prefill). Status: OPEN — not found, not ruled out. A streaming schedule (activations from DRAM, non-resident) would blow past the CBUF cap to ~128 rows @ K=2048. Prime lead: int4 poisons on 0x1040 likely due to an entry-layout mismatch — it's in the 64-bit/16-elem batch layout the streaming schedule can't drive. So set the dense 128-bit entry first (0x107c=K/32, 32 int4/entry, matching int8's entry width) and THEN sweep 0x1040/schedule. Win criterion: rows×K > 16384 at K=2048. Tool: tools/i4_multim_fuzz.c combo mode. (The earlier standalone 0x1040 sweep found nothing precisely because it lacked this dense-entry precondition.)

2. ★ int8 in BOTH modes = the stream-vs-batch SELECTOR (best lead-generator for #1). int8 is the only dtype we can run both ways: it streams today, and the shared batch recipe (0x107c=K/16) should also make it batch. Diff streaming-int8 vs batch-int8 regcmd → the differing register(s) ARE the mode selector; apply to int4 → candidate streaming trigger (turns #1 from blind fuzz into "read off the diff"). Also re-confirms D_ENTRY_PER_SLICE in a 2nd mode and reveals whether the CBUF budget is bytes or elements (int8 batch cap = 16384/K or 32768/K? — predicts int4's dense ceiling). Worth doing even though int8-batch has no direct perf use. Method: force synth_i8 into the batch recipe at small M, validate bit-exact, diff the two regcmds.

3. int4 DENSE entry packing (0x107c=K/32, standalone 2× batch). If the HW supports a dense int4 entry (32 elems/128-bit) without streaming, the batch budget doubles (16384→32768) → 16 rows @ K=2048. Smaller (2×) but lower-risk than #1, and a stepping stone to it (same K/32 precondition). Needs the companion width/atom reg for 32/entry (candidates: 0x100c precision-mode, 0x1044 K-passes) co-varied with 0x107c.

4. fp16 BATCH mode (a correctness fix, not perf). fp16 has a documented latent large-tile streaming M-scheduler bug (AGENTS.md). Applying the batch recipe to fp16 (0x107c=K/8, stride-2, resident) gives an independent correct large-M fp16 path (CBUF-capped but bit-exact) that sidesteps the buggy streaming scheduler. Test: batch recipe on synth (fp16) at the M where streaming breaks; check bit-exactness.

5. int4 wide-N N-subslice (perf completion of the landed batch path). The shipped batch path falls back to per-row for wide N at large K (per-core Ncore×K > 131072 weight budget). An N-subslice loop that keeps Ncore_sub×K ≤ 131072 per submit extends batch mode to production wide-N shapes — see the Exp-2026-07-07 log.

Spun out of the forward-softmax replay (fwd-softmax). The mainline open-source drivers/accel/rocket driver (see AGENTS.md §4 "Open-source reference") shows the NPU runs one task per PC program (TASK_CON/OPERATION_ENABLE, TASK_NUMBER(1)), and the kernel IRQ handler re-arms the next task from the job's task array — sequencing is kernel-driven via completion interrupts (PC_INTERRUPT_*_DPU_0/DPU_1 = 0x300), NOT the hardware walking a concatenated regcmd buffer via an in-regcmd "next-PC" (0101:0x0010). DMA faults surface as PC_INTERRUPT_RAW_STATUS_DMA_READ_ERROR (the softmax-replay hang's raw_status 0xc0000000 with require 0x300 = a DMA read error on a chained task, not a completion-on-a-different-bit).

RESOLVED (2026-07-10) via the forward-softmax replay experiments: the vendor rknpu driver uses HARDWARE PC-chaining (0101:0x0010 next-regcmd), NOT rocket's kernel-re-arm-on-IRQ. Evidence: a task_number=9 job with only the descriptor array + zeroed chain fails errno=110 on rknpu, whereas rocket kernel-iterates that fine. ⇒ ork-driver's existing run_chain_i8/i4 + run_multicore PC-chaining (which sets the 0101 words) is CORRECT for its target driver — no change needed; the rocket kernel-re-arm model is rocket-specific and does not port. Confirmed working paths stay as-is.

Hardware-chain MECHANISM cracked (2026-07-10, from the vendor rknpu kernel rknpu_job_commit_pc): the kernel programs the HW from the first task only — PC_DATA_ADDR=first->regcmd_addr, PC_DATA_AMOUNT from first->regcfg_amount, INT_MASK from the last task, PC_TASK_CONTROL=task_number — then the hardware walks the chain via the in-regcmd descriptor 0101:0x0010 = next regcmd addr, 0101:0x0014 = next register-amount =(n+3)/2 (== kernel (n+EXTRA+scale-1)/scale-1, EXTRA=4, RK3588 pc_data_amount_scale=2). This is the general rule behind run_chain_i8's hardcoded 0x0014=0x0037 (=(108+3)/2, its uniform tasks).

Softmax hardware-chain status: ✅ SOLVED (2026-07-10). The full 9-task forward-softmax graph now runs as ONE hardware-chained submit, bit-exact 9/9, ~471µs (vs ~2.5ms per-task = 9× floor → ~5×) — ork_npu_replay_softmax_f16 default. The full recipe (a COMBINATION, all required):

1. 0101:0x0014 next-amount =(n+3)/2 (was zeroing it → hung at transition 1).
2. Contiguous single-buffer image + single-delta blanket rebase — replay the vendor's whole 5-buffer image in ONE allocation so task4's LUT-load writes INTO the regcmd region (splitting buffers breaks the LUT chain).
3. ★ PING-PONG OFF (flags=0x1 not 0x5) — the final factor. RKNPU_JOB_PINGPONG (1<<2) double-buffers the register banks and swaps to the next task the instant the current task's register config is done — but a LUT-load has a long-latency SRAM-commit side effect not in the register bank, so ping-pong advances to the exp task before the LUT commits → flaky task4↔task5 stall (non-det counter 4↔5). Off → HW fully completes each task incl. the commit. Rule: ping-pong OFF for any chain with a LUT-load; ON is safe only for register-config- only tasks (uniform matmuls, run_chain_i8). See NPU-Quirks. Task alignment was a red herring (content-driven stride). "SRAM commit timing" was the right mechanism but the fix was disabling ping-pong, not adding a barrier. ORK_SM_PERTASK/ORK_SM_1SUBMIT/ORK_SM_NTASK preserve the alternate paths (gated). Still submit-floor-bound as a standalone op — the fused output-stage path (M4.8 ork_mm_run_f16_act) is the production win; this is the definitive chaining RE + a working single-submit softmax.

• Multi-month, mostly reverse-engineering. M1.3 + M2.2 are the make-or-break; if regcmd can't be generalized across shapes, the project caps at fixed-shape replay.
• Per-architecture graphs (like librkllmrt) and exact-numerics matching are required for quality.
• Maintenance is tied to the NPU driver/hardware.
• It is a research / sovereignty play, not near-term ROI: for plain serving, librkllmrt already delivers prefill and the prefix cache hides most of it. The win is NPU prefill + full openness + features the closed runtime lacks (multi-layer EAGLE-3), together — which nothing else gives.

• rknpu_trace.so — the LD_PRELOAD ioctl/regcmd capture shim (M1.1–M1.2).
• regcmd_diff — perturb-and-diff register localizer (M1.3).
• CPU oracle + mmtest — the correctness reference at every step.

rknpu_trace.c (LD_PRELOAD ioctl interposer, in experimental/ggml-rknpu/) run against the validated [4×32]·[32×16] fp16 matmul (mmtest, result CORRECT). Full submission anatomy captured:

Buffer inventory (6× MEM_CREATE, each one 4 KB page, IOMMU domain 0, core_mask=0x1):

Sequence: ACTION probes (GET_HW_VERSION→0x46495245="FIRE"; GET_DRV_VERSION→0x38c=v0.9.8; POWER_ON→ -1 already-on; SET_PROC_NICE=-19; GET_IOMMU_EN=1 before each alloc) → 6× MEM_CREATE+MEM_MAP+MEM_SYNC(0x3) → MEM_SYNC(→dev) on regcmd/B/A → one SUBMIT (flags=0x5, timeout=6000, task_number=3, task_counter=3, core_mask=0x1, hw_elapse_time=11667 µs) → MEM_SYNC(←dev) on C → MEM_DESTROY.

Findings:

• The whole matmul is 896 bytes of regcmd (handle 1) + a 40-byte task descriptor pointing at it. That's small and tractable to decode.
• MEM_CREATE flags: 0x403 = NON_CONTIGUOUS|CACHEABLE|IOMMU_LIMIT_IOVA_ALIGNMENT; the task buffer adds KERNEL_MAPPING (0x40b).
• IOVA dma_addrs are handed out top-down (0xffffe000, 0xffffd000, …) — the regcmd almost certainly references A/B/C by these device addresses, so M1.3's address-localization should be straightforward by diffing.
• Single core only (core_mask=0x1) for this tiny op — multi-core comes later (M5.3).

Next — M1.2: hook mmap to map handle→CPU pointer, then dump the 896-byte regcmd + the 40-byte task descriptor + A/B/C bytes for this known shape.

rknpu_dump.c (adds mmap/mmap64 hooks → handle→CPU ptr, parses the task descriptor, dumps the regcmd as words). For the [4×32]·[32×16] matmul the task descriptor decodes as one rknpu_task: enable_mask=0x0d, int_mask=0x300, int_clear=0x1ffff, int_status=0x100, regcfg_amount=108, regcmd_addr→0xffffe000 (handle 1). C output is fp32 (0x42000000=32.0) from fp16 inputs.

The regcmd is 108 register-writes as (word0, word1) u32 pairs, and the encoding is decoded:

reg_offset = word0 & 0xFFFF
block_id   = word1 >> 16        # 0x0201, 0x0801, 0x1001 = three NPU blocks
value      = ((word1 & 0xFFFF) << 16) | (word0 >> 16)

Verified by the embedded DMA addresses found in the stream (the patch points for synthesis & resident weights):

Dimension fields are visible too — reg 0x1024 → 0x001f0020 (K=32 / K−1=31), reg 0x1078 → 0x000f000f (N=16 / N−1=15). The three blocks map to register ranges: 0x1xxx↔0x0201, 0x3xxx↔0x0801, 0x4xxx↔0x1001 (CNA feature / DPU / PPU-core, NVDLA-style). Full dump saved at /tmp/rknpu/dump_4x32x16.txt.

Implication: the format isn't opaque — it's flat register writes with addresses embedded as split 32-bit fields. The M1.3 decision gate (could-the-ISA-generalize) is essentially passed; M1.3 now just needs to map which regs encode M/K/N/strides/quant by sweeping shapes.

Next — M1.3: parametric matmul harness (argv M K N), capture regcmds for several shapes, diff to finish the register dictionary; then M2.1 verbatim replay.

Parametric harness (rknpu_mm_param.c, argv M K N) swept four shapes — base (4,32,16), M=8, K=64, N=32 — all CORRECT; regcmds decoded (diffrc.mjs) and diffed. All 108 entries have fixed structure; only values change, and every dimension-dependent field resolves to a closed form:

The remaining ~90 entries are shape-invariant constants (block config, opcodes, int masks). Three NPU blocks are programmed: 0x0201 (CNA / feature+weight setup, 0x1xxx regs), 0x0801 (0x3xxx), 0x1001 (PPU/core output, 0x4xxx). Tooling: rknpu_mm_param.c, diffrc.mjs (decode+diff) in experimental/ggml-rknpu/.

Verdict: the regcmd ISA is decoded well enough to synthesize a matmul for arbitrary (M,K,N): emit the ~90 constants verbatim + compute the ~18 dimension fields + patch 3 addresses. Phase 1 complete.

Next — M2.1: verbatim replay — allocate our own buffers, write the captured regcmd with the 3 addresses patched, RKNPU_SUBMIT directly (no librknnrt), read C, expect ==K.

rknpu_replay.c drives a raw RKNPU_SUBMIT with the captured regcmd and produces the correct result ([4×32]·[32×16] all-ones → C = 32.0, fp32). librknnrt is not involved.

Three things had to be right (each cost a debugging round; documented so nobody repeats them):

1. Device node — the decisive fix. The job only executes via the primary (card) node /dev/dri/card1. On the render node /dev/dri/renderD129 the kernel accepts every buffer and the submit, then the job runs 0 tasks and times out (task counter: 0x0, soft reset). librknnrt uses the render node successfully with its own full setup, but for a from-scratch submit the card node is what works.
2. regcmd is 112 entries (224 words), not 108. Beyond the regcfg_amount=108 register-writes there are 4 trailing PC-control entries (blocks 0x0101/0x0041/0x0081) — the RKNPU_PC_DATA_EXTRA_AMOUNT=4 that drives the PC sequencer. Replaying only the 108 counted entries → PC never starts → 0 tasks. Capturing the full 224 words fixed it.
3. task_number=1 with a single subcore_task[0] for a one-task matmul (not librknnrt's task_number=3 + three subcores, which expects 3 completions that a single regcmd never signals).

Not required (verified revertible): non-cacheable buffers, RKNPU_ACT_RESET, PRIME/dmabuf export. Cacheable buffers + explicit MEM_SYNC (librknnrt-style) work fine on the card node.

Cross-check: independently confirmed by building & running mtx512/rk3588-npu on this board — its synthesized-regcmd matmul also returns RKNPU_SUBMIT 0 with correct output. That project already does M2.2 (parameterized regcmd synthesis) and even runs llama2.c, so Phase 2–3 can build on it (with attribution) rather than re-deriving from scratch.

Next — M2.2: parameterize the regcmd (emit the ~90 shape-invariant constants + compute the M/K/N fields from the M1.3 dictionary + patch addresses) and validate against CPU across a shape sweep — or adopt mtx512's npu_matmul.c synthesis as the foundation.

rknpu_synth.c builds the regcmd from the template + M1.3 closed-form dim fields and validates the NPU result against a CPU reference with real (distinct) data (small ints, exact in fp16). Correct (0 mismatches) for: 4×32×16, 8×32×16, 4×32×32, 8×32×48, 4×64×16, 4×96×16.

Two register/template corrections (all-ones had masked them):

• reg 0x1030 = K*N*2 (weights byte-size), not K*32 — it depends on K and N (the K-only M1.3 sweep couldn't tell).
• The rest of the M1.3 closed forms held.

Operand memory layout (the real find — derived by encoding indices in the data and dumping the buffers librknnrt's regcmd actually reads):

• Feature A: flat row-major [M][K] — K contiguous, not tiled.
• Weights B: [Ktile][N][32], tile-major — K split into 32-channel tiles; within a tile the weights are transposed to output-channel-major (buf[t*N*32 + n*32 + kk] = B[t*32+kk][n]). This reduces to the simple transpose at K=32.

Remaining edge: when K>32 AND N>16 simultaneously, results diverge (C[0] is right, the rest mis-placed) → the weights need nested K×N tiling (N also tiles by 16) and the output needs de-tiling. Single-tile-in-either-dimension is solved; the nested case is the next step. Then M2.3 (int8/w8a8) and Phase 3 (resident weights).

Nested-tiling (K>32 AND N>16) — RESOLVED ✅: it was a weights layout bug, not the node/IOMMU issue first hypothesized (that wrong guess came from comparing only the first 16 buffer words — they matched; a full 4 KB byte-compare showed the tile-1 region diverging). The correct weights layout is [Ntile][Ktile][16][32]: N tiles by 16, K tiles by 32, transposed within a tile (buf[nt*KT*16*32 + kt*16*32 + nl*32 + kk] = B[kt*32+kk][nt*16+nl]). This reduces to the earlier layouts when NT=1 or KT=1 — which is exactly why single-K-tile and single-N-tile both worked and only the doubly-tiled case broke. Feature stays flat [M][K]. M2.2 is now complete: from-scratch fp16 GEMM synthesis correct for arbitrary M/K/N (validated vs CPU through 32×512×128).

Tooling: rknpu_synth.c (synth+validate), layout_probe*.c (operand-layout derivation via librknnrt + dump shim).

The Goal: Discover the undocumented register(s) that enable the NPU to natively process multiple rows ($M&gt;1$) in a single hardware int4 pass, bypassing the need for PC-chaining or separate ioctls.

The Solution & Proof: Through exhaustive search and layout brute-forcing (using tools/i4_multim_fuzz.c and examples/test_layouts.c on the RK3588 silicon), we have definitively proven that the NPU is fully capable of native multi-M execution in INT4 mode.

The breakthrough relies on two key findings:

1. The PPU Stride (0x405c) Constraint: Leaving the output PPU stride register 0x405c at its default value 0 (instead of setting it to (mc-1)<<16 like in INT8) is the critical trigger. Setting it to non-zero corrupts the output and prevents rows $&gt;0$ from being written.
2. The Stride-2 Element Layout Mapping: In INT4 mode, each output element is an int16_t (2 bytes), but the output DMA default logic steps by int32_t (4 bytes) width. This causes the NPU to write computed rows with a stride of exactly 2 rows of int16_t per computed row (256 bytes per row instead of 128 bytes for $N=64$).
   • Therefore, computed logical row m maps to output buffer row 2 * m.
   • The register formula to compute $H$ rows is: mc = 2 * H. For example, setting mc = 32 computes exactly 16 rows, placing them in output rows 0, 2, 4, ..., 30. Setting mc = 64 computes exactly 32 rows, placing them in output rows 0, 2, 4, ..., 62.

Verification: The automated test examples/test_layouts.c was expanded to test large shapes (e.g., $M=32$ which corresponds to mc=64). All 32 rows computed natively in a single hardware submit, and after a simple de-tiling gather copy (reading from even rows 2 * m * N in the output buffer), the NPU output matched the CPU reference bit-exactly across all rows with 0 mismatches.

This completely eliminates the need for the slow, row-by-row PC-chaining fallback (run_chain_i4), unlocking massive prefill performance for INT4.

Two distinct budgets were characterized this session, and only the first was hunted for a lift:

• Activation budget rows·K ≤ 16384 int4 — fuzzed to enlarge (0x1104/0x1014/0x104c bankprobe); no per-task register lifts it → treated as init-time-fixed.
• Weight budget Ncore·K ≤ 131072 int4 — found later (wide-N K=2048 de-tile: 1 block computes, 2 drop). Measured, worked within (guard + N-subslice); NEVER fuzzed to enlarge. Honesty on "independent": the coupling check (M=32 K=1024 N=128, both budgets at limit → bit-exact) proves they're simultaneously satisfiable (enough for the N-subslice), but it's ONE data point — it does NOT distinguish "two separate banks" from "one shared bank ≥147456 (=16384+131072)". Distinguishing needs a test that exceeds one budget while the other is at its limit.
• ★ Open lever: enlarge the WEIGHT budget. Never hunted. If a weight-bank register lifts 131072, wide-N batches in a SINGLE submit (no N-subslice needed). Natural extension of the (activation-only) bank hunt — sweep for a reg that raises the Ncore·K ceiling at fixed K (e.g. K=2048: does any reg let 2+ blocks compute?).

⚠ SUPERSEDED — the "bank SPLIT (D_BANK)" lever below was REFUTED later the same day by the conv .rknn capture: RK3588 has no D_BANK register (weight swept to 4.7 MB, no bank-count reg found). The int4 8-row@K2048 cap is a hardware limit gated by 0x107c, not a reallocatable bank partition. See the SUPERSEDED banner on the "NVDLA banks & batch … D_BANK is the lever" section and regcmd-ISA-Reference.

NVDLA CBUF spec (nvdla.org/hw/v1/hwarch.html): CBUF = 2–32 banks × 4–32 KB, storing BOTH weight and feature data; register D_BANK (0x50bc, CDMA block) = "Number of data banks and weight banks in CBUF."

• Resolves "independent vs coupled": the two budgets are separate bank pools (weight banks vs data banks) — NOT one shared bank. Drops the earlier one-data-point hedge; the coupling check passed because they're distinct pools. Our numbers fit: activation 16384 int4 = 8 KB ≈ 1 data bank; weight 131072 int4 = 64 KB ≈ 2 banks.
• Why no register lifted either budget: NVDLA says max batch is "a hardware limitation in the design specification" — the CBUF total is FIXED; a split register only re-partitions it, can't grow it.
• ★ New untried lever — the bank SPLIT (D_BANK). We only fuzzed the CNA/DPU/PPU present-registers (0x201/0x801/0x1001); the split control lives in NVDLA's CDMA (0x50xx) block — un-swept, or set at init (NVDLA notes per-op programmability "isn't fully detailed"). Finding the RK D_BANK equivalent would let us trade weight banks ↔ data banks: more data banks → more rows at K=2048 (past 8) at the cost of narrower N/submit (fewer weight banks); or the reverse for wide-N. A workload-tunable partition, not free capacity. Fuzz target: the 0x50xx region (if present in the int4 regcmd) + the ACTION/init path.

Direct test (int8-stream config applied to int4 via the static diff): at K=2048 M=16, best = 1 contiguous row (vs 8 batch) — int4 collapses to single-row under the streaming config, i.e. it REQUIRES the batch trigger (0x405c=0/stride-2) for multi-row. Strong preliminary evidence int4 is batch-only (8-row cap @ K=2048 is real). NOT fully conclusive: the config was partial (K-passes/CNA-hint/weight-layout left at int4-batch). The rigorous close is the int8-batch A/B (get the complete verified selector set incl weight layout) — item ② remains the path. Until then, int4 streaming = not-yet-viable, batch is production.

The int8-batch controlled A/B (ork_npu_probe_i8_mm): int8 STREAM baseline correct (8 rows contiguous), but the int4 batch recipe (0x405c=0 + mc_phys=2M) on int8 → GARBAGE (all rows unfound), schedule on OR off. With RESULT 1 (int4+stream config → 1 row), the batch/stream modes are dtype-locked: int4 batch-only, int8 stream-only, neither crosses via these registers. So there's no int8 batch mode to A/B-diff → no transferable selector → int4 streaming is not-yet-viable via register config (int4's 8-row cap @ K=2048 is the real ceiling). Not fully impossible: int8-batch garbage may be a weight-layout mismatch (untested); single shape. Infra kept for a future attempt: ork_i8_fuzz_add/clear, ork_npu_probe_i8_mm, ORK_I8_PROBE_SCHED, fuzzer i8batch mode.

Prior entry overclaimed. NVDLA batch mode is not dtype-specific, and int8 ALREADY batches — its mg_max*64 M-scheduler (128 rows @ K=2048, one weight load) IS multi-row/shared-weight batching. The i8batch experiment only showed that int4's SPECIFIC recipe (0x405c=0 + mc_phys stride-2) garbles int8 — because that recipe is an int4 OUTPUT quirk (int16 clipped into int32 slots → stride-2 packing), not "int8's batch register". int8's native-int32 output batches with CONTIGUOUS output via the M-scheduler; it has no 0x405c=0 quirk. Real axis: activation RESIDENCY. int8 STREAMS activations (128 rows, schedule-limited); int4 holds them RESIDENT (8 rows, CBUF-limited). Open question (unchanged): can int4 stream activations like int8? The conclusive test not yet run: apply int8's FULL streaming config to int4 INCLUDING int8's resident WEIGHT LAYOUT (my stream test used int8's registers but left int4's batch weight tiling — incomplete). int4-streaming = OPEN (not closed), pending the weight-layout test.

Exhausted the board-testable config space: int8-stream registers on int4 (contiguous 0x405c, M-count, 0x1040 schedule swept incl the int8 value 0x84) x BOTH A-layouts (per-row & interleaved) → best 1 contiguous row (cap 8). int4's weight layout is datapath-fixed (can't substitute int8's [NT][KT][32][32]). And the vendor (librknnrt) tiled int4 M across 12 BATCH tasks rather than streaming — a HW-informed choice. Conclusion: int4 is batch-only; streaming is not reachable via regcmd config. 8-row cap @ K=2048 is the real ceiling. (Not metaphysically impossible — a stream weight layout the vendor never used could exist — but out of regcmd reach and argued-against by vendor behavior.)

Since int4 large-wide-N can't stream and the batch only fits/helps SMALL matmuls (Ncore*K<=131072, ~1.4-1.6x), the way to compound the win across a model is round-robin single-core dispatch (librkllmrt-style: round-robin, never core_mask=0x7) of the MANY small matmuls (decode GEMVs, QKV/gate-up, MoE experts): each core batches a whole small matmul independently → (a) fits the per-core weight budget, (b) NO multi-core sync barrier (the prefill-parity wall), (c) cross-matmul overlap (one core's submit hides another's CPU tile/de-tile). Hybrid dispatcher: big matmuls → multi-core N-split batch (current); many small → round-robin single-core batch. This is a scheduling change in run_i4/dispatch, orthogonal to the batch mechanism. Highest-value int4 lever now.

Prior entries loosely called int8's M-scheduler "batching." Corrected, rigorously:

• TRUE BATCH (NVDLA D_BATCH_NUMBER) = weight resident AND all rows' activations RESIDENT in CBUF; capped by BOTH budgets. int4's multi-M (0x405c=0/stride-2, rowsK<=16384 & NcoreK<=131072) IS true batch → 8 rows@K2048.
• WEIGHT-STATIONARY STREAMING (the M-scheduler) = weight resident, activations STREAMED (not CBUF-resident; AGENTS.md "activations need not be CBUF-resident, mg_max*64 is the ceiling"). int8/fp16 use this → 128 rows@K2048. This is NOT batch mode — I was wrong to call it that. int4 uses true batch because it has no working streaming schedule (int8-stream config on int4 → 1 row). int8 uses streaming (16x more rows, no activation-residency cap). Whether int8 has a SEPARATE true-batch mode (real D_BATCH_NUMBER config, not int4's stride-2 recipe) is UNTESTED — my i8 experiment used int4's recipe, which is int4-output-specific and garbled int8's native-int32 output, so it says nothing about int8 true-batch.

NVDLA hwarch spec: batching is precision-AGNOSTIC — "Feature supporting: batch (Yes/No), affects Convolution"; "MAX Batch number 1~32, limited by convolution buffer size" (not precision); and the small NVDLA config is INT8 and ships D_BATCH_NUMBER/D_BATCH_STRIDE. So int8+batch is documented/shipping. => RK3588's int8 datapath SHOULD have a real D_BATCH_NUMBER true-batch mode; ork just uses STREAMING instead (weight-stationary M-scheduler, 128 rows > batch's ~8 — no perf reason to use int8 batch). int8-true-batch on RK is untested (would be D_BATCH_NUMBER in the CDMA/0x50xx block, never swept); pure architectural completeness. Sharper open puzzle: NVDLA gives BOTH modes generally, RK int8 streams, but RK int4 CANNOT stream (tested) — so the anomaly is RK's int4 datapath lacking a streaming schedule (likely a Rockchip impl choice), not batch.

How to fuzz for int8 D_BATCH_NUMBER, bounded + fast-fail:

• WHERE: block 0x2001 / offsets 0x5000-0x50c0 (NVDLA D_BATCH_NUMBER @ 0x5058, D_BATCH_STRIDE) — the one block the int8 mm path never touches; + a few unset 0x201 CNA regs.
• SIGNAL (no "more rows" tell — int8 batch < streaming): reuse the i8batch ROWMAP. baseline stream = physrow=m (contiguous). Flag any (reg,val) where rows are still FOUND but at NON-contiguous physrows (batch stride) = candidate D_BATCH_STRIDE/NUMBER; garbage=wrong; unchanged=inert.
• FAST-FAIL: ORK_I8_PROBE_TO_MS=1500 (kernel job-timeout in-process) + state-resume + blacklist + external timeout backstop + autonomous HA power-cycle. Palette: batch#{2,4,8,16} x stride{N,2N,4N,M*N}. WHY LOW-PRIORITY (recommend NOT pursuing on-board): (1) the 0x2001/0x50xx CDMA block is almost certainly ABSENT from the int8 mm regcmd — can't inject a whole foreign block the sequencer doesn't parse (ignored or wedge); (2) zero perf value (int8 streaming beats batch 16x); (3) injection is hard-wedge-prone. The DEFINITIVE test is not a fuzz — it's a librknnrt CAPTURE (LD_PRELOAD regcmd_capture): does the vendor ever emit an int8 regcmd with D_BATCH_NUMBER set? If the vendor's int8 is always M-scheduler/streaming, int8 true-batch isn't used in practice.

NVDLA describes both modes for all precisions; RKNPU is a Rockchip impl that may expose them selectively. Map:

i8fuzz swept ALL present int8 regs (1504 candidates) for a batch toggle via the layout-change signal. Only 0x1068 relocated cleanly — and it's a uniform output ROW-OFFSET (physrow=m+v), NOT a batch stride (mapped via i8map: v=1→rows shift +1, v=4→+4). 0x1014/0x1028 = partial stride scrambles. NO coherent D_BATCH_NUMBER among present int8 regs. Combined with the 0x2001/0x50xx CDMA block being ABSENT from REGCMD_I8: => RKNPU exposes conv modes PRECISION-SELECTIVELY in the matmul regcmd: int4 = batch-only (0x405c/0x107c present; int4 can't stream), int8/fp16 = streaming-only (M-scheduler; the batch/CDMA block isn't in their mm regcmd). NVDLA supports both for all precisions in the abstract; Rockchip wired each precision for one mode. Matrix: int4 {stream:NO, batch:YES}; int8 {stream:YES, batch:NOT-in-mm-regcmd}; fp16 {stream:YES, batch:untested}. Definitive int8-batch-exists check = librknnrt capture (is D_BATCH_NUMBER/0x50xx ever emitted?), not a board fuzz.

Web survey of RK3588 NPU RE prior art:

• mtx512/rk3588-npu (deepest public regcmd RE): fp16 + int8 matmul ONLY — no int4 matmul, no batch/stream modes.
• Tomeu Vizoso Mesa/Teflon: conv-focused (MobileNet), TRM+NVDLA docs; not matmul-modes/int4.
• jas-hacks (Matharu), P-H Husson: early RKNN matmul-API RE. clehaxze: perf benchmarks only.
• Public consensus: "RK3588 supports W8A8 for LLM, not W4A16" — int4-for-LLM treated as unsupported. => No public prior art on int4 streaming/batch or int8 batch-vs-streaming. Our int4 batch scheduler (0x107c, stride-2) + the streaming-negative + the mode-matrix are NEW / state-of-the-art. CORROBORATION: mtx512 documents CBUF = 16 banks x 64KB (NPU_CBUF_BANKS=16, NPU_CBUF_BANK_SIZE=64KB, 1MB total). This independently confirms our budgets: int4 WEIGHT budget 131072 int4 = 64KB = EXACTLY one CBUF bank; activation budget 16384 int4 = 8KB. The batch-CBUF model we RE'd falls out of the documented bank structure.

⚠ SUPERSEDED — REFUTED later the same day by the conv .rknn capture (see regcmd-ISA-Reference "D_BANK: no explicit bank-count register exists", and REGCMD_CAPTURE_WIP.md). RK3588 has NO D_BANK register. A single-conv .rknn was built (build_conv.py), captured on board, and its weight swept up to 4.7 MB (≫ the ~1 MB CBUF) — no discrete bank-count register appears; conv_big still runs 1 submit / 3-core split. CBUF residency is purely dim-derived via 0x1030/0x107c(=K/16)/0x1010/0x1040, not a bank knob. ⇒ the int4 rows×K ≤ 16384 (8-row @ K2048) cap is gated by 0x107c and no register combination escapes it — a hardware limit, exhaustively fuzzed (the stream2 sweep tops out at exactly 16384/K). The "~15 idle banks, reallocate via D_BANK for 8–15×" claim below is wrong; do NOT chase D_BANK bank-realloc. The real levers within the hard cap are (1) dense int4 K/32 encoding — halve 0x107c (K/16→K/32) to double rows/submit (a packing change), and (2) multi-task STRATEGY C chaining (ork_mm_run_i4_incr / ORK_I4_INCR) — one weight load amortized over arbitrary M by chaining ≤8-row tasks.

NVDLA spec: CBUF is split into WEIGHT banks + DATA banks via a single register D_BANK (0x50bc CDMA / 0x605c CSC, "Number of data banks and weight banks in CBUF"). Batched activations live in the DATA-bank partition ONLY (subset), weight in the weight banks; total banks fixed (RK3588: 16 x 64KB = 1MB, per mtx512). So batch is NOT all-banks — it's the data-bank allocation, and more data banks <=> fewer weight banks (a trade). REFRAME of our int4 numbers: weight budget 64KB = 1 weight bank; activation 8KB = a fraction of one data bank => int4 batch uses ~1 of 16 banks; ~15 idle. The 8-row@K2048 cap is a D_BANK ALLOCATION artifact, NOT a fundamental CBUF limit — reallocating data banks via D_BANK could grow int4 rows/submit ~8-15x toward full CBUF. BUT D_BANK (0x50bc/0x605c) is in the CDMA/CSC region, NOT in the blocks our matmul regcmd carries (0x10xx/30xx/ 40xx); our bankprobe of present CNA regs found no lever. So D_BANK is init-time / out-of-matmul-regcmd — the same absent-block wall as D_BATCH_NUMBER. Lever reachable only via a librknnrt capture or the driver/DT init path (does the vendor set D_BANK differently per dtype/op?), not our per-task fuzz. HIGH-VALUE follow-up.

f16fuzz done (1504 combos): fp16 mirrors int8 EXACTLY. Only clean layout-mover is 0x1068 (=0x1/0x2/0x20000001/0x20000002, MOVED>=6) = the dtype-agnostic OUTPUT ROW-OFFSET (physrow=m+v), NOT a batch toggle. No D_BATCH_NUMBER-equivalent among present fp16 regs (same as int8). FINAL MATRIX: int4 -> TRUE BATCH (0x405c=0 stride-2), validated 1.4-1.6x, act CBUF-resident, cap = data-bank alloc int8 -> STREAMING (M-scheduler), no batch toggle in present regs, only 0x1068 row-offset, cap mg_max*64 fp16 -> STREAMING (M-scheduler), no batch toggle in present regs, only 0x1068 row-offset (== int8) CONCLUSION: in the present matmul regcmd, batch is PRECISION-SELECTIVE (only int4 carries a batch config; int8/fp16 are streaming-only). The int8/fp16 batch levers (D_BATCH_NUMBER, D_BANK) are init-time / out-of-matmul-regcmd (absent-block wall) — reachable only via librknnrt capture or the driver/DT path, NOT our per-task fuzz. This closes the fuzz-reachable mode-mapping; further batch RE needs a capture path.

Replicated the vendor's task_number=N int4 pattern (decoded from a librknnrt w4a4 capture): ONE resident int4 weight, M rows in ONE submit as task[0..1]=full priming + task[2..M-1]=12-CONFIG incremental tasks that advance ONLY the A input (0x1070 += K/2) and C output (0x4020 += N*2); weight IOVA (0x1110) CONSTANT => weight loaded ONCE and reused across all M rows. Implemented as ork_mm_run_i4_incr (src/npu.c, probe tools/i4_incr_probe.c). Key regcmd details: compact task = 16 pairs / 32 words, regcfg_amount=12; PC-chain amount (0x0014) encodes the TARGET size — 0x0037 into a full task, 0x0007 into a compact. 2 full priming tasks are REQUIRED — chaining a full task straight into a compact returns errno 110 (chain-walk fail). Cold output buffer needs a warmup rep.

Result (bit-exact vs CPU int4, single-core, K=512..4096 N=64): 7–28× vs per-row separate submits, 1.3–2.7× vs STRATEGY-A (stride-2 in-task batch). The advantage over STRATEGY A widens past STRATEGY-A's Hcap cap (16384/K rows): at M=64/K=512, incr = 2.71× (per-row cost 9.6→4.1→2.6 µs as M=8→32→64, since the weight is loaded once and amortized — STRATEGY A re-streams per capped submit). NOT Hcap-capped: batches arbitrary M with one weight load. Still matmul-level (dummy data, correctness-checked); end-to-end model validation + production wiring (ork_mm_run_i4 M>1, multi-core) are the follow-ups.

Wired STRATEGY C into the public int4 path (gated ORK_I4_INCR, default off): N-block loop (one incremental chain per 64-wide N-block; A packed once, block-independent), int4 mode-reset on entry, Sk/Sn==1 guard (falls back to STRATEGY A otherwise). BIT-EXACT through the production path — i4 example passes all shapes (M up to 64, N up to 256) both default and ORK_I4_INCR=1. Production-path perf (ork_mm_run_i4, ms/matmul):

• N=64 (single N-block): C WINS 1.6-2.2x — M64/K2048 0.900->0.550, M64/K512 0.457->0.205, M32/K512 0.236->0.128.
• N>64: C LOSES — M64/K2048/N256 A 1.338 vs C 2.143; M128/K2048/N512 A 3.918 vs C 8.046. Reason: STRATEGY A (run_i4_mc) splits N-blocks across 3 CORES; the wired STRATEGY C is SINGLE-CORE (serial N-blocks), so the per-block weight-amortization win is outweighed by the lost 3-core N-parallelism. ⇒ Single-core STRATEGY C only wins at N=64 (or the batched-verify M>1/single-tile regime). To beat the multi-core STRATEGY A at realistic wide N, STRATEGY C must go MULTI-CORE (split the N-blocks across the 3-core pool, mirroring run_i4_mc's threading) — the clear next step. Committed gated/default-off as a validated primitive.

NVDLA (which the RK3588 NPU derives from) documents a technique list (hwarch.html / unit_description) worth mining to optimize our primitives. RK3588 is NVDLA-DERIVED not identical, so each is a PROBE candidate, not assumed present. Mapping:

Kernel-source note (rknpu_job.c, rockchip develop-6.1): the submit path is DTYPE-UNIFORM — no int4/DT_I4 branch, no stub/TODO. So the int4-vs-int8 asymmetry (int4 multi-task vs int8 single-submit M-scheduler) is NOT a kernel gap; it's either a librknnrt userspace choice (bypassable by crafting our own regcmd) or a HW datapath property. Decisive open test: fuzz int4 with int8-style in-task-batch regs (0x4034=M-1, 0x405c= (M-1)<<16) for >1 row in one task. TOP UNEXPLOITED LEVERS from this list: (1) sparse weight compression (decode-bandwidth), (2) second memory interface / weight prefetch. Both PROBE candidates on RK3588.

Q (from the NVDLA unified-datapath reading): NVDLA lists INT4 as a supported precision and implies ONE datapath across precisions (Atomic-C/K parallelism scales inversely with width; no per-precision pipeline) — so int4 "should" stream like int8. Does RK3588 hide it in the driver? Test: i4_multim_fuzz stream2 — beyond the earlier stream (which fixed int4's 0x1044/0x107c + swept 0x1040 -> 1 row), this sweeps the two precision-SIZING regs it held fixed: 0x107c (entries-per-slice = activation-residency/Hcap source) x 0x1044 (K-passes) x 0x1040, with int8's contiguous streaming row-encoding, both A-layouts. Crafts the regcmd DIRECTLY (bypasses librknnrt), so a driver gap can't hide the HW capability. Result: best contiguous rows == the cap EXACTLY (K2048 M32: 8=16384/2048; K4096 M16: 4=16384/4096); NO combo exceeds it. (Mild positive: with the right 0x107c/0x1044, int4 emits CONTIGUOUS output up to the cap, not just stride-2 — but still capped.) ⇒ RK3588 int4 DIVERGES from NVDLA's unified-streaming model. int4 activations are HARD-capped at 16384 resident elements (batch-mode CBUF budget); no register combination escapes it. int4 does NOT single-submit stream like int8 — a genuine HW datapath property, NOT a driver/librknnrt gap (kernel is dtype-uniform; fuzz bypasses librknnrt). So the vendor's 96-task pattern is int4's NATIVE streaming (weight-stationary via multi-task), and STRATEGY C (multi-core incremental) is the correct & only many-row int4 path. Question CLOSED.

User has an RK3576 board at a remote site (offline until a technician restores internet). When reachable, run on it to settle "generation-wide int4 resident-cap vs RK3576-completed-it":

1. i4_multim_fuzz stream2 (K2048 M32, K4096 M16) — does RK3576 exceed the 16384/K int4 activation cap (= single-submit int4 streaming that RK3588 lacks)?
2. mm_cap create-probe of the mixed types (7/8/12 w4a16, 11 w4a8) — does RK3576 ACCEPT native w4a16 where RK3588 rejects (-5)? (Web hint: "larger models benefit from W4A16 memory efficiency on RK3576" — untested.)
3. Validate src/soc/rk3576.c params (currently validated=0, inherited from RK3588): cbuf_elems, nmax, ks. Reachable evidence so far argues these chips share the SAME int4 generation (RKNN header groups RK3588/3576 identically for int4 alignment; "fundamentally same NPU architecture") — so expect the same cap; the w4a16 create-probe is the most likely place to find a real per-SoC difference.

Investigated the two levers flagged above (sparse-weight compression, second memory interface). Both turn out unavailable on RK3588 — RK3588 is an NVDLA DERIVATIVE that doesn't populate these NVDLA-documented features:

• Second memory interface (on-chip SRAM weight prefetch): NOT viable. The rknpu driver HAS the hook (RKNPU_MEM_TRY_ALLOC_SRAM=1<<8, RKNPU_MEM_TRY_ALLOC_NBUF=1<<9, RKNPU_GET_TOTAL/FREE_SRAM_SIZE actions), but tools/sram_probe queries NPU SRAM total = 0 bytes on the board; no NPU sram node / reservation in the DT; nothing in probe-time dmesg. So there's no on-chip SRAM to allocate weights into — the lever has no hardware to stand on here. (Not a DT-reservable win at userspace level; would need the SoC to populate it.)
• Sparse weight compression: NOT accessible. No sparse/compress/prune hook anywhere in rknn_api.h / rknn_matmul_api.h (no API surface), and no compression-enable register in our captured matmul/conv regcmd. Driving NVDLA-style CSC weight-decompression would need deep regcmd RE + a way to feed compressed weights that RKNN never exposes — a speculative, uncertain track, NOT a near-term lever. ⇒ Neither NVDLA "unexploited lever" pays off on RK3588. The genuinely-shipped decode/int4 lever remains STRATEGY C (multi-core incremental batch). Reinforces: RK3588 implements a SUBSET of the NVDLA technique menu.

Earlier entry ("NPU SRAM=0 -> no HW, not viable") was WRONG (user: the SRAM is ~1MB, just not turned on). Corrected from the rknpu v0.9.8 source + board DT:

• RK3588 config has nbuf_size=0 (no NBUF — the 1MB NBUF is the RK3576 config). On RK3588 the on-chip fast memory is the SYSTEM SRAM, and the driver wires it to the NPU via a DT rockchip,sram phandle (rknpu_find_sram_resource -> of_parse_phandle(dev->of_node,"rockchip,sram",0) -> ioremap -> sram_mm allocator; then RKNPU_MEM_TRY_ALLOC_SRAM=1<<8 allocates from it; GET_TOTAL_SRAM_SIZE reports it).
• The board HAS system SRAM (DT nodes sram@ff001000, sram@10f000), but npu@fdab0000 has NO rockchip,sram phandle (only iommus) -> driver never finds an SRAM region -> GET_TOTAL_SRAM_SIZE=0 (what sram_probe saw). ENABLEMENT (kernel/DT change + reboot, not userspace): add to the npu node a rockchip,sram = <&npu_sram>; phandle pointing to an ~1MB region carved from a system SRAM block, then reboot. Driver then registers it and RKNPU_MEM_TRY_ALLOC_SRAM works. (NPU uses IOMMU here, so the driver must IOMMU-map the SRAM region too.) EXPERIMENT once enabled: alloc a hot weight tile / activation in the 1MB SRAM, run a matmul reading it, measure bandwidth vs DRAM (~11GB/s) — directly targets the weight-DMA-bound decode wall. CAVEAT: 1MB holds only a hot TILE (int8 K2048 -> N<=512; int4 K2048 -> N<=1024), not the full per-token weight stream, so the benefit is a hot-subset / activation-residency win, not a whole-model one. Still the one on-chip-bandwidth lever RK3588 has.

Checked whether the sparse-weight NVDLA lever is real on RK3588 via the vendor toolkit (rknn-toolkit2 2.3.2, which exposes sparse_infer / compress_weight / model_pruning in rknn.config):

• sparse_infer=True → EXPLICITLY REJECTED: E config: rk3588 does not support sparse inference (ValueError).
• compress_weight=True / model_pruning=True → NO-OP: built fine but produced BYTE-IDENTICAL .rknn size (317638) even on a 60%-zeroed conv weight (would shrink hugely if compression engaged). So no compression. ⇒ RK3588 has no usable weight sparsity/compression. Vendor toolkit (which enables sparse_infer for OTHER family SoCs) disables it for rk3588 specifically — authoritative that the datapath isn't wired. Considered a fuzz approach (user idea): fuzz for a register that makes a DENSE op produce garbage (=candidate sparse-enable), then validate with compressed weights. SOUND in general, but not completable on RK3588: (1) "produces garbage" is too noisy — nearly any mis-set mode reg garbles output, can't isolate sparse; (2) phase-2 validation needs weights in the real compressed format, which the RK3588 toolkit refuses to produce (no oracle); (3) vendor says unsupported. So the garbage-fuzz + validate-with-compressed strategy is the RIGHT method to try on a chip where sparse EXISTS — add it to the RK3576 task (does rknn-toolkit2 ACCEPT sparse_infer for rk3576? if so, capture its compressed regcmd+format there).

Enabling the ~956KB system-SRAM for the NPU (RKNPU_MEM_TRY_ALLOC_SRAM / GET_TOTAL_SRAM_SIZE) needs TWO parts; DietPi ships NEITHER, Armbian ships BOTH:

1. Device tree: add rockchip,sram = <&rknpu_sram> to npu@fdab0000 + carve a region from syssram@ff001000. On this board the whole 0xef000 (956KB) was claimed by rkvdec0_sram(0x78000)+rkvdec1_sram(0x77000); headless we re-carved: expanded rkvdec0_sram to 0xef000, dropped rkvdec1_sram, removed both rkvdec device rockchip,sram refs, pointed the npu at it. Verified live (running DT shows npu.rockchip,sram + only rkvdec-sram@0; dmesg: mpp_rkvdec2 ...rkvdec-core: could not find phandle sram = rkvdec correctly lost it). DTB edit is surgical (10-line diff), backed up (.orig-*.bak on board + local).
2. Kernel config: CONFIG_ROCKCHIP_RKNPU_SRAM=y (gate at rknpu_drv.c:1512 — needs CONFIG_NO_GKI=y [set] AND CONFIG_ROCKCHIP_RKNPU_SRAM [DietPi: # not set] AND iommu_en). Without it, rknpu_find_sram_resource() + rknpu_mm_create() are compiled out -> GET_TOTAL_SRAM_SIZE=0 even with the DT wired. rknpu is BUILT-IN, so this requires a KERNEL REBUILD (or booting an Armbian kernel that has it). Status: DT done+verified; blocked on kernel rebuild with CONFIG_ROCKCHIP_RKNPU_SRAM=y. Payoff still bounded (956KB = hot tile). Revertible via the .bak DTB.

Tested the "swap to Armbian's vendor kernel" idea by pulling Armbian's linux-headers + linux-dtb vendor-rk35xx .debs (v26.5.1, SAME 6.1.115-vendor-rk35xx base as the board):

• Armbian .config: # CONFIG_ROCKCHIP_RKNPU_SRAM is not set — IDENTICAL to DietPi (off).
• Armbian rk3588-rock-5b.dtb: npu node has NO rockchip,sram; rkvdec0/1 claim the whole syssram — IDENTICAL to DietPi/vendor-default. ⇒ CONFIG_ROCKCHIP_RKNPU_SRAM is a bool with no default y (Kconfig), absent from the rockchip base defconfig AND both distros' configs. So NO stock kernel (DietPi or Armbian vendor) ships it. The "Armbian has it" recollection = a PATCHED image (ezrknpu / manual RK3588_NPU_SRAM_usage.md), not stock. A kernel SWAP won't enable SRAM. ONLY path: custom kernel rebuild with CONFIG_ROCKCHIP_RKNPU_SRAM=y (rknpu is built-in, tristate but =y, so an out-of-tree module can't override it — needs a real image rebuild) + the DT (already done on the board, revertible via .bak). Weigh against the bounded 956KB (hot-tile) payoff.

Task #39 spent weeks reverse-engineering rkllm's fast int8 "M-fold" matmul on RK3588. The decisive lesson for future SoC work: the on-NPU weight (B) pack layout does NOT need board RE — it is published in the RKNN toolkit header. Read it, don't grind the board.

Source: airockchip/rknn-toolkit2 → rknpu2/runtime/Linux/librknn_api/include/rknn_matmul_api.h (conversion rknn_B_normal_layout_to_native_layout). The B matrix native layout is (N/subN, K/subK, subN, subK), K innermost, with per-chip (subN, subK):

General weight byte offset (int8):

woff(k,n) = ((n/subN)*(K/subK) + (k/subK)) * (subN*subK) + (n%subN)*subK + (k%subK)

The activation and output are NC1HWC2: input A = nc16 (C2=16 int8 / 8 fp16, M folded into WIDTH) = (k/16)*(M*16)+m*16+(k%16); output C = c4 (C2=4, int32) = (n/4)*(M*4)+m*4+(n%4). K,N pad up to the next multiple of subK/subN.

Validated: RK3588 int8 woff reproduces rkllm's matmul bit-exact — tools/re/validate_layout.c packs ork's own A(nc16)/W(woff), replays a captured rkllm regcmd, de-tiles c4, compares CPU ref → 0/9728 mismatch.

Porting recipe for a new Rockchip SoC: (1) read its (subN,subK) from the header table above; (2) pack the weight with woff; (3) activation nc16 / output c4 are shared; (4) confirm bit-exact with the validate_layout pattern (capture one real regcmd via the rknpu_dump.so shim, replay with ork-packed operands). This turns a multi-week board-RE grind into a header lookup + one confirmation run. See also [NPU-Quirks] "M-fold weight layout & capture-replay gotchas".