Exp 2026 07 08 INT4 BChain Batch Chain - oRKLLM/ork-driver GitHub Wiki

Exp 2026-07-08 β€” INT4 batchΓ—chain (ORK_I4_BCHAIN): bit-exact multi-M prefill at production K

Result

A new int4 prefill path β€” chain the H-row native batches, tiled at the weight-bank width β€” is bit-exact and beats the default run_i4_mc:

  • Microbench (i4 ORK_I4_BENCH, single-core vs 3-core default, K=2048):
    • M=128 N=2048: 10.88 β†’ 6.69 ms/matmul (1.63Γ—)
    • M=128 N=6144: 28.61 β†’ 20.29 ms (1.41Γ—)
    • M=256 N=2048: 23.86 β†’ 13.41 ms (1.78Γ—)
  • Matmul, multi-core (N-chunks split across 3 cores): M=128 N=2048 10.89 β†’ 4.15 ms (2.62Γ—), M=128 N=6144 28.85 β†’ 12.42 ms (2.32Γ—) vs default.
  • End-to-end (ork_bench, Qwen3-1.7B-UD-Q4_K_XL, hadamard W4A4, .orkpack), all PPL 1.8991 (bit-exact): baseline 22.28 β†’ 1-core bchain 27.11 (+21.7%) β†’ 3-core bchain 31.02 tok/s (+39.2%); NPU run portion 3545 β†’ 1929 ms (1.84Γ—). Prefill is now becoming quant-bound (quant 36% / run 57%).

Multi-core = split the bank-width N-chunks across cores (i4_bchain_worker, core_mask=1<<i, per-core scratch β€” the safe independent-submit pattern).

CBATCH (batch-in-compact) β€” NOT VIABLE on RK3588 (2026-07-08)

Follow-on attempt: carry the H-row batch in the lean compact/STRATEGY-C incremental task (weight-DMA regs not re-issued β†’ resident) instead of BCHAIN's full 116-reg batch tasks, to trim per-task regcmd ~7Γ—. Hardware rejects it. A multi-row (mc_phys>1) compact returns errno-110 (chain-walk timeout) for every register configuration swept (ORK_I4_CB_REGS bitmask over 0x107c/0x1020/0x102c/0x1010/ 0x4034/0x4038/0x3014; ORK_I4_CB_AMT chain-amount) β€” while the same base format with mc=1 (the proven STRATEGY-C 1-row compact) works. The batch needs the full task's weight-DMA context; a compact that omits it can't sustain mc>1. This matches the vendor: their int4 batch is task_number=96 of 1-row compacts, never multi-row.

β‡’ The duality is asymmetric on this silicon: you can chain batches (BCHAIN, full tasks) but not batch chains (CBATCH). BCHAIN's full-task-per-batch is REQUIRED, not an inefficiency β€” the per-task regcmd cost is inherent to how the NPU batches int4. BCHAIN is the final form.

Mechanism

Two int4 primitives were each leaving throughput on the table:

  • STRATEGY A (in-task multi-M batch, synth_i4 mc>1): computes H = 16384/K rows/submit but re-loads the weight every submit β†’ M/H weight streams.
  • STRATEGY C (ORK_I4_INCR, incremental-task chain): weight loaded once, but each chained task is a single row β†’ M tasks of per-task overhead. Measured slower than default at wide-N.

ORK_I4_BCHAIN is the union: chain the H-row batches so the weight loads once and each task does H rows β€” M/H batch-tasks, one weight stream per bank.

The bank-width tiling (the crux)

The naΓ―ve "chain full-N batches" fails at K=2048, N>64 β€” it hits the 0x1040 K-reduction "poison pill" wall (only row 0 computes). Diagnosis: a native multi-M batch is valid only within one weight bank. One 64-block int4 weight at K=2048 = 64Γ—2048Γ—0.5 B = 64 KB = exactly one CBUF bank, and there is no D_BANK to hold a second (see regcmd-ISA-Reference). So the batch must not straddle banks.

β‡’ Tile N by the bank width Wb = 131072/K columns (64 @ K=2048, 128 @ K=1024, 256 @ K=512) β€” each chunk is a wall-free single-bank batch. Chain H-row batches across all (M-group Γ— N-chunk) tasks in one submit. This matches the pass/fail exactly: M=8 K512 N256 (Wb=256, one bank) always worked; K2048 N256 (needs 4 banks) failed until tiled into four 64-wide chunks. After tiling: all shapes maxerr=0.

Implementation

run_i4_bchain in src/npu.c, routed from ork_mm_run_i4, gated ORK_I4_BCHAIN (default OFF, single-core). RE knobs ORK_I4_BCH_H (batch rows), ORK_I4_BCH_W (chunk width). Reuses the validated STRATEGY-A synth_i4(mc>1) regcmd + stride-2 A-pack + de-tile verbatim (no new register RE); the only new structure is the chaining + the bank-width N-tiling.

Post-BCHAIN: "quant-bound" was WEIGHT-LOAD-bound (a cold-start artifact)

With the run halved, prefill looked quant-bound (quant 36% / run 57%). A wload/actq profile split (ggml-ork.cpp, ORK_VERBOSE) showed the truth: activation FWHT+int4-quant is actqβ‰ˆ0 β€” the "quant" time is ~entirely the per-forward-pass weight-load from the .orkpack (wloadβ‰ˆ621ms). It's independent of M (amortizes with prefill length: P=128 30.8 β†’ P=228 36.1 tok/s) and unaffected by ORK_WCACHE_BUDGET_MB (budget defaults to ~RAMβˆ’1 GiB; no eviction here).

Root cause = a benchmark artifact: ork_bench's warmup decoded 1 token (M=1), which supports_op routes to CPU (M=1 < ORK_MINM) β€” so it never warmed the NPU prefill path, and the timed prefill paid the one-time cold weight-load. Warming with a prefill-shaped (M>1) batch populates the wcache first: warm int4 prefill = 38.24 tok/s vs cold 30.79 (+24%). In real serving (persistent process, wcache never evicts at this budget) prefill runs warm β€” the load is paid once and amortized. Fix committed (tools/re/ork_bench warmup β†’ 32-token M>1 batch). β‡’ NEON on the activation path is moot; the only residual prefill cost is the NPU run itself (BCHAIN's domain; 3-core scaling is contention-capped ~1.6Γ—).

The 0x1040 K-widening / bank-widening RE β€” CLOSED as a hardware wall (2026-07-08)

The one remaining lever was to make the NPU run cheaper by fewer/wider submits (the warm prefill floor is ~1.9 s = 74k tasks of 8 rows Γ— 1 bank). Two axes to widen a submit; both proved hardware-capped:

  • Rows/submit: 16384/K = 8 @ K=2048 (activation cube, exhaustively fuzzed) β€” and BCHAIN already uses it.
  • N-cols/submit (banks): probed exhaustively with the repaired ork_npu_probe_i4_mm + i4_multim_fuzz (ORK_FUZZ_N): base multi-M is 1-bank-only at K=2048 (2/4/8/16 banks β†’ 0 full-width rows); a 0x1040 sweep at all-banks (N=1024) and the full sweep of every present register at 2 banks both found no config spanning >1 bank. No reachable register escapes it (consistent with no-D_BANK). Harness note: the probe had regressed to errno-22 (int8 regcfg_amount on the shared c->task); fixed to REGCMD_I4_N/2. Run base with ORK_I4_PROBE_TO_MS low β€” a 60 s-timeout hang + SIGTERM-mid-submit hard-wedged the board once.

⚠⚠ RETRACTED 2026-10-12 β€” the conclusion below is REFUTED. Read this banner first.

"No further register lever exists" is wrong, and this page is the one a newcomer reads first. Measured on both boards (wiki: Exp-2026-10-12 INT4-BCHAIN Dispatch Envelope §§1–14, and Exp-2026-10-12 W4A4 Tier Trade Β§8):

  • Wb reaches 320 at K=2048 once H is paired via 0x1040 β€” 3626 β†’ 1707 Β΅s, 2.09Γ—, bit-exact, program count 512 β†’ 56. The shipped Wb = (131072/K)&~63 is 5Γ— below the legal ceiling.
  • H reaches 16–64 with DBNK β‰₯ 2; the ceiling is DBNKΒ·16384/K, not a fixed 8.
  • "8 rows Γ— 1 bank" is not a hardware lock. Wb=64 (NBc=1) is exempt from any weight-tile limit at every K from 512 to 8192 β€” 256 KB/program runs fine.
  • A per-program capacity law proposed on 2026-10-12 was itself refuted the same day: equal byte counts land on both sides, the CBUF bank split is inert, and raising H rescues a byte-identical tile. The real rule is a race with a LOWER bound on H (H_race(K,Wb), rising with K), which is why measurements taken at M=H=2 β€” the worst corner of a race β€” looked like capacity.
  • The two boards are the same law at two DRAM-clock points, not two dies: .236 (2112 MHz, LPDDR4X) has the uniformly larger envelope and the slower DRAM, and lowering clock grows the envelope monotonically on both parts.

What stands: the per-bank tiling mechanism, the +21.7% end-to-end result, and the capacity argument for W4A4. What does not: that the landscape was "fully mapped and closed".

β‡’ A single int4 submit is hardware-locked to 8 rows Γ— 1 bank; BCHAIN's per-bank tiling is the optimal int4 schedule and the ~1.9 s run floor is genuine hardware. The int4-prefill lever landscape is fully mapped and closed: code (BCHAIN), benchmark honesty (warm), config (ubatch) β†’ ~47 tok/s ceiling; no further register lever exists. (Absent-block D_BANK would need the capture path, already concluded absent.)

Head-to-head: native W4A4 (BCHAIN) vs inflated JIT W4A8 β€” same model, warm, ork_bench (2026-07-08)

The decisive comparison. Same Qwen3-1.7B-UD-Q4_K_XL.gguf, warm, both via ork_bench; W4A8 = ORK_MIXED_DISPATCH (int4 weights inflated to int8 at pack, computed on the W8A8 datapath β€” int4 as storage, not compute):

W4A4-BCHAIN (SOTA) Inflated W4A8
prefill ubatch 128 37.9 tok/s 92.8 (2.4Γ—)
prefill ubatch 512 47.6 tok/s 154.6 (3.25Γ—)
PPL 1.899 1.420
resident orkpack 657 MB 1.4 GB (2.1Γ—)

Inflated W4A8 dominates native W4A4 on BOTH speed (~3Γ—) AND quality β€” W4A4 wins only on RAM (Β½). Cause: W8A8's mg_max*64 M-tile amortization (100s of rows/submit) vs W4A4's HW-locked 8-rowsΓ—1-bank submit (the wall proven above); and int8 activations are far more faithful than int4-act+Hadamard. β‡’ the productive q4 path on RK3588 is inflateβ†’W8A8; native W4A4 compute is a capacity-only fallback (fit >4 GiB-IOVA models at Β½ weight RAM), never a speed/quality play. (Inflated-W4A8 154 t/s β‰ˆ 53% of rkllm w8a8 ~292 β€” the residual gap is the known large-M prefill kernel, not the quant path.)

Perspective

int4 prefill is still 0.093Γ— of rkllm's w8a8 (27 vs 292 tok/s) β€” int4 is not prefill-competitive vs W8A8 on RK3588, and this doesn't change that. But it is the first structural int4-prefill gain that is both faster than the default and bit-exact, and it composes with the multi-core split. It's most valuable in the small-N/MoE/verify regime where int4 batching already pays off.