Exp 2026 07 27 K Slice Size Sweep and the Sched Optimum - oRKLLM/ork-driver GitHub Wiki

K-slice-size sweep: why KS=1024 is the int8 prefill optimum (the sched flag decoded)

Board-validated on RK3588 (10.3.0.236, governors=performance, DDR 2112 MHz / A76 2.4 GHz). Shape: Qwen2.5-7B ffn_down, K=18944, N=3584, M=228 β€” the widest-K matmul in the 7B, and the one that forces the K-slice accumulate path (K > kmax=4096, so it cannot ride the full-K Bf envelope). Probe: mc_prof 228 18944 3584 <iters> with ORK_KTILE=<KS> (per-core copy/submit/ accumulate breakdown, blocking mcworker). Throughput reported as GMAC/s = 15.48e6 / (Β΅s/matmul) (228Β·18944Β·3584 = 15.48 G MAC).

TL;DR

The K-slice size KS (default 1024) is a sharp, provable optimum at 1024 β€” every other value is 30–60 % slower. The cause is not the slice count or the "cube K": it is a single code flag, sched = (Kp==1024 || Kp==512), that gates two coupled effects at once, and only Kp∈{512,1024} turns it on:

  1. The M-tile size (mcworker, src/npu.c:3890): sched=1 β†’ chunk = mg_max*64 (=320 rows at Kp=1024); sched=0 β†’ chunk = (RB/2)/Kp (β‰ˆ54 rows). The int8 matmul is weight-DMA-bound (each M-tile submit re-streams the whole KpΓ—Ncore weight slice from DRAM regardless of row count β€” see Weight-DMA Amortization), so a 320-row tile streams the slice once for M=228 while a 54-row tile streams it ~5Γ—. This is the dominant term.
  2. The 0x1040 K-reduction schedule (synth_i8, src/npu.c:854): sched=1 writes a K-tuned value (base = 177 βˆ’ 15Β·(Kp/512 βˆ’ 1), stepped per 64-row group); sched=0 leaves the template default. This is the same register behind the small-K zero-output boundary (Exp-2026-07-12 Matmul Shape Boundaries); the formula is bit-exact-validated only at Kp∈{512,1024} for int8.

The two are inseparable: the big weight-amortizing M-tile (mg_max*64) is only bit-exact when the matching 0x1040 schedule is programmed, and that schedule is validated only at Kp∈{512,1024}. So you cannot buy the big M-tile at an arbitrary Kp β€” leaving 1024 forces sched=0, which forces the small (RB/2)/Kp M-tile, which forces the ~5Γ— weight re-stream.

The sweep (M=228, ffn_down 18944Γ—3584)

KS (ORK_KTILE) K-slices sched M-tile (chunk) M-tiles for M=228 weight re-streams/slice GMAC/s
512 37 1 704 1 1Γ— 486
768 25 0 74 4 4Γ— 513
896 22 0 63 4 4Γ— 536
1024 19 1 320 1 1Γ— 764 β—„ peak
1056 18 0 54 5 5Γ— 554
1120 17 0 51 5 5Γ— 574
1184 16 0 48 5 5Γ— 579
2368 8 0 24 10 10Γ— 483
4736 4 0 12 19 19Γ— 313
2048 β€” 0 β€” β€” β€” WEDGED core0 (unverified sched=0 at that Kp; needed sudo reboot)

(RB = 2Β·CBUF = 114688, CBUF = cbuf_elems = 57344 int8; RB/2 = 57344, so (RB/2)/Kp at Kp=1056 β†’ 54.)

Why 1024 and not 512 (both are sched=1)

512 gives an even bigger M-tile (704 rows) but 37 K-slices vs 1024's 19. Each K-slice carries fixed per-slice overhead independent of the compute: a chained PC submit, an A re-tile gather ([mco][Kp] host memcpy β€” mcworker, src/npu.c:3924-3928), and a host int32 partial-accumulate. 512 doubles that fixed cost, so despite the larger tile it measures slower (486 vs 764). 1024 is the larger of the two validated sched=1 points β†’ the fewest K-slices while its 320-row tile still covers M=228 in a single pass. That balance is the peak.

Correction to an earlier framing

An intermediate "17/19-slice bracket" (KS 1056/1120 vs 1024) was described as "holds the M-tile at 320, so it isolates the schedule reg." That was wrong. sched=0 forces the (RB/2)/Kp M-tile (54/51 rows), so the bracket varied the M-tile (320β†’54) and the schedule together β€” it does not isolate either. The honest reading of the data: the M-tile / weight-DMA is the dominant term, and sched is the gate that grants access to the big tile. (A clean isolation would require programming the big M-tile with sched=0, which miscomputes β€” the coupling is exactly the point.)

M-dependence of the optimum

Kp=1024's 320-row M-tile covers M ≀ 320 in one pass. For larger prefill batches (M > 320) it needs β‰₯2 M-tiles and the balance point could shift; for the M ≀ 320 regime (our 228-token ubatch) 1024 is ideal. This is the roofline/M-tile interplay β€” arithmetic intensity β‰ˆ M-tile rows, ridge β‰ˆ 128 rows for this shape (see Prefill NPU Utilization).

What this closes, and where the live lever is

The K-slice axis is exhausted. Every Kp is either {512, 1024} (sched=1, fast) or not (sched=0, ~5Γ— weight re-stream). 1024 is the provable optimum; the KS default is correct; finer values (768/896) and the "even-split" multiples-of-8 (all sched=0) are all slower. KS is not a lever.

Two remaining levers for the wide-K path, in priority order:

  1. Pipeline the host int32 partial-accumulate against the NPU (the live lever). mc_prof @ M=228 measured ~94 ms/core serial accumulate vs ~235 ms NPU ioctl β†’ a 1.69Γ— ceiling if the accumulate overlaps the next slice's submit. No schedule change, wedge-safe. This is the one that can still move the 7B ffn_down.
  2. Extend the sched=1 band to larger Kp (HW-RE, uncertain). Validating the 0x1040 formula + mg_max*64 M-tile at Kp=2048/4096 would let the deep-K matmul use fewer efficient slices β€” but mg_max shrinks as Kp grows (Kp=4096 β†’ M-tile 64 < 228 β†’ 4 tiles), so even a validated sched=1 at 4096 would not beat 1024 at M=228; the win, if any, is at much larger M. Wedge-prone (Kp=2048 wedged core0 in this sweep). Lower priority.

Full lever inventory (int8 prefill matmul) β€” for reference

# lever effect / status
A M-tile = mg_max*64 weight-DMA amortization; dominant single-core term; coupled to sched. [Weight-DMA-Amortization]
B sched / 0x1040 schedule gates the big M-tile; validated Kp∈{512,1024} int8, pow2 K∈[128,2048) fp16. [this page, Exp-07-12]
C K-slice count ceil(K/KS) fixed per-slice overhead (submit + A-gather + accumulate); fewer better β†’ 1024 > 512.
D host int32 partial-accumulate ~94 ms/core @M=228; 1.69Γ— overlap ceiling β€” the live lever.
E A re-tile gather [mco][Kp] per-slice host memcpy (g_mc_copy).
F N-tiling / 3-core balance non-even N=3584/3 imbalance; slowest core gates. [open #36]
G multi-core (3 cores) ~3Γ— if balanced.
H DDR governor / DMA BW ~30 GB/s Γ· 3 β‰ˆ 10 GB/s/core; performance governor mandatory (parked ~halves).
I input zero-copy (A) βˆ’17 % full-K prefill (default on); output ZC ~0 end-to-end.
J weight residency (IOMMU domains) resident avoids per-call re-import.
K doorbell vs blocking submit NEUTRAL for prefill (compute-bound, submit-idle ~7 %); the payoff is decode/latency. [Exp-07-16]
L full-K Bf envelope (K ≀ 4096) avoids K-slicing entirely; ffn_down K=18944 cannot use it β†’ forced onto this path.
M compute-bound ceiling M-sweep ridge ~128 rows, plateau ~89 Β΅s/row; arithmetic intensity β‰ˆ M-tile.

Reproduce

# governors=performance first
for KS in 512 768 896 1024 1056 1120 1184 2368 4736; do
  sudo env ORK_KTILE=$KS ORK_MM_TIMEOUT=3000 timeout 90 ./mc_prof 228 18944 3584 10
done   # GMAC/s = 15.48e6 / (Β΅s/matmul); single-stream; sudo reboot on any wedge

Related: PC-chaining M-tiles & the rkllm schedule Β· Weight-DMA Amortization Β· Matmul Shape Boundaries (the 0x1040 boundary) Β· Optimization Roadmap.

⚠️ **GitHub.com Fallback** ⚠️