Exp 2026 07 27 K Slice Size Sweep and the Sched Optimum - oRKLLM/ork-driver GitHub Wiki
Board-validated on RK3588 (10.3.0.236, governors=performance, DDR 2112 MHz / A76 2.4 GHz).
Shape: Qwen2.5-7B ffn_down, K=18944, N=3584, M=228 β the widest-K matmul in the 7B, and the
one that forces the K-slice accumulate path (K > kmax=4096, so it cannot ride the full-K Bf
envelope). Probe: mc_prof 228 18944 3584 <iters> with ORK_KTILE=<KS> (per-core copy/submit/
accumulate breakdown, blocking mcworker). Throughput reported as GMAC/s = 15.48e6 / (Β΅s/matmul)
(228Β·18944Β·3584 = 15.48 G MAC).
The K-slice size KS (default 1024) is a sharp, provable optimum at 1024 β every other value is
30β60 % slower. The cause is not the slice count or the "cube K": it is a single code flag,
sched = (Kp==1024 || Kp==512), that gates two coupled effects at once, and only Kpβ{512,1024}
turns it on:
-
The M-tile size (
mcworker,src/npu.c:3890):sched=1βchunk = mg_max*64(=320 rows at Kp=1024);sched=0βchunk = (RB/2)/Kp(β54 rows). The int8 matmul is weight-DMA-bound (each M-tile submit re-streams the wholeKpΓNcoreweight slice from DRAM regardless of row count β see Weight-DMA Amortization), so a 320-row tile streams the slice once for M=228 while a 54-row tile streams it ~5Γ. This is the dominant term. -
The
0x1040K-reduction schedule (synth_i8,src/npu.c:854):sched=1writes a K-tuned value (base = 177 β 15Β·(Kp/512 β 1), stepped per 64-row group);sched=0leaves the template default. This is the same register behind the small-K zero-output boundary (Exp-2026-07-12 Matmul Shape Boundaries); the formula is bit-exact-validated only at Kpβ{512,1024} for int8.
The two are inseparable: the big weight-amortizing M-tile (mg_max*64) is only bit-exact when the
matching 0x1040 schedule is programmed, and that schedule is validated only at Kpβ{512,1024}. So you
cannot buy the big M-tile at an arbitrary Kp β leaving 1024 forces sched=0, which forces the small
(RB/2)/Kp M-tile, which forces the ~5Γ weight re-stream.
KS (ORK_KTILE) |
K-slices | sched |
M-tile (chunk) |
M-tiles for M=228 | weight re-streams/slice | GMAC/s |
|---|---|---|---|---|---|---|
| 512 | 37 | 1 | 704 | 1 | 1Γ | 486 |
| 768 | 25 | 0 | 74 | 4 | 4Γ | 513 |
| 896 | 22 | 0 | 63 | 4 | 4Γ | 536 |
| 1024 | 19 | 1 | 320 | 1 | 1Γ | 764 β peak |
| 1056 | 18 | 0 | 54 | 5 | 5Γ | 554 |
| 1120 | 17 | 0 | 51 | 5 | 5Γ | 574 |
| 1184 | 16 | 0 | 48 | 5 | 5Γ | 579 |
| 2368 | 8 | 0 | 24 | 10 | 10Γ | 483 |
| 4736 | 4 | 0 | 12 | 19 | 19Γ | 313 |
| 2048 | β | 0 | β | β | β |
WEDGED core0 (unverified sched=0 at that Kp; needed sudo reboot) |
(RB = 2Β·CBUF = 114688, CBUF = cbuf_elems = 57344 int8; RB/2 = 57344, so (RB/2)/Kp at Kp=1056 β 54.)
512 gives an even bigger M-tile (704 rows) but 37 K-slices vs 1024's 19. Each K-slice carries
fixed per-slice overhead independent of the compute: a chained PC submit, an A re-tile gather
([mco][Kp] host memcpy β mcworker, src/npu.c:3924-3928), and a host int32 partial-accumulate.
512 doubles that fixed cost, so despite the larger tile it measures slower (486 vs 764). 1024 is the
larger of the two validated sched=1 points β the fewest K-slices while its 320-row tile still
covers M=228 in a single pass. That balance is the peak.
An intermediate "17/19-slice bracket" (KS 1056/1120 vs 1024) was described as "holds the M-tile at 320,
so it isolates the schedule reg." That was wrong. sched=0 forces the (RB/2)/Kp M-tile (54/51
rows), so the bracket varied the M-tile (320β54) and the schedule together β it does not isolate
either. The honest reading of the data: the M-tile / weight-DMA is the dominant term, and sched
is the gate that grants access to the big tile. (A clean isolation would require programming the big
M-tile with sched=0, which miscomputes β the coupling is exactly the point.)
Kp=1024's 320-row M-tile covers M β€ 320 in one pass. For larger prefill batches (M > 320) it needs β₯2 M-tiles and the balance point could shift; for the M β€ 320 regime (our 228-token ubatch) 1024 is ideal. This is the roofline/M-tile interplay β arithmetic intensity β M-tile rows, ridge β 128 rows for this shape (see Prefill NPU Utilization).
The K-slice axis is exhausted. Every Kp is either {512, 1024} (sched=1, fast) or not (sched=0,
~5Γ weight re-stream). 1024 is the provable optimum; the KS default is correct; finer values (768/896)
and the "even-split" multiples-of-8 (all sched=0) are all slower. KS is not a lever.
Two remaining levers for the wide-K path, in priority order:
-
Pipeline the host int32 partial-accumulate against the NPU (the live lever).
mc_prof@ M=228 measured ~94 ms/core serial accumulate vs ~235 ms NPU ioctl β a 1.69Γ ceiling if the accumulate overlaps the next slice's submit. No schedule change, wedge-safe. This is the one that can still move the 7Bffn_down. -
Extend the
sched=1band to largerKp(HW-RE, uncertain). Validating the0x1040formula +mg_max*64M-tile at Kp=2048/4096 would let the deep-K matmul use fewer efficient slices β butmg_maxshrinks as Kp grows (Kp=4096 β M-tile 64 < 228 β 4 tiles), so even a validatedsched=1at 4096 would not beat 1024 at M=228; the win, if any, is at much larger M. Wedge-prone (Kp=2048 wedged core0 in this sweep). Lower priority.
| # | lever | effect / status |
|---|---|---|
| A | M-tile = mg_max*64 |
weight-DMA amortization; dominant single-core term; coupled to sched. [Weight-DMA-Amortization] |
| B | sched / 0x1040 schedule |
gates the big M-tile; validated Kpβ{512,1024} int8, pow2 Kβ[128,2048) fp16. [this page, Exp-07-12] |
| C |
K-slice count ceil(K/KS)
|
fixed per-slice overhead (submit + A-gather + accumulate); fewer better β 1024 > 512. |
| D | host int32 partial-accumulate | ~94 ms/core @M=228; 1.69Γ overlap ceiling β the live lever. |
| E |
A re-tile gather [mco][Kp]
|
per-slice host memcpy (g_mc_copy). |
| F | N-tiling / 3-core balance | non-even N=3584/3 imbalance; slowest core gates. [open #36] |
| G | multi-core (3 cores) | ~3Γ if balanced. |
| H | DDR governor / DMA BW | ~30 GB/s Γ· 3 β 10 GB/s/core; performance governor mandatory (parked ~halves). |
| I | input zero-copy (A) | β17 % full-K prefill (default on); output ZC ~0 end-to-end. |
| J | weight residency (IOMMU domains) | resident avoids per-call re-import. |
| K | doorbell vs blocking submit | NEUTRAL for prefill (compute-bound, submit-idle ~7 %); the payoff is decode/latency. [Exp-07-16] |
| L | full-K Bf envelope (K β€ 4096) |
avoids K-slicing entirely; ffn_down K=18944 cannot use it β forced onto this path. |
| M | compute-bound ceiling | M-sweep ridge ~128 rows, plateau ~89 Β΅s/row; arithmetic intensity β M-tile. |
# governors=performance first
for KS in 512 768 896 1024 1056 1120 1184 2368 4736; do
sudo env ORK_KTILE=$KS ORK_MM_TIMEOUT=3000 timeout 90 ./mc_prof 228 18944 3584 10
done # GMAC/s = 15.48e6 / (Β΅s/matmul); single-stream; sudo reboot on any wedgeRelated: PC-chaining M-tiles & the rkllm schedule Β·
Weight-DMA Amortization Β·
Matmul Shape Boundaries (the 0x1040 boundary) Β·
Optimization Roadmap.