Experiment — Weight-DMA amortization (raise the M-tile cap to the 0x1040 schedule max)

Date: 2026-06-30 · Status: ✅ KEPT (default on, no knob) · Path: ork-driver · Commit: 6775334

Hypothesis (the "per-row feed" / single-core gap)

Full-K prefill closed most of the prefill gap but left a residual single-core "per-row" deficit (~20% of an int8 core's peak; rknn ~2–3.5× our per-row throughput), attributed loosely to the "systolic activation-feed." This experiment characterizes that gap from first principles instead of trusting the prior framing.

What the gap actually is — weight-DMA-bound, not row/compute-bound

Parametric mc_prof sweeps, single-core, K=2048 N=2048 (NPU pinned at 1.0 GHz, the max):

• µs/submit is FLAT vs rows: mc=4→493, mc=8→494, mc=16→501, mc=31→516. Processing 4 rows costs the same as 31 — rows are nearly free.
• µs/submit scales ∝ N: N=512→228, N=2048→510, N=4096→914 ≈ 130µs fixed + 0.19µs/N-col.

What scales with N but not rows = the weight load (K×N bytes). ~4 MB at N=2048 in ~380µs ≈ 11 GB/s = DRAM bandwidth. Conclusion: each M-tile submit re-streams the entire K×N weight from DRAM, and that dominates. The lever is therefore rows per weight-stream — a bigger M-tile amortizes the one weight load over more rows (M/mc reloads instead of M/31).

(This also explains why earlier "tile size is throughput-neutral" tests looked flat: the M-tile was pinned at R-1=31 by the cap below, so they never varied the thing that matters. And the 0x1010 resident-row hint / ORK_RCAP is measured neutral — it changes neither correctness nor speed.)

The bug: a disproven R-1 cap throttling the schedule-valid mg_max*64

The code computed chunk = mg_max*64 (the 0x1040 K-reduction schedule limit) then threw it away with if (chunk > R-1) chunk = R-1, where R = pow2_floor(2*cbuf/K) ≈ 31. That R-1 / "CBUF-resident rows" limit was a wrong RE finding: activations stream (they need not be CBUF-resident), and the real bit-exact ceiling is the schedule itself, mg_max*64.

Validated (full CPU-ref, all rows): mc = mg_max*64 is exactly correct and mc+1 miscomputes, at every K:

K	old cap R-1	real ceiling mg_max*64	mg_max*64 + 1
512	31	704 ✓	705 → FAIL
1024	31	320 ✓	321 → FAIL
2048	31	128 ✓	129 → FAIL (mism 123k)
3584	31	64 ✓	65 → FAIL
4096	31	64 ✓	65 → FAIL

The break is the schedule register clamping below 0x1b once mg = ceil(mc/64) is too large.

Implementation

Replace if(chunk > R-1) chunk = R-1 with chunk = mg_max*64 at all full-K M-tile sites (run, mcworker, run_multicore sizing, chain_fullk_mcap_i8). No knob — default on (a validated bit-exact improvement). cbuf_elems no longer sets the int8 M-tile (only the neutral 0x1010 hint); fp16 and the wide-K (K>4096) K-slice path keep their own caps. Stale comments corrected; docs in AGENTS.md "Weight-DMA amortization".

Correctness

make test fully green (test_matmul, quant, i4, layer, decode, model, test_sn3, test_chain_i4, test_affinity). New tool tools/sparse_check.c = fast sampled correctness for shapes whose full O(M·N·K) ref takes minutes.

Benchmark (mc_prof, µs/matmul, N=2048, governors pinned)

shape		before (R-1, mc=31)	after (mg_max*64)	speedup
K=2048	1-core	10691	5125	2.09×
K=2048	3-core	4522	2778	1.63×
K=3584	1-core	18155	12116	1.50×
K=3584	3-core	—	5008	—

(3-core scaling ratio drops — single-core sped up more than multi-core — so test_speed's scaling floor was lowered and its absolute-latency guards tightened to the new numbers.)

Verdict

✅ Kept, default on. The single-core "per-row" gap was weight-DMA amortization all along — the M-tile was capped ~2–4× too small by a false CBUF-rows finding. Raising it to the schedule-valid mg_max*64 gives ~2.1× single-core / ~1.6× three-core at K=2048, ~1.5× at K=3584, bit-exact.

Open follow-ups: (1) end-to-end prefill gain (prefill is partly CPU-bound — quant/resolve — so the matmul win is partially hidden; measure with llama-bench).

Follow-up 2026-06-30 — wide-K (K>4096) is STRUCTURAL; fp16 cap is REAL (both closed)

Wide-K prefill (K>4096) does NOT get the amortization — both big-M-tile routes wedge the NPU:

• Full-K-in-one-op past K≤4096: NOT VIABLE. Bit-exact map (single-core, clean board, vs a working K-slice baseline at the same K): K=6144 mc=16 (=R) → WEDGE, mc=32 → WEDGE; K=8192 mc=8 (=R) → WEDGE; K=18944 mc=64 → WEDGE. The K-slice baseline at every one of those K is bit-exact. So it's the full-K submit itself (96+ on-chip K-passes + the 0x1040 schedule clamped to 0x1b at huge Kp) that faults past the envelope — not a small-mc/large-mc boundary. Root: full-K needs the M rows resident (R=pow2_floor(2·cbuf/K)), and R collapses as K grows (16@K6144 → 8@K8192 → 4@K18944), below even the K-slice's own per-slice tile.
• Category B (force the Kp=2048 K-slices to the full-K-style mg_max*64=128 tile via sched=1): WEDGE. The K-split Bb tile layout ≠ full-K Bf, so sched=1/mc>R on a partial accumulated slice faults (it is NOT the same submit as the validated full-K Kp=2048 tile, as first assumed).
• Viable wide-K path remains the existing sched=0 K-slice (chunk=(RB/2)/Kp≈28 at Kp=2048, already near the R=32 resident-row budget; ≤~10% headroom, and mc>R wedges like the above).

fp16 M-tile cap: the comment was RIGHT (not conservative like int8's R-1). fp16's sched=1 M-scheduler is bit-exact at mc≤8 and garbage at mc≥9 (K=2048), pinned exactly — a real limit, so the int8 mg_max*64 lever does NOT transfer to fp16. Validating this surfaced + fixed a latent correctness bug: the single-core run() loop enabled fp16 sched=1 for Kp≥2048 (missing the Kp<2048 guard the multi-core path has) → fp16 single-core K≥2048, M>8 produced GARBAGE (untested: examples are multi-core). Fixed (commit 4815fe6) + single-core regression tests added (test_matmul fp16/int8, i4 int4).

Net: the weight-DMA M-tile lever is an int8, K≤4096 win (~2×); wide-K and fp16 are at their structural limits. Tools: tools/sparse_check.c, tools/fp16_sparse_check.c.