Board 10.3.0.236, kernel 6.1.115-vendor-rk35xx-fence (record it: r104's numbers are on the same kernel, so they compare; anything from the -sram kernel does not). Governors verified before every timing: DDR performance @2112 MHz, cpu4..7 performance @2.4 GHz. Workload: tools/train_tiny_e2e.c, one full forward+backward+AdamW step of a full-size Qwen3.5-0.8B hybrid (24 layers, 18 GDN + 6 attention, d=1024, ffn=3584, vocab=248320), B=1 T=1024, 752M fp32 parameters, 4 A76 threads (taskset -c 4-7, OMP_NUM_THREADS=4).

Correctness unchanged at the fp16-operand level (worst spot-check gradient tensor 2.451e-03 against 3.351e-03 before this work), the RMSNorm positive control still failing at 3.1e-1, and make test green on the board. The last row is an environment variable, not a library default — §6 says why.

The two bottom rows are a within-sequence A/B and the single-core arm ran FIRST, i.e. from the position that eats mode entry, the first DMA and the page warm. It still won by 1.49x, which is well outside the ~1.13x arm-position bias this board is known for.

A note on the baseline. The earlier r104 report quotes 140.67 s/step for the same tree and the same fixture; rebuilt here with cc -O2 -fopenmp -DWITH_NPU it is 112.626 s/step with a byte-identical loss (0.899750). The 1.25x is build flags, not work. Every ratio on this page is against 112.626 — a number measured on the same board, the same kernel and the same hour.

head_dX is the tied output head's dX: fp16 M=1024 K=248320 N=1024, 520 GFLOP. It ran at 65.9 GFLOP/s (7.9 s). The forward at the same FLOPs ran at 923 and dW at 310. A 14x spread on one arithmetic shape is not a hardware limit; it is a schedule.

The chain:

1. c->soc->ks = 2048, so K=248320 becomes 122 K-slices of Kp=2048 (plus a 512 tail).
2. orki_f16_sched(K) is (K&(K-1))==0 && K>=128 && K<2048. 2048 is excluded by its own upper bound, so every full fp16 K-slice lands on sched=0.
3. sched=0 leaves 0x1040 at its template default — one CBUF data bank — so orki_f16_mcap(2048,0) = 16384/2048 = 8 rows per program.
4. colsplit's fp16 arm caps a chain at 512 programs. 122 * ceil(1024/8) = 15616, so it declines and the whole matmul falls to orki_run's single-core K-slice loop.
5. A submit re-streams its entire Kp*Nc weight slice from DRAM regardless of how many rows it carries (AGENTS.md, "weight-DMA amortization"). 15616 submits x 4 MB = 64 GB of weight DMA. 64 GB / 7.9 s = 8.3 GB/s — exactly the single-core DRAM weight-stream rate this project has measured before. The shape was weight-DMA bound and the arithmetic was never the issue.

orki_f16_ks() — the fp16 twin of orki_int8_ks — returns 1024, and is called at every fp16 K-slicing site (orki_pack, ork_f16_mm_scratch, ork_f16_mm_repack, orki_run, both colsplit arms, the fp16 blob loader, the shape planner) so pack and run cannot drift apart. Kp=1024 is inside the sched=1 window, where the measured cap is 180224/1024 = 176 rows: 22x fewer weight re-streams for identical arithmetic. ORK_F16_KTILE overrides for A/B.

Note what the lever actually is. Weight traffic is K*N*2*ceil(M/cap) — independent of Kp except through the cap. A smaller slice is not itself the win; landing inside the sched window is. Going below 1024 would buy a taller cap at the cost of more host-side K-slice accumulate passes.

int8 is NOT moved onto this. It has its own orki_int8_ks (also 1024) and its own measured mg_max*64 ceiling; orki_f16_mcap's comment explains why the two must not be unified.

f16_mm_rows_probe M K N 3 (fp64 reference on sampled rows, wrong-column positive control), under npu_guard.sh -- timeout, 0 new soft-reset/fault/timeout dmesg lines after each run's start, every case PASS with its control FAILING. "run 2" (warm) GFLOP/s:

The effect appears exactly where the mechanism predicts and nowhere else. (relerr also improved on the sliced shapes, 3.1e-7 -> 2.0e-7, which is what more and smaller fp32 host-accumulated partials should do.)

head dX is still single-core: colsplit's 512-program cap is now hit at 243 * 6 = 1458. Raising it would cost roughly 2 GB more IOVA for this one matmul (508 MB CONTIG weight + 508 MB gathered A + ~1 GB of per-core partials) against a 4 GiB domain, on the code path with the fp16 concurrent-fetch CDMA-wild history. Not attempted.

orki_pack's fp16 arm and ork_f16_mm_repack each carried the same four-deep loop building the [NT][KT][16][32] tile, with the read strided by N: at N=248320 that is one cache line fetched per element, 254M of them, on every call (the harness re-tiles per use from a scratch pool). Now one shared orki_f16_tile, walking kk outermost so each read is a contiguous run of 16 fp16 and the strided access lands on the 1 KB destination tile, which is L1-resident; parallel over the N-tile via ork_parallel_for, as the int8 arm already was.

Dispatching the pool unconditionally was a regression, and a sharp one: the GDN chunk path re-tiles 64-wide weights ~10^5 times a step, and waking/joining the pool for 4 N-tiles cost +6 s/step (NPU gdn 18.5 -> 24.4 s) with nothing else changed. orki_f16_tile now runs serial below 64 Ki elements.

orki_run's run_loop also had two scalar loops that are not scalar work: the A gather (a run contiguous in both operands — now one memcpy per row) and the K-slice accumulate (now NEON, with the first slice storing instead of accumulating into a known zero). Both bit-identical.

Several fp16 paths run one program per task and therefore need the whole K as a single [NT][K/32][16][32] cube: the fused ork_f16_mm_run_silu (and the rsqrt LUT built on it in npu/norm.c), ork_f16_mm_run_f16out, and the stream/chain entries. All gated on Sk==1, which at the old K-slice meant K<=2048 and at 1024 would mean K<=1024 — silently refusing a range that used to work. make test caught it (test_bmm [rsqrt-lut] run failed at n=2048).

orki_f16_bfull re-lays the slices into that cube once and caches it on the weight. It is not w->Bbc: Bbc concatenates whole slices, and the tile is N-tile-major, so slice s's nt block does not follow slice s−1's. The re-lay is one memcpy per (nt, slice). K<=1024 still takes the untouched Sk==1 branch, so everything the suite already covered is byte-for-byte the old path, and the Sk>1 branch is validated by the very case that caught the bug.

Measured per step, before -> after:

What mattered, in order:

• Cross-entropy. The old form called libm exp twice per logit in double — 508M calls a step. exp(z-lse)/M is exp(z-mx)/(s*M), so the second pass is a multiply by a constant: one NEON vexpq_f32 pass, the sum accumulated in double (two float64x2 lanes — 248k positive terms would lose ~1e-3 relative in fp32), the normalisation folded in. 15.5x, and the step-1 loss moved closer to MLX (|d| 4.59e-5 -> 4.90e-6).
• The fused scale+transpose. The operand scaling is an exact power of two, and the old path materialised a whole extra fp32 copy to apply it before calling the library transpose. On the 1 GB head operands that pre-scale pass cost more than the transpose. Folding the multiply into the 4x4 register-transpose block removes it; vmulq_f32 by an exact power of two is the exponent add the scaling wants, and unlike a raw exponent add on the bit pattern it gets zero, subnormal and overflow right.
• Everything else is the same recipe: NEON intrinsics plus an OpenMP row/block split, with a scalar fallback kept. The GDN depthwise conv transposes its 4 taps once so the kernel can run along the (contiguous) channel axis; its backward parallelises over channels, because dqkv and conv->g both accumulate across t.

Two deliberate non-parallelisations: rms_bwd runs serial when it has a dw to accumulate (a cross-row reduction — splitting it would make the gradient depend on the thread count), and the matmul kernels do not nest (omp_in_parallel()).

The chunkwise Gated-DeltaNet backward issues 110,592 matmuls a step — 18 layers x 16 heads x 16 chunks x 24 [MM] steps — of 64-wide shapes, about 1 MFLOP each. On the NPU they cost 19.1 s of submit plus 6.1 s of host fp16 prep plus 4.4 s of scalar recurrence: 29.5 s, and nearly all of it is the per-submit floor. 174 GFLOP at an effective ~6 GFLOP/s.

Batching was considered and not taken. ork_bmm_fp16_fused would coalesce the 16 heads into one ioctl, but it packs B ephemerally per batch (bcreate + tile + sync + free), and even a perfect batching gets 18 x 16 x 24 = 6912 submits x ~167 us = 1.15 s of floor plus the 6.1 s of fp16 operand prep that does not go away. A CPU path pays neither.

So: the 16 heads are independent, and with the chunk matmuls on the CPU the head loop becomes an OpenMP loop over 4 A76 cores with per-thread chunk scratch (it must stay serial for --gdn npu — one ork_npu context is a single submit stream). The matmuls themselves get a 4x16 register-blocked NEON nn/tn and a 4x4 k-vectorised nt, allocating nothing (the old nt transposed B into a malloc on every one of 10^5 calls).

29.5 s -> 3.86 s wall (7.6x), and 110,592 NPU submits disappear. It is also more accurate: fp32 operands instead of fp16.

Per step, GDN on the CPU, steps 2..4 of a 4-step run (step 1 excluded as warm-up). Both NPU core counts, since §6's finding makes the choice a real one:

Everything outside the NPU bucket is identical between the two, as it must be — the only thing that changed is how the matmuls were dispatched.

The NPU is now 73% of the step and that is where the next work is. Per class, per step:

Costing every call at the rate its own shape achieves standalone predicts ~9.5 s of run against a measured 29.65 s. The residue is ~20 s spread over 849 calls — about 24 ms per call, essentially independent of shape, which is what a fixed per-call cost looks like and not what a bandwidth or FLOP limit looks like. It is not the mode-transition layer (ork_npu_enter returns at its from==to guard) and it is not the weight re-tile (measured separately, 3.2 s). The remaining suspects are inside ork_dyn_begin_colsplit / ork_dyn_end: the A gather into maf[0], three pthread_create/join per call, and — for Sk==1 shapes, which is all of attention and every K<=1024 linear — the f32 output copy-back, which takes the single-threaded path in ork_dyn_end because the per-core parallel accumulate is gated on oSk > 1. (A side effect of the K-slice change: K=2048..6144 now have Sk=2..6 and so moved ONTO the parallel accumulate.)

Attention is the extreme case: 19 GFLOP/s in-model against 237/277 standalone, 27.8 ms per call against 2.0 ms. This was already true before any of this work (5.71 s at baseline).

ORK_NPU_MC=1 caps the core budget at one, which for fp16 bypasses ork_dyn_begin_colsplit entirely and runs orki_run's single-core loop. The same step, same tree, same board:

One core is 1.49x faster end to end than three, and 7.8x faster on attention. The gradient table and the step-1 loss are IDENTICAL to the digit between the two (worst tensor 2.451e-03, loss |d| 3.930e-05), and --break rms still fails at 3.120e-01 on both, so this is a cost result, not a correctness one — test_f16colsplit also still shows nc=3 == nc=1 bit-exact.

Ruled out by measurement, not by reading: it is not an ACT_RESET (ORK_DEBUG_RESET=1 over two full steps counts zero), not the mode-transition layer (ork_npu_enter returns at its from==to guard), not the weight re-tile (timed separately). What is left sits inside ork_dyn_begin_colsplit / ork_dyn_end: the A gather into maf[0], three pthread_create/join per call, and — for Sk==1, which is all of attention and every K<=1024 linear — the f32 output copy-back, which takes the single-threaded ork_dyn_end path because the per-core parallel accumulate is gated on oSk > 1; that path's own comment calls it "5-9x slower". A side effect of the K-slice change is that K=2048..6144 now have Sk=2..6 and so moved ONTO the parallel accumulate, which is part of why linear suffers less than attention.

The default was NOT changed. One workload is not enough to flip a library-wide policy that ggml-ork and every other fp16 caller depends on, and the ratio must depend on per-call size — colsplit was built for, and validated on, far larger matmuls than a 1024x256x1024 attention tile. The finding is recorded so the next person measures ORK_NPU_MC=1 before assuming three cores help on a many-small-fp16-matmul workload. Root-causing the fixed cost is the open item.

RESOLVED, same day — see Exp-2026-10-09 Colsplit Whole-Buffer Cache Maintenance. The fixed cost is whole-buffer DMA cache maintenance over dead bytes: orki_bsync() syncs b->size and the per-core colsplit scratch only ever grows, so every call was charged ~1.47 GB of cache maintenance (24.6 ms of a 30.4 ms call) against a 3.5 ms submit. All three suspects named above are refuted by the phase profile — pool dispatch+join is 0.024 ms/call (the pool is already persistent), the Sk==1 copy-back is 1.838 ms, and the "A gather" cost is its trailing whole-buffer orki_bsync, not the memcpy. Syncing the live extent: 3-core 44.752 -> 24.964 s/step, three cores now beat one by 1.21x, and the ORK_NPU_MC=1 advice above is retired.

Minor, measured, declined: the 36 CPU fallbacks are GDN in_proj_a/in_proj_b dX, whose contraction is 16 and fails K%32. Padding K to 32 with zero rows would be exact and would put them on the NPU — but they now cost 0.01 s/step in total in the NEON cpu_nn. There is nothing to win; the instrumentation to answer that question is in the harness (cpu-fallback <s> in <n> calls).

./tte <fixture> --all npu --scale 1 --cpufast --noemu --steps 0:

• step-1 loss C 0.899708998 vs MLX gpu 0.899704099 — |d| 4.90e-06 (was 4.59e-05).

• 14 spot-check gradient tensors, max|C-ref|/max|ref| vs MLX gpu. On the all-NPU arm the worst moved from 3.35e-03 (layers.23.self_attn.k_proj.weight) to 4.43e-03 (layers.0.input_layernorm.weight) — eight of the fourteen improved, six got worse, all inside the fp16-operand band. It is ATTRIBUTED rather than argued, with two reference knobs:

  arm	worst tensor	step-1 loss |d|
  before any of this work	3.351e-03	4.59e-05
  ORK_CE_REF=1 ORK_RMS_REF=1 (both reassociations off)	3.336e-03	4.90e-06
  ORK_CE_REF=1 (NEON RMSNorm backward only)	3.671e-03	4.90e-06
  ORK_RMS_REF=1 (NEON cross-entropy only)	3.744e-03	4.90e-06
  default (both NEON)	4.432e-03	4.90e-06
  ORK_F16_KTILE=2048 (old K-slice, both NEON)	6.489e-03	3.37e-05
  --gdn cpu, default numerics (the recommended config)	2.451e-03	3.93e-05

  Reading it: the host shift is the two reassociations (the NEON cross-entropy's vexpq_f32 and RMSNorm-backward's four-lane dot), each worth a few 1e-4 on a tensor already at 3.3e-3, and with both off the table sits at the pre-change level. The K-slice change goes the other way — at matched host numerics it improves the worst tensor 6.49e-03 -> 4.43e-03 and the loss 3.37e-05 -> 4.90e-06, which is what more and smaller fp32-accumulated K partials should do. The loss is identical across all four host arms, so the forward is untouched and the loss result is the library's.

  Caveat worth stating: "worst of 14" is an order statistic over quantities that are themselves fp16-level noise, on a tensor whose own max|g| is 2.95e-03. Everything here is one band, and the config that actually gets recommended is the best of the lot. Keeping the NEON forms is the call: 15.5x on cross-entropy, the loss strictly closer to MLX, and reversible with one env var.

• positive control --break rms still FAILS at 3.110e-01 (all-NPU) / 3.120e-01 (--gdn cpu), unchanged — the test did not go blind.

• make test on the board: ALL TESTS PASSED, including test_f16_wide_k, test_f16_narrow_n, test_f16colsplit, test_slice_f16, test_bmm and test_f16_load (pack -> dump -> load bit-exact, which is what proves pack and run agree on the new slice). tests/sbc_attest.txt refreshed and verified against the committed tree. The 56 dmesg lines during the suite are all RKNPU: soft reset (deliberate per-call ACT_RESETs) — zero non-reset fault/timeout/ not-dispatching lines.

• The model static golden had to be regenerated: the K-slice moves the fp32 K-partial accumulation, so the NPU output bytes change by design — exactly the "deliberate output-changing edit" case the golden rule names. Its CPU-vs-NPU tolerance check was unaffected (maxabs 0.001239 at ref 7.38 for NL=1, 0.02413 at 25.2 for NL=12).

• make test also caught a real regression, which is the point of it: test_bmm's [rsqrt-lut] at n=2048 started failing, and an A/B of that one test against the pre-change tree (pre PASS / post FAIL) confirmed it was mine rather than the known int16-LUT flakiness. The fused single-submit paths need the whole K as one tile and gated on Sk==1, which at the old K-slice meant K<=2048; see §3's orki_f16_bfull. After the fix [rsqrt-lut fused] OK M=8 n=2048 maxrel=0.0006, identical to the pre-change arm.

• Behaviour-preserving check on the final tree: moving orki_f16_ks into f16/regcmd.c and compacting the accumulate reproduced the gradient table and the loss to the digit (worst 2.451e-03, loss |d| 3.930e-05).

On the board, in an ork-driver checkout, with the MLX fixture at $F:

make -j8 libork_npu.a f16_mm_rows_probe
cc -O2 -fopenmp -DWITH_NPU -Iinclude -Isrc -Itools -o tte tools/train_tiny_e2e.c libork_npu.a -lm -pthread

# one shape, alone, fp64 reference on sampled rows + a wrong-column control that must FAIL
sudo tools/util/npu_guard.sh -- timeout -s TERM 600 \
     env OMP_NUM_THREADS=4 taskset -c 4-7 ./f16_mm_rows_probe 1024 248320 1024 3

# the step: correctness, then the positive control, then timing
sudo tools/util/npu_guard.sh -- timeout -s TERM 3600 env OMP_NUM_THREADS=4 \
     taskset -c 4-7 ./tte $F --all npu --gdn cpu --scale 1 --cpufast --noemu --steps 0
sudo tools/util/npu_guard.sh -- timeout -s TERM 3600 env OMP_NUM_THREADS=4 \
     taskset -c 4-7 ./tte $F --all npu --gdn cpu --scale 1 --cpufast --noemu --steps 0 --break rms
sudo tools/util/npu_guard.sh -- timeout -s TERM 3600 env OMP_NUM_THREADS=4 \
     taskset -c 4-7 ./tte $F --all npu --gdn cpu --scale 1 --cpufast --nocheck --noemu --steps 4

# A/B the library change against itself
... env ORK_F16_KTILE=2048 ...           # the old 2048 K-slice
# A/B the host numerics
... env ORK_CE_REF=1 ORK_RMS_REF=1 ...   # libm-double cross-entropy + scalar RMSNorm-backward dot

Every NPU run goes through sudo tools/util/npu_guard.sh -- timeout -s TERM <s> <cmd> (the guard holds an flock for the command's lifetime, so check-and-launch is not a race), governors are verified before timing, and kernel faults are counted by dmesg timestamp after each run's start — never by line count, because the ring rotates.