Exp 2026 06 26 MoE Decode Partition and Submit Floor - oRKLLM/ork-driver GitHub Wiki

MoE Decode Partition (M1b) + the "21 ms submit floor" that wasn't

Date: 2026-06-26 Β· SoC: RK3588 (10.3.0.236), governors pinned (DDR 2112 MHz / A76 2.4 GHz), -t 4, orkllm up (idle, non-contending) Commits: ggml-ork edf5e855b (M1b hot-expert partition) Β· ggml-ork a72692188 (cold-path vec_dot reuse) Β· ork-driver dd6e7e6 (v0.3.5; chain/stream/load primitives) Β· probe tools/moe_m1_probe.c Knobs: ORK_MOE_NPU=1, ORK_MOE_HOT=<K> (hot experts/tensor), ORK_MOE_HOT_GIB (byte budget), ORK_MOE_COLD_THREADS, ORK_PROFILE=1, ORK_OFF=1 (force all-CPU ref)

Continues M1a persist/load + the 4 GiB IOVA wall and M3 routing-skew sizing. This is the decode partition (M1b) wired into ggml-ork, plus an adversarial verification that overturns a wrong "MoE-on-NPU decode is dead" reading.

1. M1b β€” the decode partition, and fixing the broken live MoE path

The live ggml_backend_ork_mul_mat_id_i8 path (default round-robin pool ORK_MOE_CACHE=384) wedged LFM2.5-8B-A1B before generating: CREATE: Bad address (EFAULT on DRM_IOCTL_RKNPU_MEM_CREATE) β†’ graph_compute -1 β†’ llama_decode -3.

Diagnosis β€” over-residency at the 4 GiB IOVA cap, NOT a bad regcmd. The pool is keyed per-shape; all 22 MoE layers Γ— 32 experts churn one 384-slot pool Γ— 3 shapes β†’ far more than 4 GiB of fresh DMA buffers β†’ the IOMMU rejects MEM_CREATE. Proof: ORK_MOE_CACHE=8 generates coherent text, 0 CREATE failures, exit 0. The expert shapes themselves run fine.

Fix (edf5e855b) β€” a static hot-expert partition in the MUL_MAT_ID int8 handler:

  • Backbone (attn/router/norms/dense FFN) stays resident in the wcache.
  • Hot cache: top-K experts per layer-tensor pinned NPU-resident (LRU within the cap, IOVA freed on evict), run via ork_mm_run_chain_i8 with a per-task run_i8 fallback.
  • Cold experts β†’ CPU.
  • Global byte budget (ORK_MOE_HOT_GIB, default 2.5 GiB) gates admission so backbone + hot experts never exceed the 4 GiB IOVA window; over-budget experts degrade to CPU instead of EFAULTing.

Gate (LFM2.5-8B-A1B, deterministic): output matches the ORK_OFF=1 CPU reference for ~15 tokens then diverges by one token = expected int8-activation drift βœ…. Peak resident 1.805 GiB (K=8) / 2.000 GiB (K=12), both < 4 GiB βœ…. Zero CREATE failures, exit 0, NPU idle/responsive after every run βœ….

2. Cold-path optimization (a72692188) β€” reuse ggml's kernel, don't hand-roll

ork is a buffer-less BLAS-style backend: when offload_op returns true the scheduler routes the entire MUL_MAT_ID op to ork, so cold experts can't simply "stay on the CPU backend" without touching scheduler internals. Instead the handler reuses ggml's own ggml_get_type_traits_cpu(type)->vec_dot (the asimddp/SDOT kernel) against the raw quantized weight rows + the activation pre-quantized to vec_dot_type β€” eliminating the old path's full Q4_Kβ†’f32 plane dequant and scalar triple-loop, fanned across a threadpool. Cold expert ~590 Β΅s β†’ effectively free relative to the rest (β‰ˆ1/35th of the apparent NPU hot cost). Symbol resolves at runtime against libggml-cpu.so; no CMake change.

3. β˜… The "21 ms/expert submit floor" β€” VERIFIED AS AN ARTIFACT, not hardware

A profiling pass reported the partition was bottlenecked by a ~21 ms/expert "NPU chain-submit floor" at M=1 decode (NPU ~35Γ— slower than CPU) and concluded MoE-on-NPU decode can't win. That contradicted M0 (NPU expert 3–10Γ— faster) and contradicted itself (a "submit floor" with "repack ~0%" while the same run reported the hot pool thrashing and repacking every rep). It was adversarially verified.

Test 1 β€” isolated resident submit, M=1 (tools/moe_m1_probe.c, warm median Γ—40)

Real Qwen3.6-35B-A3B expert dims are gate/up 2048Γ—512, down 512Γ—2048 (smaller than assumed); both measured, weights packed once and resident:

dims run_i8 chain (S=8) run_stream CPU (NEON) best NPU vs CPU
Qwen gate/up 2048Γ—512 281 Β΅s 165 Β΅s 67 Β΅s 73 Β΅s NPU 1.10Γ—
Qwen down 512Γ—2048 212 Β΅s 172 Β΅s 54 Β΅s 76 Β΅s NPU 1.41Γ—
LFM gate/up 2048Γ—1792 554 Β΅s 404 Β΅s 152 Β΅s 263 Β΅s NPU 1.73Γ—
LFM down 1792Γ—2048 555 Β΅s rc=-3 rc=-3 263 Β΅s CPU 0.47Γ—

The resident submit is 54–281 Β΅s β€” 80–400Γ— faster than the claimed 21 ms, and run_stream (cross-core) beats CPU even at M=1 decode. (LFM down-proj K=1792 is not %512 β†’ chain/stream reject it rc=-3 β†’ run_i8-only, the one shape CPU wins; the real Qwen down-proj K=512 is chainable, so this doesn't bite the target.)

Test 2 β€” pack-vs-submit split inside the partition (instrumented, M=1 decode, real model)

The existing moe_pack/moe_deq profile accumulators were dead code (never written in the hot handler) β€” which is why a prior read concluded "repack ~0%". With live-pack timing wired in:

hot_K EFAULT resident hit-rate ms / "chain-submit" fallback calls
8 0 0.94 GiB 8.9% 13.3 ms 0
16 0 1.88 GiB 29.9% 26.2 ms 0
24–128 31k–43k 1.98 GiB ~32% ~28 ms ~0

Pack is 3.7–4.7 ms/pack (separately timed); the per-task fallback fires 0 times (the chain path succeeds). So the 13–28 ms is not pack and not fallback. It rises with resident footprint/hit-rate independent of EFAULT (the EFAULT-free hot_K=16 is still 26 ms), so the IOVA overcommit at hot_Kβ‰₯24 roughly doubles it but is not the root cause.

What the 21 ms actually is

A free+repack-each-rep microbench isolates the churn the partition suffers (8-slot pool over 128 experts β†’ ~75–90% cold, constant LRU evict + live re-pack): repack alone = 3.1 ms (Qwen) / 11 ms (LFM), and run_i8 itself only moves 281 β†’ 418 Β΅s under fresh IOVA (cold IOVA is cheap). The inflation to 13–28 ms is the eviction churn (MEM_CREATE/MEM_DESTROY ioctls at ~25–30 % hit-rate) serializing on the single-stream NPU against the chain submit β€” residency thrash, not a per-submit hardware cost.

Verdict

βœ… MoE-on-NPU decode is back in play. The resident per-expert GEMM (54–67 Β΅s via run_stream) beats CPU even at M=1 β€” the long "dead end" history was always repack churn + residency thrash + single-core chain, never the GEMM. The committed cold-path vec_dot reuse stands regardless.

Path (micro-validated; end-to-end re-bench is the open M2 step):

  1. Persist experts to .orkpack (M2) β†’ kills the 3–5 ms live-pack.
  2. Switch the partition run_chain_i8 β†’ run_stream_i8 β†’ 2.5Γ— even resident (165 β†’ 67 Β΅s).
  3. Size the no-EFAULT cache to the IOVA budget (hot_K=16 already holds 1.88 GiB EFAULT-free) β†’ kills the thrash; the cap is per-tensor count, not bytes.
  4. Re-benchmark decode only after experts are resident β€” today's 0.15–0.70 t/s is thrash, not the mechanism, so it is not a valid measurement.

β˜… Update 2026-06-27 β€” M2 end-to-end decode: the micro-win does NOT survive M=1 (it loses), and why

Built the full M2 path (persist .orkpack 7.77 GB int8 + run_stream hot experts + conformance gate routing the K=1792 ffn_down to CPU + cold-CPU vec_dot) and benched LFM2.5-8B-A1B decode vs the CPU reference, -t 4, governors at max:

metric M2 (best: HOT=4) CPU (ORK_OFF=1)
prefill pp64 ~10.6 t/s 36.2
decode tg64 6.56 19.09 β†’ M2 ~2.9Γ— slower
peak resident 0.602 GiB β€”
hit-rate 16.8 % β€”
EFAULT 0 β€”

Verdict: M2 LOSES at M=1 decode (~2.9Γ—). NOT a default β€” kept behind ORK_MOE_NPU (off by default), NOT recommended on RK3588-class hardware. The resident GEMM is fine; three RK3588-specific walls sink the end-to-end case:

  1. Usable IOVA is only ~1.2 GiB, not ~3.5. The GGUF backbone weights already consume ~2.8 GiB of the 4 GiB cap, so the hot cache maxes out ~HOT=4 (16.8 % hit); HOT=16 EFAULTs. (Corrects this page's earlier "hot_K=16 holds 1.88 GiB" β€” that ignored the resident backbone.)
  2. The cold-expert path is memory-BANDWIDTH-bound, not thread-bound. Each cold expert re-reads its full ~3.67 MB weight from DRAM per token; a row-parallel rewrite was neutral (COLD_THREADS=1 770 Β΅s vs threaded 688 Β΅s). A per-expert GEMV structurally loses to ggml's fused, cache-blocked batched MUL_MAT_ID. With ~83 % of experts cold, this dominates.
  3. ork_mm_load_i8 is 5.9 ms/load (rebuilds the full-K Bf: un-tile→re-tile + per-tile bsync + a 2nd DMA buffer) — a frozen-submodule cost paid per first-touch. (Persist was working — the earlier "176 live-packs" was a mislabeled counter; they're disk loads, 0 packs.)

Structural floors no knob fixes: ffn_down (K=1792, not chain-conforming) is always CPU-cold (~88 cold GEMVs/token); 4-of-32 experts/token thrashes a small resident cap.

The lever is M>1, not residency. At M=1 every cost is per-token and un-amortized and the cold majority is memory-bound. At M>1 (batched verify / prefill) the 5.9 ms load + the NPU submit amortize across tokens, and the cold GEMV becomes a real M-row GEMM (compute-bound β†’ competitive). Closing M=1 specifically would need frozen-layer work (a cheaper ork_mm_load_i8 that skips the Bf rebuild for decode shapes; NPU-enabling K=1792; or routing cold experts to ggml's native batched kernel).

Disposition (per the "keep ruled-out approaches runnable behind env flags" rule): M2 is preserved as an experimental, default-OFF setting (ORK_MOE_NPU), wired into oRKLLM with an RK3588 "not recommended" disclaimer. Revisit when: a wider-IOVA device (no 32-bit IOMMU), DDR5 / higher bandwidth, or the M>1 batched-verify path.

Honesty / caveats

  • CPU reference: my NEON probe 73–76 Β΅s; the integration's threaded ggml vec_dot 194–214 Β΅s; an earlier note quoted 590 Β΅s. The verdict (resident NPU 54–67 Β΅s) holds against all three.
  • The probe was built against submodule 15cb782 (v0.3.4+15); the integration pins dd6e7e6 (v0.3.5) β€” adjacent versions, identical chain/stream/envelope semantics.
  • Verification instrumentation in the board ggml-ork.cpp was left uncommitted (measurement only; backup at /tmp/ggml-ork.cpp.bak); orkllm never stopped; board left as found.