Experiment — Hybrid M-Gate: decode on CPU, prefill on NPU

Date: 2026-06-13 · Status: ✅ KEPT (default behaviour) · Path: ggml-ork (llama.cpp integration)

Hypothesis

The ggml-ork backend offloads every MUL_MAT to the NPU. If the ~365 µs per-matmul NPU submit floor dominates token-by-token decode (M=1), then routing small-M matmuls to the CPU and keeping only large-M (prefill) matmuls on the NPU should be strictly better than all-NPU.

What was wrong

ggml_backend_ork_device_supports_op gated on (M >= min_batch || N >= min_batch). Every weight has a large output dim N, so the N >= min clause was always true — dragging M=1 decode matmuls onto the NPU, where they are a net loss.

Implementation

• Commit: 4956b054 (oRKLLM/llama.cpp-rockchip, ork-backend)
• Symbol: ggml-ork.cpp : ggml_backend_ork_device_supports_op
• Change: gate on M only — M >= min_m && K >= 32 (dropped the || N>=min clause).
• Knob: ORK_MINM (default 32). ORK_MINM=1 reproduces the old all-NPU behaviour.

case GGML_OP_MUL_MAT: {
    const int64_t K = src0->ne[0], N = op->ne[0], M = op->ne[1];
    // Measured (RK3588, Qwen3-1.7B-w8a8): the ~365us/matmul NPU submit floor makes per-token
    // DECODE (M=1) a net LOSS vs CPU (4.7 vs 9.4 tok/s) — ~197 submits/token at 365us each is
    // ~72ms before any compute benefit, and M=1 matmuls are tiny. PREFILL (large M) is the
    // opposite: M>1 amortizes the floor over many rows, so NPU wins (39.6 vs 13.6 tok/s).
    // Gate on M (the token/batch dim) ONLY — NOT N. The old `M>=min || N>=min` always passed
    // because every weight has a large N, dragging M=1 decode onto the NPU. ORK_MINM tunes it.
    static const int min_m = getenv("ORK_MINM") ? atoi(getenv("ORK_MINM")) : 32;
    return ggml_is_contiguous(src0) && ggml_is_contiguous(src1) &&
           src1->type == GGML_TYPE_F32 &&
           K % 32 == 0 && N % 64 == 0 &&           // K%32; N%64 satisfies both int8 (%32) and int4 (%64)
           M >= min_m && K >= 32 &&
           (src0->type == GGML_TYPE_F32 || ggml_get_type_traits(src0->type)->to_float != NULL);
}

Method

llama-bench (proper warmup + repetition — the only trustworthy harness for ggml-ork; short llama-cli prompts and the lazy weight-pack both produce artifacts). Real model Qwen3-1.7B-w8a8 (Q8_0 GGUF), RK3588, governors verified (DDR 2112 MHz / A76 2.4 GHz).

llama-bench -m qwen3_1.7b.gguf -p 512 -n 128 -t 8 -r 2

Results

Standardized run, llama-bench -p 512 -n 128 -r 2, DDR 2112 / CPU 2.4 GHz, 2026-06-13 (all-NPU decode isolated with -p 0 -n 128):

Config	pp512 (prefill)	tg128 (decode)
Hybrid (ORK_MINM=32)	135.28 ± 0.97	9.52 ± 0.04
Pure CPU (ORK_OFF=1)	35.49 ± 0.23	9.43 ± 0.18
All-NPU (ORK_MINM=1)	131.42 ± 2.29	5.03 ± 0.15
librkllmrt (closed, same model, rknpu_bench)	183.9	11.71

Verdict

✅ The hybrid keeps the 3.7× prefill win (130.9 vs 35.4 CPU) and recovers decode to the CPU baseline (9.46 ≈ 9.52), eliminating all-NPU's 2× decode penalty. Per-matmul NPU offload cannot beat the CPU on M=1 decode — that needs a batched/persistent-program submit model like librkllmrt's. Decode is now CPU-bound; prefill is the genuine NPU win.

See also: Run-Multicore Phase Profiling (the diagnostic that pointed here), Open vs Closed apples-to-apples.