Experiment — Submit-latency floor profiling (root cause)

Date: 2026-06-12 · Status: 🎯 ROOT CAUSE established · Path: ggml-ork

Hypothesis

The ORK decode rate is a flat ~10 tok/s for both 135M and 0.5B models — suspiciously size-independent. Hypothesis: decode is overhead-bound, not compute-bound — dominated by a fixed per-matmul cost (NPU submit latency × ~200 matmuls/token), not the arithmetic.

Implementation

• Commit: 60a83221 (oRKLLM/llama.cpp-rockchip) — ORK_PROFILE=1 per-matmul breakdown (split decode M=1 vs prefill M>1).
• Symbol: counters on ggml_backend_ork_context, stamped around each phase of the int8 matmul.

// ORK_PROFILE=1: accumulate where time goes, report on free (split decode M=1 vs prefill M>1)
double t_quant = 0, t_run = 0, t_deq = 0; long n_mm = 0; int profile = 0;
double t_run_dec = 0, t_run_pf = 0; long n_dec = 0, n_pf = 0, m_pf = 0;

const double t0 = ctx->profile ? ork_now_us() : 0;
/* ... per-row activation quant ... */
const double t1 = ctx->profile ? ork_now_us() : 0;
if (ork_mm_run_i8(ctx->npu, ow.w, M, ai, ci)) return false;     // the NPU submit
const double t2 = ctx->profile ? ork_now_us() : 0;
/* ... int32 -> fp32 dequant ... */
if (ctx->profile) { double t3 = ork_now_us();
    ctx->t_quant += t1-t0; ctx->t_run += t2-t1; ctx->t_deq += t3-t2; ctx->n_mm++;
    if (M > 1) { ctx->t_run_pf += t2-t1; ctx->n_pf++; ctx->m_pf += M; }
    else       { ctx->t_run_dec += t2-t1; ctx->n_dec++; } }   // <-- split M=1 from M>1

Results (Qwen2.5-0.5B-Q8_0)

• NPU run 97% / quant 2% / dequant 1% · ~205 µs/matmul.
• A small 0.5B projection (K=896) costs the same ~200 µs as a giant standalone 2048² matmul — because the RKNPU_SUBMIT round-trip (ioctl → job-queue → NPU → IRQ → sync) dominates, not the math.
• ~168 matmuls/token (7 proj × 24 layers) → ~34 ms submit floor/token. Governors verified at performance (not throttling/DDR).

Benchmark (standardized — 2026-06-13, reference model)

Re-confirmed on Qwen3-1.7B-w8a8, all-NPU (ORK_MINM=1 ORK_PROFILE=1), warm, DDR 2112 / CPU 2.4 GHz:

ork profile: 9653 matmuls | quant 48ms (1%) run 3658ms (98%) dequant 21ms (1%) | 386.1 us/matmul
ork profile: decode  (M=1)  9267 matmuls, run 359.0 us/matmul
ork profile: prefill (M>1) 386 matmuls, avgM 6.0, run 859.8 us/matmul (143.30 us/row)

NPU run is 98% of decode at ~386 µs/matmul on the reference model too — the floor is model-independent, exactly as predicted. End-to-end (standard llama-bench): all-NPU decode 5.03 tok/s, hybrid 9.52, CPU 9.43.

Closed-runtime baseline (per-matmul, RKNN matmul API — the applicable apples-to-apples): the same isolated int8 matmul through librknnrt vs ork_mm_run_i8, single core (RKNN matmul is single-core), 200 warm iters — see Matmul apples-to-apples:

matmul (M=1 decode)	ork 1-core	rknn 1-core	ork/rknn
2048×2048	495.9 µs	398.8 µs	1.24×
6144×2048	1515.0 µs	1178.7 µs	1.29×

The closed RKNN kernel hits the same submit floor (~399 µs vs our 496 µs for a 2048² M=1 matmul) — confirming the floor is hardware, runtime-independent. It's just ~1.24× tighter per matmul, which is exactly the ~1.2× end-to-end decode gap. rkllm full-model cross-check: its decode 11.71 tok/s ≈ 86 ms/token ÷ ~197 matmuls ≈ ~440 µs effective/matmul — same order.

Verdict

🎯 Confirmed: decode is submit-latency-floor-bound. The CPU has no per-op floor (small int8 GEMM in tens of µs across 8 NEON cores), so a tuned RK3588 CPU beats the NPU on small-model decode (26 tok/s on 0.5B). Standalone benches looked great only because they amortized the floor over one giant matmul in a tight loop — the opposite of a transformer's many small ops. This is the central finding that frames every later decode experiment (fusion, task-batching, activation reuse, hybrid M-gate).