Decode Profiling — where the time actually goes (GGUF Q8_0, Qwen3-1.7B)

Date: 2026-06-23 Status: ✅ Resolved — characterized; no regression. Dense GGUF-Q8_0 decode is bandwidth-bound at ~4 t/s. Board: RK3588 khazad-dum, -t 4, governors pinned (dmc=performance@2112MHz).

Question

Open-stack decode (ggml-ork → ork-driver) measured ~4 t/s on Qwen3-1.7B GGUF Q8_0 via llama-bench -t 4 -p 0 -n 32, vs a remembered ~11–12.8. Regression, or expected?

Method

• Added ORK_PROFILE per-matmul host timing to ork-driver (run_i8/run_i4 wrapped with a monotonic timer, printed in ork_npu_free via fprintf(stderr) — survives llama-bench's GGML_LOG suppression, unlike the ggml-ork-side ORK_PROFILE). Committed to ork-driver main.
• llama-bench for clean tg/pp (no HTTP server, does warmup+reps). ORK_VERBOSE to see invoked paths. A/B by swapping ggml-ork.cpp variants and rebuilding.

Findings

1. ork-driver is only ~12% of decode wall time.

[ork PROFILE] run_i8: 1625 calls, 1874 ms total, 1153 us/call   (~25 calls/token → ~29 ms/token)

Decode is ~251 ms/token; ork-driver matmuls ≈ 29 ms. The other ~88% (222 ms/token) is the ggml/CPU path (norms, rope, softmax, sampling, dequant, and matmuls that fall back to CPU) + weight-read bandwidth (~1.7 GB/token for a Q8_0 1.7B). Of the 1.15 ms/call, NPU compute is ~0.2 ms; ~0.95 ms is host overhead (quant A → copy → submit → dequant C), of which the submit ioctl floor is ~365 µs.

2. No regression — the chaining-debug work slightly improved decode. A/B on the same GGUF, governors pinned:

ggml-ork variant	decode
full .orig (pre-chaining-debug)	3.06 t/s
as-found (if(target_qbits==8) threshold=1 → i8 M=1 decode on NPU)	3.99 t/s
M=1 dense forced to CPU	3.06 t/s

Routing barely moves it (3.06↔3.99) because decode is bandwidth-bound regardless of where the dense matmuls run.

3. The ~11–12.8 was the w8a8 (.rkllm) path, not GGUF Q8_0 — a different quantization/runtime. Comparing them violates the apples-to-apples rule (see Benchmark Standards). To hit ~11–12.8 on a 1.7B, use the w8a8 path.

4. Chain-batching does NOT fire for decode (confirmed). With a chain-length probe in graph_compute, every chainable decode matmul (attn_q/k/v, ffn_down) is detected (type=ORK_CHAIN_I8) but always at len=1. The detector only groups consecutive same-type nodes, and decode's independent matmuls are not consecutive: Qwen3's QK-norm (RMS-norm) + rope/reshape sit between q/k/v, and ffn_gate/up (N=6144) are excluded by the N ≤ 4096 chain cap — both break adjacency.

Upside of fixing chain-batching (for dense decode): negligible

Chaining only amortizes the submit ioctl (~365 µs), not quant/dequant. The submit floor is ~9 ms of the 251 ms/token, so even perfect decode chaining buys <~4% (≈ 4.0 → 4.1 t/s). Not worth the complexity for dense GGUF-Q8_0 decode.

Where the same change IS high-value: MoE (many independent same-input expert matmuls/token; CPU expert-weight streaming is a documented ~32 ms catastrophe) and prefill. The fix would be: a look-ahead pass collecting independent same-input chainable matmuls across intervening non-matmul nodes (the norms/rope depend on each matmul's output, not on the sibling matmuls), plus raising the N≤4096 chain cap toward nmax=8192. The run_chain_i8/i4 primitives (Exp 2026-06-23 Chaining Resolution) are ready for it.

Takeaways / methodology notes

• Always pin governors before timing (parked DDR ~halves decode); ork-driver now warns at init if dmc≠performance, and tools/bench_two_turn.sh pins+verifies+aborts.
• Profile before optimizing: the entire decode "regression" hunt would have chased ork-driver (12%) instead of the real 88% without the per-matmul timing.
• Don't compare across quant paths (GGUF Q8_0 ≠ w8a8).
• Env knobs: ORK_PROFILE=1 (per-matmul timing), ORK_VERBOSE=1 (invoked paths), ORK_MINM (M-threshold for NPU offload), ORK_NPU_MC=n (force multicore — neutral for decode).