Date: 2026-06-13 · Status: 📊 MEASUREMENT (corrects earlier over-optimism) · Path: ggml-ork vs librkllmrt

Earlier notes claimed "ggml-ork prefill 280 tok/s beats librkllmrt's 181, decode matches." That was apples-to-oranges: the 280 was a smaller 0.5B model, and the "decode match" was the standalone examples/bench.c kernel loop — not the llama.cpp integration. This experiment forces a fair comparison: same model, same path, warm.

• Model: Qwen3-1.7B-w8a8 — Q8_0 GGUF for ggml-ork, matching .rkllm (v1.2.3) for librkllmrt.
• ggml-ork: llama.cpp integration, warmed up. librkllmrt: tools/rknpu_bench (dlopen's the runtime; packs at model load, outside the timed window).
• RK3588, governors verified (DDR 2112 / A76 2.4 GHz).

# governors must be pinned BEFORE timing (else DDR parks low and ~halves decode)
cat /sys/class/devfreq/dmc/governor                       # -> performance
cat /sys/devices/system/cpu/cpu4/cpufreq/scaling_governor  # -> performance

# ggml-ork integration (warm; ORK_OFF=1 for CPU baseline, ORK_PROFILE=1 for the breakdown)
sudo build/bin/llama-cli -m /tmp/gguf/qwen3_1.7b.gguf -p '<prompt>' -n 64 -t 8 -no-cnv

# closed librkllmrt (tools/rknpu_bench dlopen's the runtime; packs at model load)
sudo env LD_LIBRARY_PATH=/tmp/rknn /tmp/rknn/rknpu_bench <librkllmrt-v1.2.3.so> <model.rkllm> 64

Standardized (llama-bench -r 2, DDR 2112 / CPU 2.4 GHz, 2026-06-13):

At -t 4 with full-K prefill, the open stack is at parity on prefill (178 = 97% of 184) and BEATS librkllmrt on decode (12.8 vs 11.71). Two changes got it there from the original 136/9.5: big-core thread count (-t 4) and full-K prefill. The original llama-cli figures (prefill 46.8 / decode 4.49, "3.9× / 2.6×") were polluted by short prompts + the lazy weight-pack; ignore them.

Profiling (ORK_PROFILE=1): quant 2% + dequant 1% = only 3% of decode. The remaining 97% is the per-node ork_mm_run (NPU submit). The per-matmul apples-to-apples vs the closed RKNN matmul API (full writeup) splits it precisely:

So per-matmul, decode is at parity and the only real kernel gap is ork's single-core large-M throughput (prefill). The end-to-end decode win at -t 4 comes from running on the big cluster (the closed runtime's decode is bounded by the same NPU submit floor). Remaining work: ork's large-M tiling for prefill, and heterogeneous CPU-thread scheduling so -t 4 isn't a manual flag.

📊 On the same model and same path, the closed runtime is meaningfully faster. The open stack's value is prefill (NPU genuinely wins vs CPU) and power/simplicity, not raw decode speed. This result is why the project adopted the rule: always run the standardized same-model warm benchmark before any perf claim.