Benchmark Standards & Methodology

Every performance number in this wiki — and every claim in code review, commit messages, or chat — must come from the standardized benchmark below. No number from memory, extrapolation, a different model, or a different code path. This page is the canonical procedure; each Experiment Log entry carries a benchmark run to this standard.

Why this exists

The rule was written after repeated over-optimistic claims that didn't survive a fair test. Two failure modes recur:

1. Mixed configs (apples-to-oranges). A claimed "ggml-ork prefill 280 beats librkllmrt's 181" was actually a smaller 0.5B model on ggml-ork compared against a 1.7B model on librkllmrt — and the "decode match" was the standalone kernel loop, not the integration. Same-model, same-path measurement showed the closed runtime was 2.6× / 3.9× faster. See Open vs Closed.
2. Measurement artifacts. Short prompts (10 tokens → avgM≈6) are nothing like real prefill (M=512). And the lazy one-time weight-pack lands inside the timed window unless warmup exercises the same M as the measured op. Both produced false prefill numbers until llama-bench (real warmup + repetition) was used. See Hybrid M-Gate.

The standard

1. Always compare against the closed runtime — rkllm or rknn, whichever is applicable. This is mandatory, not optional: an open-stack number alone is meaningless without the closed reference on the same work. Pick the one that matches what you're measuring:
   • Full-model throughput (tok/s, an end-to-end experiment) → librkllmrt on the matching .rkllm via rknpu_bench. Same family + size + quant.
   • A single matmul / kernel-level cost (submit latency, tiling, a per-op question) → the RKNN matmul API (librknnrt, rknn_matmul_*) on the same (M,K,N) + dtype, via tools/rknn_vs_ork. rkllm has no matmul API, so this is the only true kernel-level apples-to-apples.
   • If neither closed path can run the exact thing (e.g. a probe the closed runtime doesn't expose), say so explicitly and give the closest closed reference + why it's not exact.
2. Same model / same shape on both sides. ggml-ork: a GGUF (Q8_0). librkllmrt: the matching .rkllm. RKNN: identical (M,K,N) + dtype to the ork matmul. Same family + size + quant. Never compare across model sizes or paths.
3. Use llama-bench, NOT short llama-cli prompts — it warms with a real pp512 batch and repeats. Short llama-cli prompts lie on prefill (unrepresentative M + lazy pack in the timed window). For matmul-level work use rknn_vs_ork (warm, N iterations).
4. Verify governors at max BEFORE timing. Decode is memory-bandwidth-bound; a parked DDR governor ~halves it.
5. Report BOTH prefill and decode (or, for matmul, the relevant M=1 and M≫1 shapes), for every arm, with ≥2 repetitions (mean ± stddev).
6. Match the core count when comparing kernels. The RKNN matmul API is single-core (rejects 3-core mask) — pin ork to 1 core for the clean comparison and show multi-core separately.
7. State the model, the path (integration vs standalone), and warm vs cold — never conflate.

Reference setup

• Board: RK3588 @ 10.3.0.236. Model: Qwen3-1.7B-w8a8 (/tmp/gguf/qwen3_1.7b.gguf Q8_0 for ggml-ork; matching .rkllm v1.2.3 for librkllmrt at /var/lib/orkllm/models/...w8a8...).
• Closed-runtime references (mandatory comparison):
  • Full-model: librkllmrt v1.2.3 — prefill 183.9 / decode 11.71 tok/s (rknpu_bench, same .rkllm, 2026-06-13).
  • Per-matmul: RKNN matmul API (librknnrt, headers + lib in /tmp/rknn) — see Matmul apples-to-apples.

Verify governors (must print performance + max freq)

cat /sys/class/devfreq/dmc/governor                        # -> performance
cat /sys/class/devfreq/dmc/cur_freq                        # -> 2112000000
cat /sys/devices/system/cpu/cpu4/cpufreq/scaling_governor   # -> performance
cat /sys/devices/system/cpu/cpu4/cpufreq/scaling_cur_freq   # -> 2400000

If not pinned (oRKLLM's perf_governor does this automatically while a model is loaded):

echo performance | sudo tee /sys/class/devfreq/dmc/governor
for c in 4 5 6 7; do echo performance | sudo tee /sys/devices/system/cpu/cpu$c/cpufreq/scaling_governor; done

Run (ggml-ork)

sudo build/bin/llama-bench -m /tmp/gguf/qwen3_1.7b.gguf -p 512 -n 128 -t 8 -r 2

Run (librkllmrt — closed full-model baseline)

sudo /tmp/rknn/rknpu_bench /var/lib/orkllm/runtimes/librkllmrt-aarch64-v1.2.3.so \
  /var/lib/orkllm/models/Qwen3-1.7B-Base-rk3588-w8a8-normal-v1.2.3-RKLLM/Qwen3-1.7B-Base-rk3588-w8a8-normal-v1.2.3-RKLLM.rkllm 128

Run (RKNN matmul API — closed per-matmul baseline)

# ork-driver tools/rknn_vs_ork: same int8 (M,K,N) via ork_mm_run_i8 vs rknn_matmul_run
make rknn_vs_ork RKNN_DIR=/tmp/rknn
sudo env LD_LIBRARY_PATH=/tmp/rknn ./rknn_vs_ork 200

Env knobs for A/B experiments

The ggml-ork backend exposes knobs so each experiment's "on" and "off" arms can be measured on the same reference model. Hold everything else fixed; change one knob.

Knob	Effect	Used by
ORK_OFF=1	Force every matmul to CPU (pure-CPU baseline)	all (baseline)
ORK_MINM=<n>	Offload gate: matmuls with M≥n go to NPU (default 32). 1 = all-NPU; 99999 ≈ off	Hybrid M-Gate
ORK_QUANT=4	int4: compact int4 STORAGE + W8A8-inflate compute (route B — coherent, no wedge; native W4A4 is opt-in via ORK_MIXED_W4A4/ORK_HADAMARD). RECOMMENDED: build the .orkpack from an F16/F32/BF16 source so the NF4 codebook auto-selects	BR-7 (int4 P=128: NF4-from-F16 215/6.77 vs uniform-from-Q8 172/2.96), int4 W4A4
ORK_FUSE=1	Fuse adjacent same-input matmuls (QKV/gate-up)	Fusion
ORK_NOREUSE=1	Disable activation-quant reuse	Activation reuse
ORK_NOCACHE=1	Re-pack the weight every matmul (no NPU-resident cache)	Weight caching
ORK_PROFILE=1	Print per-section timing (quant / NPU run / dequant; decode vs prefill; run_multicore phase split)	Submit-floor, run_multicore

Note on ORK_NOCACHE: re-packing every matmul is pathologically slow (the pre-cache regime), so that one arm is benchmarked with reduced -p/-n and labelled as such — it demonstrates the order-of-magnitude effect, not a serving rate.

A/B matrix template

M=/tmp/gguf/qwen3_1.7b.gguf
run() { echo "### $1"; shift; sudo env "$@" build/bin/llama-bench -m $M -p 512 -n 128 -t 8 -r 2 | grep -E "pp512|tg128"; }
run CPU_baseline    ORK_OFF=1
run hybrid_default  ORK_MINM=32
run all_NPU         ORK_MINM=1
run all_NPU_fuse    ORK_MINM=1 ORK_FUSE=1
# ... one row per arm; change exactly one knob vs the control

Reporting format

Each experiment page reports a table with the arms compared and the control, always including prefill + decode:

Config	pp512 (prefill)	tg128 (decode)
arm under test	… ± …	… ± …
control	… ± …	… ± …

Profiling-only experiments (no tok/s comparison) report the ORK_PROFILE breakdown on the reference model instead, plus the standardized end-to-end numbers for context.