Experiment — Perf/watt & thermal proxy

Date: 2026-06-13 · Status: ⚠️ INCONCLUSIVE (no wattmeter) · Path: board (RK3588)

Hypothesis

oRKLLM's thesis for low-power SBC serving is energy-efficiency. Hypothesis: even where the 8-core CPU is faster than the NPU on small-model decode, the NPU wins on perf/watt (3 low-power cores, frees the CPU).

Method / finding

• No real wattmeter on the board. The USB-C PD sensor (hwmon7 tcpm-source-psy) reports the negotiated 45 W contract (curr1=2250 mA=current_max, in0=20 V), not actual draw — it's static under load. No shunt / INA226. Watts cannot be read in software — needs an external USB-C power meter.

# what's exposed — PD CONTRACT, not draw (static under load):
grep -H . /sys/class/hwmon/hwmon7/{curr1_input,in0_input,curr1_max} 2>/dev/null
#   curr1_input: 2250 (mA) == curr1_max  -> the negotiated limit, not measured current
#   in0_input:   20000 (mV)
# thermal proxy instead (per-zone temps, milli-°C):
for z in /sys/class/thermal/thermal_zone*; do printf '%s %s\n' "$(cat $z/type)" "$(cat $z/temp)"; done

• Temperature proxy (idle 49 °C; sustained 0.5B decode, n=160):
  • NPU int8: 9.07 tok/s, SoC 58 °C / bigcore 60 °C / npu 56 °C, 17.5 s
  • CPU 8-thread: 26.45 tok/s, SoC 65 °C / bigcore 69 °C / npu 61 °C, 6.0 s

Benchmark (standardized — 2026-06-13)

Throughput to standard (Qwen3-1.7B-w8a8, llama-bench -r 2, DDR 2112 / CPU 2.4 GHz) — needed because perf/watt = throughput ÷ power:

Config	pp512	tg128
hybrid (NPU prefill + CPU decode)	135.28 ± 0.97	9.52 ± 0.04
pure CPU	35.49 ± 0.23	9.43 ± 0.18
librkllmrt (closed, all-NPU)	183.9	11.71

Decode tok/s is ~equal (hybrid routes decode to CPU); prefill is 3.8× on the NPU. Power is the missing axis — see below. The thermal proxy numbers are from the earlier 0.5B sustained-load test; a standardized perf/watt needs an external meter on the reference model.

Verdict

⚠️ The NPU runs ~7–9 °C cooler ⇒ lower instantaneous power (✓ the low-power intuition). But energy/token likely favours the CPU for small models: energy = power × time, and the CPU is 2.9× faster (finishes in ~1/3 the wall time), so its speed beats the NPU's lower power. The NPU's perf/watt edge should appear on larger models (CPU becomes bandwidth-bound → throughput collapses while power stays high; NPU power + throughput stay flat). A definitive perf/watt verdict needs an external meter + a 1.5B+ model. Honest summary: lower-power = NPU; perf/watt for small models = likely CPU.