Experiment — Pin NPU-driver threads to big CPU cores (big.LITTLE affinity)

Date: 2026-06-13 · Status: ✅ KEPT (correct fix; end-to-end neutral on int8, +57% int4 prefill) · Path: ork-driver

Hypothesis

Tier 1a of the roadmap: prefill multi-core scaling was anomalously bad (3 cores → only 1.06× over 1 core, while decode scaled 1.76×). Hypothesis: the pool worker threads aren't CPU-affinity-pinned, so under sustained load the scheduler parks them on the A55 little cores (1.8 GHz), ~2× slower than the calling thread that landed on an A76 big core (2.4 GHz). Since run_multicore waits for the slowest worker, multi-core collapses.

Diagnosis (Tier 1a)

Instrumented mcworker's prefill path to split per-core copy / submit / accumulate time (ork_npu_mc_timing, tool mc_prof). At 200 warm iters (steady state), Qwen3-1.7B 512×2048×2048:

core	copy µs/submit	submit µs/submit	acc µs/submit
0 (calling thread)	227	384	246
1 (pool worker)	511	570	708
2 (pool worker)	517	590	680

Cores 1 & 2 were ~2× slower on every phase — including pure CPU work — confirming they ran on little cores. grep-confirmed: no setaffinity/CPU_SET anywhere in the driver.

Implementation

• Commit: 34b539e (oRKLLM/ork-driver) — pin_big_core(id) maps NPU-driver thread id → CPU (ncpu-1-id) (the big cluster is high-numbered on RK3576/RK3588), applied at every CPU-touching spawn point: int8 pool workers + calling thread, int4 workers (i4_mcworker/i4_mcworker_g) + their calling thread, and the single-core path. ORK_NO_AFFINITY=1 disables.
• Tools: tools/mc_prof.c + ork_npu_mc_timing() / ork_npu_mc_reset().

Benchmark (standardized — 2026-06-13)

Microbenchmark (mc_prof, 512×2048×2048 int8, 200 warm iters): 3-core prefill 7670 → 4439 µs/matmul, scaling 1.11× → 1.92×; per-core phases now balanced (~222 / 497 / 277 µs on all three).

End-to-end (llama-bench, Qwen3-1.7B-w8a8, -r 2, vs pre-pinning):

Arm	pre-pinning (pp512 / tg128)	with pinning
CPU (ORK_OFF=1)	35.5 / 9.43	35.3 / 9.44
hybrid (ORK_MINM=32)	135.3 / 9.52	136.5 / 9.50
all-NPU (ORK_MINM=1)	131.4 / 5.03	134.4 / 4.2*
int4 (ORK_QUANT=4)	3.66 / 2.53	5.73 / 2.73

* all-NPU int8 decode is noisy run-to-run (~4.2–5.0); within variance.

Verdict

✅ Correct and necessary fix, kept — the workers genuinely were on the little cores and are now balanced (proven at the matmul level: standalone 3-core prefill 7665→4365 µs/matmul, decode 2048² 280→176). int4 prefill improved 1.57× (3.66 → 5.73) directly. int8 end-to-end at the default -t 8 looked unchanged (135→136) — but that's because llama.cpp's 8-thread CPU pool spins on all cores and starves these now-pinned NPU-driver threads. At -t 4 (big cluster only) the affinity pays off end-to-end: prefill 155, decode 12.84 (beats librkllmrt) — see Thread count & big cores. (An earlier draft of this page blamed a budget-3/Bf pack regime for the flat end-to-end; that was wrong — the apparent non-improvement was a stale /tmp/rknn_vs_ork binary plus the CPU-thread contention, not Bf.) Pinning is the right foundation; the integration also needs the right thread count / heterogeneous scheduling.