Date: 2026-06-13 · Status: 🏆 BEATS librkllmrt on decode at -t 4 · Path: ggml-ork integration

CPU-affinity pinning fixed the standalone multi-core (mc_prof/rknn_vs_ork prefill 7665→4365 µs/matmul, scaling 1.92×), but llama-bench pp512 stayed ~136. Why doesn't the integration benefit?

Profiled the integration's prefill matmul (ORK_PROFILE=1, real long prompt, affinity ON):

ork profile: prefill (M>1) 193 matmuls, avgM 512.0, run 7596 us/matmul
ork profile: run_multicore 193 calls | setup 796 (10%) submit 6309 (83%) copy 490 (6%)

7596 µs/matmul — the slow number, not the 4365 the standalone gets with the same code. So the ork worker threads are slow inside the integration specifically. Cause: llama.cpp's CPU threadpool busy-waits (spins) on all 8 cores while the single NPU matmul node runs, starving the 4 ork NPU-driver threads (calling thread + 3 pinned workers) that need the big cores. Plus the 4 little A55 cores in an -t 8 pool drag every CPU-side op (the threadpool waits on its slowest thread).

Qwen3-1.7B-w8a8, llama-bench, warm, DDR 2112 / CPU 2.4 GHz. Sweeping -t (CPU thread count):

1. -t 4 (big cores only) is optimal for both phases: prefill 136→155 (+14%), decode 9.24→12.84 (+39%).
2. Decode at -t 4 (12.84) BEATS librkllmrt (11.71) — the open stack is faster than the closed runtime on token generation when it isn't dragged by the little cores. Prefill reaches 84% of librkllmrt (155 vs 184).
3. The integration's prefill was never NPU-matmul-bound — it was CPU-thread contention + little-core drag. The affinity fix is necessary (workers must be on big cores) but insufficient while llama.cpp also spins 8 threads across all cores.
4. -t 8 is a poor default on big.LITTLE: the 4 slow A55 cores hold back the threadpool barrier and fight the NPU-driver threads for the big cores.

Tried making it automatic: pin the calling/main thread to the big cluster at ork_npu_init (before llama.cpp spawns its threadpool, so the pool inherits the mask — the taskset -c <big> effect without the flag). Measured (no taskset):

It doesn't help -t 4 (already optimal) and craters -t 8 — restricting 8 threads onto 4 cores oversubscribes 2:1 and thrashes (taskset's freely-migratable set behaves differently from our inherited-mask + single-core-worker mix). Reverted (ork-driver b9a5c4e); the driver pins only its own dedicated NPU-driver threads, never the whole process. Thread count is the user's lever.

🏆 Run the open stack with -t <big-core-count> (4 on RK3588), not -t 8. That alone makes decode beat librkllmrt and lifts prefill to 84% — and forcing it from the driver backfires (above), so it stays a serving/CLI choice. The remaining prefill gap is ork's single-core large-M matmul kernel (3.3× slower than RKNN's per core — see Matmul apples-to-apples), now the main lever. The proper automatic fix is heterogeneous scheduling — the CPU threadpool should sleep (not spin) while the NPU matmul runs, and the thread count should match the big cluster — but that belongs in the llama.cpp threadpool / serving layer, not a process-wide affinity hammer. See Optimization Roadmap.