Date: 2026-06-13 · Status: 🎯 PINS THE OPEN-vs-CLOSED GAP · Path: ork-driver vs librknnrt (RKNN matmul API)

The full-model comparison (ggml-ork vs librkllmrt) is apples-to-apples at the model level, but librkllmrt is a whole LLM runtime with no matmul API — so for kernel-level questions (submit cost, tiling) it's one fixed point, not a controllable comparison. The closed librknnrt (RKNN) does expose a matmul API (rknn_matmul_*). So we can run the same int8 (M,K,N) matmul through the open path (ork_mm_run_i8) and the closed path (rknn_matmul_run) — both submitting to the same NPU via the same DRM driver — and measure per-matmul cost directly.

• Tool: tools/rknn_vs_ork.c (ork-driver) — packs B once, then times N warm iterations of the same int8→int32 matmul on each runtime.
• Build: make rknn_vs_ork (needs librknnrt.so + rknn_matmul_api.h present, e.g. /tmp/rknn).
• Run: sudo env LD_LIBRARY_PATH=/tmp/rknn ./rknn_vs_ork [iters]

// closed path — RKNN matmul API (librknnrt), int8 x int8 -> int32
rknn_matmul_info info = { .M=M, .K=K, .N=N, .type=RKNN_INT8_MM_INT8_TO_INT32 };
rknn_matmul_io_attr io; rknn_matmul_create(&ctx, &info, &io);
rknn_tensor_mem *Am=rknn_create_mem(ctx,io.A.size), *Bm=rknn_create_mem(ctx,io.B.size), *Cm=rknn_create_mem(ctx,io.C.size);
rknn_matmul_set_io_mem(ctx,Am,&io.A); rknn_matmul_set_io_mem(ctx,Bm,&io.B); rknn_matmul_set_io_mem(ctx,Cm,&io.C);
for (int i=0;i<iters;i++) rknn_matmul_run(ctx);            // timed

// open path — ork-driver (raw regcmd), same shape/dtype
ork_w *w = ork_mm_pack_i8(c, K, N, B);                      // pack once (resident)
for (int i=0;i<iters;i++) ork_mm_run_i8(c, w, M, A, C);    // timed

Note (a finding in itself): the RKNN matmul API rejects the 3-core mask (Not support core mask: 7, fallback to single core auto mode) — it is single-core only. So the clean apples-to-apples pins ork to 1 core too (ork_npu_set_core_budget(c,1)); ork's 3-core result is shown alongside since that's what oRKLLM actually uses.

RK3588, int8→int32, 200 warm iterations, DDR 2112 / CPU 2.4 GHz. µs per matmul (lower = faster). Shapes are Qwen3-1.7B projections (M=1 decode; M=512 prefill). Post-affinity-fix numbers (an earlier draft showed pre-affinity ork-3core figures from a stale binary — now corrected).

1. Decode, same single core: ork ≈ RKNN (1.0–1.3×, ~1.1× typical; the 2048² case is 1.02× once the calling thread is pinned to a big core). ork's per-matmul decode kernel is essentially at parity with the closed runtime. Framework overhead is only 3% (see Submit-floor profiling).
2. ork 3-core beats RKNN 1-core on decode (176 vs 432 µs at 2048², 2.45×) — the RKNN matmul API can't use 3 cores (rejects core mask 7).
3. Prefill (M=512): ork's single-core large-M kernel is 3.3× slower than RKNN's (8427 vs 2573 µs). Multi-core (3 cores, post-affinity) narrows it to 1.7× (4365 vs 2573) but can't fully close it — RKNN's single core still beats ork's three. ork's large-M tiling / MAC utilisation is the remaining prefill lever (the end-to-end integration gap, by contrast, was CPU-thread contention — see Thread count & big cores).

The closed rknn_matmul_api.h (on-board /tmp/rknn) documents its own native layouts for RK3588/3576 int8 — the definitive prior art for the "native layout" lever:

This corrects the original 1c-i premise: ork already uses RKNN's native B layout, so there is no 2.5× native-B win to chase. External prior art (clehaxze.tw, RK3588 NPU GEMM) measured native vs normal A layout at ~900 vs ~750 GFLOPS fp16 (~20%) — so native A looked like a lever.

Tested it directly (risk-free, on RKNN itself) → ruled out. rknn_vs_ork <iters> a runs RKNN's matmul with AC_layout=0 (normal A) vs =1 (native A) at the prefill shape (M=512, 2048² int8):

So native A is slower, not faster, for int8 prefill on this HW — the fp16 blog result doesn't transfer (different dtype; its win is avoiding the runtime's A-rearrange, which amortizes away for a large-M batch). Reordering ork's A to native would not help — 1c-i native-A is dead, and we confirmed it without the risky ork regcmd-RE. (RKNN also exposes rknn_B_normal_layout_to_native_layout() and splits K into 8192-segments for K>8192.)

Re-reading our own gap with this: ork's full-K single-core M=512 = 4763 µs; of that the NPU submit phase is ~2904 µs ≈ RKNN's whole-matmul 2573 µs (≈1.13×, not 2.5×). The rest is host-side activation copy + result writeout (~1.4 ms) — and part of the apparent gap is a benchmark artifact: rknn_vs_ork sets RKNN's A once and reuses it across the timed loop, while ork re-copies A every call. A real model pays the A-copy on both sides.

🎯 This is the apples-to-apples that the full-model tables can't give. It splits the open-vs-closed gap cleanly: decode ≈ 1.24× (per-matmul kernel efficiency) and prefill ≈ 3× (large-M tiling). Both are ork-driver kernel problems, not ggml-integration or framework-overhead problems — pointing optimization straight at the regcmd matmul kernel (per-submit cost for decode; M-tiling for prefill). See Open vs Closed for the model-level view and Benchmark Standards for the method.