Experiment — int8 W8A8 path (the practical default)

Date: 2026-06-12 · Status: ✅ KEPT (backend default) · Path: ggml-ork

Hypothesis

fp16/W16A16 (the first cut) works but uses 2 bytes/weight and the NPU MAC is same-precision. Hypothesis: int8 W8A8 — per-channel int8 weights + per-row int8 activations, int32 accumulate, then dequant d[m][n] = aScale[m]·bScale[n]·C[m][n] — halves weight bytes at acceptable accuracy and is the natural target from GGUF Q8_0.

Implementation

• Commit: 0eabf5f (oRKLLM/llama.cpp-rockchip, ork-backend); default unless ORK_QUANT=4.
• Symbol: ggml-ork.cpp : ggml_backend_ork_mul_mat_i8.

Per-channel weight quant (cache-miss path, done once per weight):

float mx = 1e-9f;
for (int k = 0; k < K; k++) { float v = fabsf(f32[(size_t) n*K + k]); if (v > mx) mx = v; }
float s = mx / 127.0f; ow.bscale[n] = s;
for (int k = 0; k < K; k++) {           // B[k][n] = src0[n][k]
    int q = (int) lrintf(f32[(size_t) n*K + k] / s);
    bi[(size_t) k*N + n] = (int8_t) (q > 127 ? 127 : q < -127 ? -127 : q);
}
// ... ow.w = ork_mm_pack_i8(ctx->npu, K, N, bi);

Per-row activation quant (SIMD-friendly inner loops) → NPU run → int32→fp32 dequant:

for (int m = 0; m < M; m++) {
    const float * yr = y + (size_t) m*K; int8_t * ar = ai + (size_t) m*K;
    float mx = 1e-9f;
    for (int k = 0; k < K; k++) { float v = fabsf(yr[k]); mx = v > mx ? v : mx; }
    as[m] = mx / 127.0f;
    const float inv = 127.0f / mx;
    for (int k = 0; k < K; k++) {
        float q = yr[k] * inv;
        int qi = (int) (q + copysignf(0.5f, q));        // branchless round
        ar[k] = (int8_t) (qi > 127 ? 127 : qi < -127 ? -127 : qi);
    }
}
if (ork_mm_run_i8(ctx->npu, ow.w, M, ai, ci)) return false;   // int32 result in ci
// dequant: d[m][n] = aScale[m] * bScale[n] * C[m][n]
const float * bs = ow.bscale.data();
for (int m = 0; m < M; m++) {
    const float rs = as[m]; const int32_t * cr = ci + (size_t) m*N; float * dr = d + (size_t) m*N;
    for (int n = 0; n < N; n++) dr[n] = rs * bs[n] * (float) cr[n];
}

supports_op gates K%32==0 && N%64==0 (N%64 satisfies both int8 %32 and int4 %64).

Benchmark (standardized — 2026-06-13)

Qwen3-1.7B-w8a8, llama-bench -r 2, DDR 2112 / CPU 2.4 GHz. int8 is the default path; compared against the pure-CPU baseline and the int4 alternative (all NPU arms ORK_MINM=1; hybrid is the shipped default).

Config	pp512	tg128
int8 hybrid (ORK_MINM=32, shipped)	135.28 ± 0.97	9.52 ± 0.04
int8 all-NPU (ORK_MINM=1)	131.42 ± 2.29	5.03 ± 0.15
int4 all-NPU (ORK_QUANT=4)	3.66 ± 0.01	2.53 ± 0.00
pure CPU (ORK_OFF=1)	35.49 ± 0.23	9.43 ± 0.18
librkllmrt W8A8 (closed, same scheme + model)	183.9	11.71

librkllmrt uses the same W8A8 scheme — so this is a like-for-like quant comparison. The closed runtime is 1.36× prefill / 1.23× decode ahead of the shipped hybrid.

Profile (all-NPU int8, reference): quant 1% / NPU run 98% / dequant 1%, 386 µs/matmul — the path is submit-bound, not quant-bound.

Results

• Coherent output; ~1% matmul error vs fp32; half the weight bytes of fp16.

Verdict

✅ The practical path and the backend default. int4 (W4A4) is experimental; fp16 was the superseded first cut. Decode rate is ~10 tok/s regardless of model size here — the tell that it's submit-floor-bound, not compute-bound.