Experiment — Activation-quant reuse

Date: 2026-06-13 · Status: ✅ KEPT (default on) · Path: ggml-ork

Hypothesis

In a transformer block, Q/K/V projections share the same (normed) hidden-state activation, and gate/up share their input. The int8 path re-quantizes the activation (F32 → per-row int8) on every matmul. Hypothesis: skip re-quantizing when the previous matmul had the same src1 (same M, K) — the cached ai/as buffers still hold it (mathematically identical). This is the safe, salvageable part of fusion without the slow wide matmul.

Implementation

• Commit: b72a84a4 (oRKLLM/llama.cpp-rockchip, ork-backend)
• Symbol: ggml-ork.cpp : ggml_backend_ork_mul_mat_i8 — reuse gate src1->data == ctx->last_src1 && M==last_M && K==last_K; the fused-group path sets last_src1 = nullptr to avoid stale reuse.
• Always on (no knob); verified coherent output.

// activation: per-row int8 quant. Reused if the previous matmul had the SAME src1
// (QKV / gate-up share their input) — ai/as still hold it.
const bool reuse_act = (src1->data == ctx->last_src1) && (M == ctx->last_M) && (K == ctx->last_K)
                       && i12 == 0 && i13 == 0;
if (!reuse_act) {
    for (int m = 0; m < M; m++) {
        const float * yr = y + (size_t) m*K; int8_t * ar = ai + (size_t) m*K;
        float mx = 1e-9f;
        for (int k = 0; k < K; k++) { float v = fabsf(yr[k]); mx = v > mx ? v : mx; }
        as[m] = mx / 127.0f; const float inv = 127.0f / mx;
        for (int k = 0; k < K; k++) {
            float q = yr[k] * inv; int qi = (int) (q + copysignf(0.5f, q));
            ar[k] = (int8_t) (qi > 127 ? 127 : qi < -127 ? -127 : qi);
        }
    }
    ctx->last_src1 = src1->data; ctx->last_M = M; ctx->last_K = K;   // remember for next matmul
}

The ctx fields that carry the cached activation between calls:

const void * last_src1 = nullptr; int last_M = 0, last_K = 0;   // holds while ctx->ai/as match

Benchmark (standardized — 2026-06-13)

Qwen3-1.7B-w8a8, llama-bench (-r 2, decode arm isolated with -p 0 -n 128), DDR 2112 / CPU 2.4 GHz. Both all-NPU (ORK_MINM=1); the only difference is ORK_NOREUSE.

Config	pp512	tg128
reuse on (default)	131.42 ± 2.29	5.03 ± 0.15
reuse off (ORK_NOREUSE=1)	121.70 ± 3.16	5.13 ± 0.01
librkllmrt (closed ref, same model)	183.9	11.71

(librkllmrt has no reuse toggle — it's the fixed closed reference; the A/B is between the two ggml-ork arms.)

Micro-level (earlier): activation-quant time −37% (40 → 25 ms over the run).

Verdict

✅ Kept — and the standard relocates the benefit to prefill. Reuse gives ~8% prefill (131.4 vs 121.7, outside the error bars) but is neutral on decode (5.03 vs 5.13 — within noise). That makes sense: in prefill the shared activation (Q/K/V off the normed hidden state; gate/up off the same x) is a large M×K quantization done once instead of 2–3×; in M=1 decode the quant is tiny vs the 98%-NPU-submit floor. Confirms the CPU-side decode ceiling is ~3% (quant 1% + dequant 1% on the reference profile); the decode answer is routing decode to the CPU, not quant tricks. Free and correct, so kept on.