Date: 2026-06-12 · Status: ✅ KEPT (foundational) · Path: ggml-ork

The first ggml-ork cut packed+uploaded each weight plane to the NPU on every matmul call. Model weights are constant during inference. Hypothesis: pack+quantize each weight once, keep it NPU-resident, and key it by the weight plane's host pointer — eliminating per-call pack churn.

• Commit: 4484b5b (oRKLLM/llama.cpp-rockchip, ork-backend)
• Symbol: ggml-ork.cpp — ork_weight cache keyed by src0->data, freed on backend teardown.

struct ork_weight {
    ork_w * w = nullptr;        // NPU-resident packed weight
    std::vector<float> bscale;  // per-channel (int8) or per-group (int4) dequant scale
    int gsize = 0;              // int4 group size (0 for int8)
};

// model weights are constant during inference, so pack+quantize each once (NPU-resident) and
// reuse, keyed by the weight plane's host pointer.
std::unordered_map<const void *, ork_weight> wcache;

Lookup / build-once on the matmul hot path:

auto it = ctx->wcache.find(x);          // x = src0->data (the weight plane)
if (it == ctx->wcache.end()) {
    // ... dequant src0 -> f32, per-channel int8 quantize -> bi, record bscale ...
    ow.w = ork_mm_pack_i8(ctx->npu, K, N, bi);   // pack + upload ONCE
    if (!ow.w) return false;
    it = ctx->wcache.emplace(x, std::move(ow)).first;
}
const ork_weight & ow = it->second;     // every subsequent token reuses the resident weight

Teardown:

for (auto & kv : ctx->wcache) ork_w_free(kv.second.w);

Qwen3-1.7B-w8a8, all-NPU (ORK_MINM=1), DDR 2112 / CPU 2.4 GHz. The "off" arm uses the new ORK_NOCACHE=1 knob (re-pack the weight every matmul + free), benchmarked with reduced tokens since it is pathologically slow.

librkllmrt packs weights once at model-load (resident for the session) — the same strategy this experiment introduced; it never re-packs per matmul.

The standardized A/B is binary: with weight caching the model runs; without it, re-packing every matmul reproduces the original ork bcreate EFAULT (CREATE: Bad address) and the run dies. Caching isn't an optimization here so much as a correctness prerequisite at 1.7B scale.

Earlier (SmolLM2-135M, when no-cache still limped along): decode 17×, 0.53 → 9.34 tok/s (107 ms/token).

• Decode 17×: 0.53 → 9.34 tok/s (107 ms/token) on SmolLM2-135M.
• Also removed the intermittent CREATE: Bad address (ork bcreate EFAULT) caused by per-call MEM_CREATE churn — reconfirmed 2026-06-13 via ORK_NOCACHE=1.

✅ Foundational and kept. This is what makes the backend usable; every later experiment builds on it. The corollary lesson: the pack is lazy on first use, so benchmarks must warm up first or the one-time pack pollutes the timed window (see the prefill artifacts in Hybrid M-Gate / Open vs Closed).