Tools: src/npu.c (ork_mm_pack_i4a8 / ork_mm_pack_i4a8_im / ork_w_dump_i4a8 / ork_mm_load_i4a8; quant_kind, ORK_NF4, ORK_SR) · tools/gguf_tier_map.c · examples/quant + a wikitext-2 PPL harness (RK3588 10.3.0.236, governors pinned). Commits dc782b2 (NF4+SR), 93364b7 (imatrix), d656955 (compact persist), 378045d (bscale NULL-guard), 2c38a27+f78bc04 (gguf_tier_map).

Three accuracy levers landed on the int4-stored / int8-computed path (ork_mm_pack_i4a8): an NF4 codebook (bitsandbytes' 16 non-uniform levels, NEON vqtbl1q_s8 inflate), an imatrix-weighted per-channel scale (clip-ratio grid search minimizing importance-weighted reconstruction error), and a compact 'O4N1' persist (~½ the tiled-int8 dump). A PPL gate on Qwen3-1.7B (wikitext-2, 20 chunks, board) settles the design: fp16-source values + Q4_K_M allocation + imatrix + NF4 = 11.33 PPL @ ~58% int8 storage — beating double-quant-from-Q4_K_M (12.48) by −1.15 and validating the rule "fp16 is the source; the mixed GGUF is an allocation oracle only, never re-quant its bytes."

Per-output-channel absmax, then nearest of the 16 fixed bitsandbytes NF4 levels → a 4-bit index stored in the existing Bi4 nibble store. Inflate is NEON vqtbl1q_s8 against an int8 LUT = round(level * 127); the dequant scale is bscale = absmax / 127. It reuses the existing tile_f32_i8 path (the LUT puts the codebook into the in-range int8 codes, so the downstream tiling is unchanged).

check (RK3588)	result
NF4 correctness vs CPU ref	RMS 0.873%
NF4 reconstruction err, Gaussian N(0,1) weights	0.1102
uniform-int4 reconstruction err, same weights	0.1581
ratio (NF4 / uniform)	0.697 — NF4 wins

NF4's non-uniform levels match the Gaussian weight distribution → lower reconstruction error than uniform int4 on the same weights.

ORK_SR dithers each weight between its two bracketing NF4 levels (probability ∝ distance), instead of always taking the nearest. Design note: SR (and imatrix, below) is a dot-product / PPL effect, not a per-weight reconstruction-RMS win — SR raises per-weight RMS while removing the systematic accumulation bias that a deterministic nearest-pick introduces across a length-K dot product. Judge it by PPL, not by RMS.

The subtlety: the imatrix is per-INPUT-channel (K) while the scale is per-OUTPUT-channel (N) — they are orthogonal, so importance can't re-weight a nearest-level pick (a given output channel's scale touches all K inputs). Instead the scale is chosen by a clip-ratio grid search: for each output channel, pick the per-channel scale minimizing Σ_k imatrix[k]·(w − dequant)².

check	result
weighted-error reduction (grid vs absmax)	6.4× (4.001e5 → 6.273e4)
imatrix = NULL	byte-identical to absmax (the API is a pure superset)

ork_w_dump_i4a8 emits an 'O4N1' blob = hdr{magic, ver, K, N, quant_kind} + bscale[N] + the Bi4 nibble store; ork_mm_load_i4a8 reloads it straight into a packed weight. ~½ the tiled-int8 dump size. Validated: streamed output bit-identical to resident, byte-identical re-dump, for both UNIFORM and NF4. (378045d is a NULL-guard fix on the bscale_out argument.)

tools/gguf_tier_map.c parses any GGUF's per-tensor ggml_type and assigns each tensor a tier {int8 | int4} by an effective-bits threshold (≥5-bit → int8). On Qwen3-1.7B-Q4_K_M it splits 140 q4_K → int4 + 28 q6_K → int8, matching the convert split exactly. This per-tensor tier doubles as the allocation oracle for mixed-precision packing (and, later, as an engine-assignment oracle — see Heterogeneous Serving and Scheduling).

The decisive comparison — what actually feeds the NPU vs the baselines:

config	PPL	notes
Q4_K_M native (CPU)	10.92	the reference floor
all-int8 NPU	11.19	the int8-activation anchor
fp16-values + Q4_K_M-allocation + imatrix + NF4 (ours)	11.33	~58% int8 storage
double-quant-from-Q4_K_M	12.48	re-quantized the GGUF's already-quantized bytes
UD-Q4_K_XL	17.91

Two conclusions. (1) Clean fp16-source + imatrix beat double-quant by −1.15 → re-quantizing an already-quantized GGUF compounds error; the mixed GGUF must be used as an allocation oracle only (which tensor gets which tier), never as the values to re-quantize — go back to the fp16 source for the bytes. (2) The residual gap to Q4_K_M (11.33 vs 10.92) is the int8-ACTIVATION cost (w8a8/w4a8 — the NPU's same-precision MAC quantizes activations), bounded above by the all-int8 anchor (11.19). It is not a weight-quant deficiency.

✅ A mixed int8 (attn/q6_K-tier) + int4-NF4 (FFN/q4_K-tier) weight scheme, quantized from the fp16 source with imatrix-weighted scales, reaches PPL 11.33 at ~58% int8 storage — within 0.4 of Q4_K_M and beating every re-quant-the-GGUF approach. The NF4 codebook + imatrix scale + compact persist are committed; gguf_tier_map is the oracle that drives the allocation. The remaining gap is the int8-activation cost, not the weights.

ORK_NF4=1 (codebook) · ORK_SR=1 (stochastic rounding into NF4) · ork_mm_pack_i4a8_im(..., imatrix, ...) (NULL ⇒ absmax) · gguf_tier_map <gguf> [--emit-map].

• Feed gguf_tier_map --emit-map directly into ggml-ork's per-tensor pack-tier selection (end-to-end mixed-precision load).
• The int8-activation residual is the same wall as everywhere — a coherent w4a8/W4A4 activation path (Tier 4) would close it, but is not yet viable.