Board 10.3.0.236 · Kernel 6.1.115-vendor-rk35xx-fence (NOT the -sram build; results are not comparable across kernels) · ork-driver 1.0.154 (eb3e66f) · fork diverged-history @ 9e3001b16, clean clone ~/llama-pinned built at that pin · governors verified before every arm (DDR performance @ 2112 MHz, A76 cpu4..7 performance).

Harness ork_ppl <model> <text> <window> <ubatch> <maxwin> · Corpus first 131072 bytes of wikitext-2-raw/wiki.test.raw (/home/michael/ppl_screen.txt) · 256-token windows, KV cleared between chunks, ubatch=256.

The goal for 4-bit on this stack is parity with GGUF 4-bit, not with int8, and it is to be reached by stacking techniques — GPTQ, rotation, activation clipping, mixed-precision promotion — none of which is expected to close the gap alone. That target had never been measured on the same harness as the W4A4 numbers, so "how far from parity" had no value. Every previous W4A4 figure was quoted against int8 or against RTN.

This entry measures the target line and the current default stack against it, on one harness, one model, one corpus.

2 × 256-token windows (510 scored tokens). This is the comparable set; everything below shares corpus, window count, window size, ubatch and binary.

Longer run, 64 × 256-token windows (16320 scored tokens), for the target only:

W4A4 at 4 windows measured 91.07; its running PPL climbs steeply with window count (56.0 → 62.0 → 77.3 → 91.1 across windows 1-4), so window counts must match before any two numbers are compared. An earlier draft of this experiment compared a 4-window W4A4 figure against a 64-window Q4_K_M figure; that comparison was meaningless and is retracted.

• int8 already beats the GGUF 4-bit target on this model (22.92 vs 24.66), at 8 bits against 5.5.
• W4A4 is 3.7× the target. For scale, GPTQ's recorded 34%-over-RTN improvement, if it transferred unchanged, would move 83.96 to roughly 55 — a large step and a real contribution to a budget several techniques must close together, but not parity on its own. That 55 is an extrapolation across a different corpus and configuration and has NOT been measured here.
• The GGUF 4-bit variants are nearly the same size on this model — Q4_0 489 MiB, Q4_K_S 492 MiB, Q4_K_M 516 MiB — because at 0.8B the embedding and output head dominate the file and are kept at higher precision in all of them. So "Q4_K_M is a generous 5.50-bpw yardstick" is weaker than it sounds: there is no much-cheaper GGUF 4-bit to compare against here.

Q4_K is weight-only: it dequantises to fp16/fp32 and runs against full-precision activations, effectively W4A16, carrying no activation-error term. W4A4 pays both halves, which Exp-2026-08-23 measured at parity (A/W = 0.97) and adding in quadrature. Parity with GGUF 4-bit therefore means matching a format that pays one of the two costs we pay.

The W4A4 penalty on this model is 2.6× larger than previously recorded. Candidate explanations, none tested:

1. The model is a Gated DeltaNet, not a plain transformer. ORK_OP_STATS shows GATED_DELTA_NET ×108 and SSM_CONV ×342 in the graph. Linear-attention state may be far more quantisation-sensitive than softmax attention, and none of the W4A4 quality work was developed or validated on a GDN model.
2. Per-channel vs grouped scales. ORK_I4_GROUP defaults to 0 (per-row). An older note records per-channel int4 at PPL ~104, close to what is seen here — but Exp-2026-08-23 rejected per-group scales as a poor trade (−1.26% PPL for +5.7% size), which argues the grouping axis is not worth 60 points of PPL. These two records disagree and a probe re-run should settle it.
3. Short windows with KV cleared inflate all arms, but that cannot explain a ratio difference.

Until this is resolved, do not generalise 83.96 to W4A4 as a technique — it may be a property of this model family.

Two confident conclusions were stated during this session and both were artifacts of the instrument, not findings. Both are the same failure the wiki already records for dmesg "soft reset": trusting a readout without checking what it measures.

1. "The NPU executed zero jobs." /sys/kernel/debug/rknpu/counters reported commit 0 / irq 0 / done 0 through a whole run. The counters are gated and default to OFF after a reboot — the file says so in its own body (gated counters: off, (write "on" to start counting these)). They had read 228007 earlier only because a previous session had enabled them. Write on to the counters file after every reboot before trusting a zero.
2. "supports_op claims 1358 matmuls but zero handlers run." The [ORK ROUTE-STATS] table looked empty. The grep pattern matched the table's header line but not its data rows, which were present the whole time (chain_i4 ×96, mul_mat_i8 ×204). There was no routing bug.

A third, procedural: the first W4A4 arm coupled the pack build to the measurement under one timeout, so a slow build consumed the window and the run produced no number at all in 45 minutes. Build the pack as its own step, then measure against the finished pack. The W4A4 pack for this model is 102 weights / 1027.4 MiB and takes ~5 minutes of pure CPU (pack building is CPU-only by design — ork_i4_w_dump_cpu; 0% NPU during a build is expected and is not evidence of a CPU fallback).

Attempted, not measured. ORK_GPTQ=1 ORK_QUANT=4 ork_bench …:

[ork_bench] ORK_GPTQ two-phase: calibration batches of M=512, count derived from the largest weight K
[ork_bench] ORK_GPTQ: largest K=0 -> 0.00 GiB/layer of Hessian; 12 GiB budget -> window = 24 layer(s)
[ORK GPTQ] calibration window = layers [0,24)
[ork_bench] ORK_GPTQ: window 0 registered no weights — skipping

Phase 1 registers only native-W4A4 weights (the branch that calls ork_i4_mm_pack / ork_i4_mm_pack_grouped and defers persistence to gptq_finalize). After the discovery decode ggml_backend_ork_gptq_max_k() returns 0, so the window registers nothing, finalize has nothing to quantise, and the run completes having written a plain RTN int4 pack.

This is the dangerous part: it exits 0 and writes a valid pack. A GPTQ run on this model is byte-indistinguishable from a non-GPTQ run, so any PPL attributed to "GPTQ" here would actually be RTN. The only signal is that one skipped line in the log.

The obvious suspect was the model family (Gated DeltaNet, fingerprint n_ff=0, GATED_DELTA_NET and SSM_CONV in the graph instead of a conventional FFN). That was tested and refuted. The same probe on qwen3-0.6b-f16, a plain dense Qwen3 transformer:

[ork_bench] ORK_GPTQ: largest K=0 -> 0.00 GiB/layer of Hessian; 12 GiB budget -> window = 28 layer(s)
[ork_bench] ORK_GPTQ: window 0 registered no weights — skipping
[ORK RECIPE] model fingerprint: n_layer=28 n_embd=1024 n_ff=0 attn_kv=1024 soc=rk3588

Identical: zero weights registered, and n_ff=0 reported for a model that certainly has an FFN. So GPTQ registration is broken generally in this build, not on one architecture.

Nor is it that packing fails — packing works and takes the int4 tier:

[ORK PERSIST] SUCCESS: orkpack written (102 weights, 1027.4 MiB)
[ORK PERSIST] ... is an int4 pack — selecting the int4 tier (no ORK_QUANT needed)

So weights pack as int4 through a path that is not the native-W4A4 branch carrying the phase-1 registration hook (the branch that calls ork_i4_mm_pack / ork_i4_mm_pack_grouped and defers persistence to gptq_finalize). The likely split is the i4a8 tier — int4 storage inflated to int8 — versus native W4A4 (DT_I4), with only the latter hooked. Unconfirmed.

Consequence for the stacking plan: the GPTQ rung is blocked on every model until registration is understood; moving to a plain transformer does not route around it. The non-GPTQ rungs (rotation, activation clipping, grouping, mixed-precision promotion) are unaffected and can be laddered now.

• Grouped int4 (ORK_I4_GROUP=G) against the 24.66 target — settles the contradiction in §3(2).
• GPTQ arm (ORK_GPTQ=1 + calibration + gptq_finalize) against the same target. Worth running as a stacking contribution; the ~55 figure above is extrapolated and must be replaced with a measurement.
• bf16 floor on this screen. Not yet measured, and without it none of the above can be expressed as a ratio to full precision. Published Q4_K_M behaviour is ~+0.18 PPL over f16 (Llama-3-8B), so the floor should sit just under 24.66 if the harness is sound — a cheap validity check that has not been run.
• Is the GDN model representative? Repeat the int8/W4A4 ratio on a plain transformer of similar size.

Code: tools/re/ork_ppl.cpp (harness) · ggml-ork.cpp ork_supports_op_inner / ork_route_stat (ORK_OP_STATS=1) · ORK_QUANT=4, ORK_I4_GROUP, ORK_I4_NOCLIP, ORK_I4_NOROT, ORK_OFF=1 (CPU baseline — supports_op returns false unconditionally).

Everything above was measured with ORK_MIXED_W4A4 unset, which — discovered later the same session — silently selects a different datapath. Part II is the corrected work: a rung-by-rung decomposition on a plain dense transformer, with each technique isolated.

Model qwen3-0.6b-f16.gguf (dense Qwen3, f16 source, 28 layers, n_embd 1024, n_ff 3072) · target its own Q4_K_M built from the same f16 (5.09 BPW, 456 MiB) · screen wikitext-2 first 128 KiB, 64 × 256-token windows, ubatch=256, KV cleared between windows · ork-driver 1.0.154 · kernel 6.1.115-vendor-rk35xx-fence · governors verified.

static bool ork_w4a4_native_on(void) { return env_enabled("ORK_MIXED_W4A4"); }
...
bool native_w4a4 = (target_qbits == 4) && ctx->hadamard;
ork_route_stat(native_w4a4 ? "mul_mat_i4_hadamard" : "mul_mat_i8");

With ORK_QUANT=4 alone you do not get W4A4. You get int4 weights inflated to int8 and run through mul_mat_i8 — effectively W4A8 — with no Hadamard rotation (rotation is inseparable from ctx->hadamard) and no GPTQ registration (phase 1 hooks only the native branch, so gptq_max_k() returns 0 and the run writes a plain RTN pack while exiting 0).

That one flag explains every anomaly in Part I. It is the single most important thing on this page.

Overall 1737 → 48.78. Remaining gap to GGUF 4-bit: 1.31× at ~5 bpw, or 1.55× at the capacity-honest 4.0 bpw (arm A — the number that matters for the 20+GB tier, which cannot spend the extra bits promotion costs).

THE TABLE IS NOT A STAIRCASE. Rungs J and A are both measured against G, not against each other — A was built on plain Hadamard, not on RHT. Reading it top-to-bottom as compounding steps overstates the total, because the techniques overlap (§8b).

PPL here is deterministic for a given pack, so repeated runs are identical; the uncertainty is the 16320-token sample, not the measurement. The tell is internal inconsistency at small effect sizes: RHT measured −1.9% at the GPTQ rung (O) but +1.2% once promotion was added (P vs F). Those cannot both be clean effects of the same change.

So: effects ≥20% here are safe (rotation, both clips, GPTQ, promotion). Effects under ~2% should not be used to rank configurations — which retroactively weakens the act-order (−3.6%) and RHT (+2.6%) readings as magnitudes, though act-order's sign was consistent wherever it was measured.

The clearest pattern of the session, and it generalises beyond any one knob:

Both the transform and GPTQ attack weight-side conditioning — the transform by making the distribution easier to quantize, GPTQ by compensating residuals into not-yet-quantized columns. Once GPTQ's error feedback is doing that work, a better-conditioned starting point has little left to give, and can cost.

WUSH's own numbers show exactly this shape: +2.8 PPL over Hadamard with RTN, only +0.7 with GPTQ (arXiv 2512.00956). That reading dropped its expected value here from "the main remaining hope" to plausibly zero — and it is also non-orthogonal, so the activation side needs an inverse their fused GPU kernel supplies and our fixed-function NPU cannot. Do not invest in a data-aware transform on top of a GPTQ stack without re-deriving this.

Corollary for where effort goes: every weight-side refinement tried today disappointed once GPTQ was present. The remaining headroom is on the activation half, which only clipping touches.

Rotation is not one technique among several — it is the technique. ÷11.3 against ÷1.3-1.5 for everything else. Any W4A4 result measured without it is measuring a broken configuration.

ORK_GPTQ_ACTORDER=1 (implemented this session in ork_i4_gptq) quantizes columns in descending diag(H) order — the standard desc_act refinement — then writes codes back to their original positions. It made perplexity WORSE: 59.56 vs 57.49 (+3.6%).

Not a bug: a broken un-permutation scrambles codes and lands in the hundreds or thousands; 59.56 is a sane number from a working round-trip. (The existing test_gptq property test does not cover this — with H=I every diagonal is equal, so the sort yields the identity permutation and the round-trip is never exercised. Both assertions pass identically with act-order on and off, which is itself the tell.)

Why it loses here, and it is the same property that made it free. Act-order's benefit in the literature comes largely from its interaction with group boundaries — ordering columns so the important ones fall favourably within a group. We use per-row scales: one scale per output channel covering every column, so there are no boundaries and no such benefit — while the disruption to GPTQ's error-propagation order remains and costs. The property that let us skip storing a permutation (per-row scales make column order irrelevant to dequantization) is exactly the property that removes the upside.

Do not re-try act-order while scales are per-row. It would become worth re-testing only alongside grouped scales — which are themselves parked (§11).

Two priors at comparable budget, both on top of GPTQ:

The Q4_K_M prior wins while spending 6 MiB LESS. The two lists barely overlap: qerr's top picks are attn_output — which Q4_K_M deliberately leaves at 4 bits — and it never selects attn_v, which Q4_K_M bumps in every other layer.

Why: qerr measures the wrong thing. It is a weight's own rounding error, not how much that error reaches the output. And on this model it carries almost no signal anyway — measured distribution:

qerr: max 0.2209  median 0.1470  min 0.1306  |  top decile holds 12.8% of total error

A 1.69x spread, and the worst 10% of weights hold 12.8% of the error — i.e. their proportional share. There is no concentration to exploit, so worst-first promotion returns roughly linearly in budget. This independently reproduces a published finding: "at INT4/g16 the spread between modules is too small to allocate against" (arXiv 2609.16391).

Tool: tools/orkpack_qerr_rank.c (reads a pack's index only — no blobs, no model load — prints the distribution and emits an ORK_I4_INT8_LAYERS list for a budget; [exclude_file] charges an existing policy's picks against the budget so the two priors can be combined).

Promotion is not free for the capacity tier. It buys ÷1.16 at +27% resident bytes — the exact resource W4A4 exists to protect (§12). For a 20+GB model it should be off or near-off.

• NF4 on native W4A4 — permanently impossible. The 4-bit value indexes a codebook, which requires an index→level indirection before multiplication. The int4 MAC multiplies codes directly, and the only LUT in the pipeline is in SDP, after accumulation — by which point the individual weights are summed and gone. NF4 works only on the i4a8 path, where it inflates to DT_I8 before the MAC.
• Grouped scales — runs on the NPU, but ~24x the cost. A K-group is structurally an expert ([G x N], Sk=1), so all groups go in one BCHAIN submit (a measured 12x win over the old row-decomposed path), but the NPU returns Sk * M * N int32 partials and the per-group scale-and-sum happens in the drain. Net ~24x a per-channel matmul for a recorded −1.26% PPL. Parked.
• W4A8 does not help the capacity tier. Bi4 is a host-side nibble store "for .orkpack/streaming dump"; the NPU side runs int8 (DT_I8). So i4a8 is 4 bits on disk and 8 bits resident — double W4A4 in the ~4 GiB IOVA window. It remains the right choice for the smaller-model tier, where residency is not binding and it can carry rotation + GPTQ + NF4 that Q4_K lacks.

The same ctx->hadamard welding blocks the second tier: rotation and GPTQ registration are reachable only through native W4A4, so "rotated, GPTQ'd, NF4 W4A8" cannot be built today. Decoupling the axes — weight bits, activation bits, rotation, codebook — unlocks a production tier, not just a cleaner experiment.

The activation half is where the headroom is (§8b). Two candidates, and the important engineering point is that they need identical machinery:

out[n] = Σ_j W[n][j]·a[j]   with   W[n][j] = c_j·r_n·q[n][j]
       = r_n · Σ_j q[n][j]·(c_j·a[j])          ← c_j folds into the ACTIVATION

• SmoothQuant / AWQ migration — c_j from calibration statistics, which we already have in diag(H). Moves error from activations (only clipping) to weights (where GPTQ absorbs it), so it is complementary by construction rather than a substitute — unlike everything in §8b.
• SINQ (arXiv 2509.22944, code) — the same per-channel scale derived calibration-free by a Sinkhorn-Knopp iteration normalising per-row and per-column variances. Evaluated on the Qwen3 family, and aimed squarely at our failure mode: "representing outliers causes precision issues in parameters that share the same scales as these outliers … especially pronounced for calibration-free, uniform quantization." Caveat: it is weight-side normalisation, so it may hit the same GPTQ substitution wall; SmoothQuant is the more robust bet.

Build the mechanism once — persist c[K] per weight, apply before the rotation — and both are a different formula for c_j. Costs: one O(K) multiply on the activation row (same class as RHT's sign flip) and K floats per weight (~0.8% of a 1.5 MiB weight). It is the first change here that needs a pack format change, which is also the first thing that would make new packs unreadable to older builds — exactly what the §"pack header gate" work exists to catch.

Other open items:

• Wider clip grids — now tunable (ORK_I4_CLIP_N/_STEP, ORK_GPTQ_CLIP_N/_STEP) but not yet swept. The floors (0.5625 activations, 0.781 weights) remain guesses.
• H-weighted clip (ORK_GPTQ_HWCLIP=1) — implemented as a cheap stand-in for Schur Replay, not yet measured.
• A bigger screen. Per §8a this one cannot resolve <2%, and most remaining levers are in that range.
• Scale. Every number here is 0.6B, near the worst case for 4-bit.

Code landed: ork-driver b8ba773 (act-order, tunable + H-weighted clip grid, orkpack_qerr_rank) · fork 927cd9a40 (weight MSE clip, RHT, tunable clip grid). All new knobs default off; the weight MSE clip is the one behaviour change, and it is on.

• Scale. Every number here is a 0.6B model, near the worst case for 4-bit. PrismQuant reports Llama-3.1-70B under W4A4KV4 within 0.22 points of full precision (arXiv 2609.32429, grouped asymmetric — not our configuration, but the size trend is the point). The tier this work serves is 20+GB, and the gap may narrow substantially with scale.
• Pending arms: hybrid promotion prior; RHT+GPTQ; RHT+GPTQ+promotion (every rung above J was measured on plain Hadamard and is stale by RHT's contribution).