Experiment — Hadamard rotation for int4 (W4A4) accuracy

Date: 2026-06-13 · Status: ✅ MECHANISM CONFIRMED (matmul-level); go/no-go deferred to perplexity · Path: CPU math probe (no NPU)

Hypothesis

int4 W4A4 runs on the NPU but is incoherent — the same-precision MAC forces 4-bit activations, and activation outliers blow out the per-group absmax scale, so the 16 int4 levels land mostly on near-zero values. A Hadamard rotation Q = H/√K (orthonormal, symmetric) is the QuaRot/SpinQuant fix: A·B = (A·Q)·(Q·B) exactly in fp32, but the rotated A'=A·Q, B'=Q·B have their outliers spread evenly across channels, so per-group int4 quant of A',B' is far more accurate. Hypothesis: a Hadamard pre-rotation materially reduces W4A4 matmul error — the prerequisite for int4 ever being usable.

Method — measure before integrating

Rather than wire Hadamard into the model pipeline first, two pure-CPU probes measure the matmul quant error (RMS rel vs the fp32 reference), plain vs Hadamard-rotated, across bit-widths:

• tools/hadamard_int4.c — synthetic Gaussian A,B + injected activation outliers. fwht_norm() (fast Walsh-Hadamard + ×1/√K) is the rotation; q_dequant_vec(x,out,K,G,bits) is per-group symmetric int quant→dequant for any bit-width; matmul_q_err(...,abits,bbits,rotate) is the RMS metric.
• tools/hadamard_real.c — same harness but the weight B is a real Q4_K tensor dequantized from a GGUF (random data is the worst case for quant; this removes that objection). Activations stay synthetic Gaussian + outliers (real LLM activations are ~Gaussian with heavy tails — that's the thing Hadamard tames). outlier_scale is a CLI arg so the benefit-vs-severity trend is measurable.

Both print 5 rows: W8A8 (sanity), W4A4 plain / +Hadamard, and W4A8 (act8/wt4 — not HW-supported, included as a diagnostic to isolate which operand's int4 hurts).

Results

Synthetic (hadamard_int4, M=16 K=2048 N=512, outliers ×20):

quant	G=128	G=32
W8A8 sanity	2.06%	1.31%
W4A4 plain	26.4%	18.8%
W4A4 + Hadamard	16.4%	13.2%
W4A8 plain (act8/wt4)	11.6%	9.5%
W4A8 + Hadamard	11.7%	9.4%

Real weights (hadamard_real, token_embd Q4_K, K=512 N=512 M=16):

quant	×20 G128	×20 G64	×4 G128
W8A8 sanity	3.05%	2.37%	1.35%
W4A4 plain	46.8%	32.1%	22.9%
W4A4 + Hadamard	18.8%	17.3%	13.5%
W4A8 plain (act8/wt4)	20.7%	17.1%	15.0%
W4A8 + Hadamard	13.8%	12.8%	9.7%

Three robust facts, consistent across synthetic and real weights:

1. Hadamard cuts W4A4 error 40–60% (46→19, 32→17, 26→16). The mechanism works.
2. The benefit scales with outlier severity — ×20 outliers: 46→19; ×4 outliers: 23→13.5. This is the QuaRot signature: the rotation buys exactly as much as there is outlier mass to spread. Real LLM activation outliers (often 10–100×) sit at the high end.
3. It's the activations. Hadamard helps W4A4 (int4 acts carry the outliers) much more than W4A8 (int8 acts already absorb outliers; the remaining error is the int4 weights, which have no outliers to spread). Confirms activation-int4 is the dominant error source — and W4A8 would be far better, but this NPU can't do W4A8 (same-precision MAC).

Verdict

✅ Mechanism confirmed — Hadamard is the right lever and the only one that addresses the W4A4 outlier problem. But matmul RMS is a diagnostic, not the go/no-go. int4 quant error is unbiased and averages out across a matmul (N reductions) and across residual-connected layers — which is precisely why QuaRot/SpinQuant int4 LLMs are coherent at ~0.5 perplexity cost despite double-digit per-matmul RMS. So an 18% rotated-W4A4 RMS does not mean "broken"; the real verdict needs end-to-end perplexity from a Hadamard-wired int4 path.

Sequencing call: 4b (speed) gates 4a (accuracy). Even if Hadamard makes int4 coherent, int4 today is ~23× slower on prefill / ~1.8× on decode than int8 (the per-group K/G-submit explosion on a submit-floor-bound NPU). int4 is only worth the integration cost if 4b submit-count reduction (per-channel int4 → full-K single submit, + graph fusion) also lands — otherwise int8 strictly wins. The rational order is therefore: prove 4b can get int4 within ~2× of int8, then spend the Hadamard-pipeline integration to make it coherent. This probe de-risks 4a cheaply (mechanism proven) so that decision can be made on the perf result.

Reproduce

make hadamard_int4 && ./hadamard_int4 16 2048 512 128         # synthetic, no model
make hadamard_real && ./hadamard_real model.gguf 16 512 128 20 # real Q4_K weights, outliers ×20

Both are CPU-only (-lm, no NPU) — they run on the build host. hadamard_real finds the first Q4_K 2D tensor and uses the largest power-of-2 channel slice (plain FWHT needs power-of-2 K).