int8 fused-SiLU output quantization — the FFN-chain PPL gap, and how to (partly) close it

Question: the on-NPU int8 SwiGLU chain is coherent + faster at small batch but loses PPL (hybrid ~16, all-int8 ~22 vs baseline ~9/11.5). Where does the gap live, and can it be closed on-NPU?

The gap is the int8 quantization of the silu OUTPUT (ablation, ORK_GATE_ABLATE)

Isolated the gate's three error sources. Decisive: same int8 gate matmul + same LUT-index path, but fp32 silu OUTPUT → PPL 8.94 ≈ baseline; flip only the silu output to int8 → ~16. So the ~7-PPL gap is entirely the per-tensor int8 quantization of silu(g) (stored int8 to feed the ewmul), NOT the LUT-input resolution (the LUT input is the int32 accumulator → ~1024-entry int16 LUT, already fine). The output CVT (0x4010) is int8-only on the matmul-fused program (int16/fp16 there wedges — see the fp16 exp log).

Why per-tensor int8 output is lossy: outlier-driven scale

s_silu = 1.15·max|silu|. A few large-gate outlier neurons blow up the max, so the bulk small silu values get coarse int8. Classic LLM activation-outlier problem — but on the OUTPUT scale, not the input.

What works: per-layer precision sprinkle (ORK_FFN_SILU_CPU_GMAX, fork a9d3e069b)

int8 fused silu on most layers (fast) + exact CPU fp32 silu on the few high-gmax outlier layers (their int32 gate acc → per-channel host dequant → fp32 silu = ablation mode-0 quality). gmax (max|gate|) is computed free in prep; a layer above threshold uses the exact path. Sweep (Qwen3-1.7B-Q8_0, 16 chunks):

config	exact layers	PPL
baseline (per-op)	–	11.50
int8 chain (fused all)	0	21.93
gmax>30	7 / 28	12.31
gmax>20	13	11.91
gmax>12	19	11.77

Knee at ~7/28 layers = 92% of the gap recovered. Portability: gmax scale is per-model → ship as top-K / median+c·MAD, auto-calibrated per model at load, NOT a fixed threshold.

What DOESN'T work: clipping the output outliers (ORK_FFN_SILU_PCT, fork d7e1b56aa) — REFUTED

Tested "clip silu-output outliers so the per-tensor scale stays small" (a plausible on-NPU fix). Sweep (16ch, no-clip=21.93): [email protected]=86.7, @99.5=155, @99.0=274, @98.0=584 — monotonically WORSE. Clipping just 0.1% of silu values quadrupled PPL. The large silu outliers carry critical signal (they dominate the down-projection sum) — the LLM.int8() lesson: preserve outlier RANGE, don't clip. This confirms the sprinkle's premise: give outliers precision (per-channel/exact), don't remove them.

Option map for the int8 output gap (external-advice audit)

• Clip outliers → REFUTED (above), catastrophic.
• Asymmetric zero-point (reclaim wasted [-128,0] codes via 0x4080/R) → preserves range, ~1 bit, safe but modest; won't reach the sprinkle. Not measured (banked as marginal).
• Per-channel output scale → the right diagnosis, but the fused stage carries only a scalar R (per-channel requant is host-side on this HW, as RKNN does). = exactly what the CPU sprinkle does on the outlier layers.
• Un-fuse (int32 matmul → standalone SDP fp16/int16 silu → separate ewmul) → recovers the full gap but surrenders fusion + a SRAM roundtrip = architecturally rkllm/baseline. The guaranteed fallback.

Verdict

Per-tensor int8 on the fused silu output has a hard ~21.9 ceiling that no in-stage trick escapes (one global scale can't serve both the critical outliers and the bulk). The CPU-fp32 sprinkle (per-channel/exact on the ~7 outlier layers, 12.31 PPL) is the practical on-device recovery; un-fuse is the full-recovery fallback.