MoE Routing Skew — sizing the hot-expert cache (Milestone M3-skew) ✅ hot-cache VIABLE

Date: 2026-06-26 Status: ✅ VIABLE — measured routing skew on both MoE deploy targets; a per-layer hot-expert cache is worth building. Operating point ~3–3.5 GB cache (top ~26–28/256, ~10–11% experts/layer) → ~50% of decode expert lookups NPU-resident on the Qwen3.6-35B-A3B target, within the 4 GiB window alongside the pinned backbone. Confirms the Branch-A decode partition. Tool/data: llama-moe-skew (llama.cpp harness, ~/llama.cpp/examples/moe-skew); /tmp/skew_{lfm2,qwen36}.csv on box 10.3.0.241 (CPU-only, no NPU).

Question

The 4 GiB IOVA wall makes resident-all-experts impossible for any real MoE. The M3 plan is a hot-expert cache — keep the most-routed experts NPU-resident, stream the cold tail. That only works if routing is skewed (a small expert set takes most of the traffic). Is it, on the actual deploy targets? And how big a cache buys what hit-rate?

Method

• Harness llama-moe-skew reads the contiguous ffn_moe_argsort tensor per layer to recover the top-k expert selections, accumulating per-expert, per-layer routing counts.
• ⚠️ GOTCHA (caught): the ffn_moe_topk view is non-contiguous — reading it gives a FALSE "uniform" result. Use the contiguous ffn_moe_argsort tensor instead.
• Profiled on the UD-Q4 GGUF: routing skew is precision-invariant, and UD/k-quant keeps the router gate high-precision, so q4 routing ≈ f16. (An earlier f16-convert path was wasted effort — q4 is fine for routing measurement.)
• Box CPU-only, no NPU (this is a routing-statistics measurement, not a perf run).
• Skew quantified by Gini coefficient (0 = uniform).

Result — both models skewed; the 35B target strongly so

model	experts	top-k	Gini (0=uniform)
LFM2-8B-A1B	32	4	0.44
Qwen3.6-35B-A3B (deploy target)	256	8	0.65

• Skew RISES with layer depth — deeper/majority layers specialize harder → good for caching (the layers that dominate the token budget are the most cacheable).
• Per-expert int8 is only 3.15 MB on the 35B (FFN dim 512), so a useful cache is small.

Qwen3.6-35B-A3B — hit-rate vs cache size (pin top-K experts per layer)

pin per layer	% of 256	cache size	decode hit-rate
13	5%	1.6 GB	33%
26	10%	3.3 GB	48%
51	20%	6.4 GB	67%
77	30%	9.7 GB	79%
128	50%	16 GB	92%

Verdict

✅ Hot-expert cache VIABLE. Operating point ~3–3.5 GB cache (top ~26–28/256, ~10–11% experts/layer) → ~50% of decode expert lookups NPU-resident on the 35B target — alongside the ~0.5–1 GB pinned backbone, fitting inside the 4 GiB window. The cold ~50% goes to CPU/GPU. Pushing >50% hit (≥6 GB) exceeds the window → stream the cold tail (M2). Policy = per-LAYER top-K (a global pin underperforms — skew is layer-specific). This confirms the Branch-A decode partition: NPU runs the hot half resident, the engine partition handles the tail.

Follow-on

• M2 — stream the cold expert tail through the resident window on the prefill path (Tier 9 fill chain).
• Wire the per-layer top-K cache (LRU within the budget) into the MoE expert-persist path so cached experts reload from .orkpack.