Exp-2026-08 — int4 expert-parallel coalesce (ORK_I4_COALESCE): 1.6× on the matmuls, UNWIRED

Repo: ggml-ork (ggml-ork.cpp, NPU int4 BCHAIN MoE path). Status: WIP — real partial win at the matmul level, not yet wired into an end-to-end gain. Preserved: branch wip/moe-saga-2026-08-18

Patch (in this zip): parent-experiments-stashed.patch (coalesce block in ggml-ork.cpp) — git apply onto the banked tree to restore this experiment; the git-native copy is the snapshot branch above.

(llama.cpp-rockchip) — the coalesce path (~ggml-ork.cpp:3632–3633, co_ci[] dequant) + probe ork-driver/examples/test_moe_par.c. Knob: ORK_I4_COALESCE.

Hypothesis

The default per-expert BCHAIN colsplits each tiny expert matmul across 3 NPU cores → floor-bound (3-core == 1-core). Running WHOLE experts per core (expert-parallel, ork_mm_run_i4_experts) should beat colsplit by amortizing dispatch across a full expert.

Method

ORK_I4_COALESCE=1 routes routed rows through ork_mm_run_i4_experts (expert-parallel, same-domain experts grouped) instead of the per-expert colsplit; co_ci[q] feeds the shared dequant. Probe test_moe_par.c (24 experts, M=32) A/Bs nc=1 vs nc=3.

Result

• test_moe_par: nc=1 13455 µs → nc=3 8410 µs = 1.60× (≈53 % of ideal 3×; likely DRAM contention on 3 concurrent weight streams). Expert-parallel BEATS colsplit (which is 1× on tiny matmuls).
• BUT end-to-end unrealized: the NPU int4 expert path is dispatch-floored overall (~4M tiny programs), and the CPU NF4 route superseded experts-on-NPU as the prefill answer.

Conclusion

WIP. A genuine 1.6× at the matmul level, but (a) gated off, (b) not wired into the shipped path, and (c) the CPU NF4 route is the current prefill win, so the NPU-expert line is lower priority.

Follow-up (needed IF experts-on-NPU is revisited)

Wire coalesce as the default int4 MoE path (replace per-expert colsplit) + domain-aware grouping (~16 experts/domain, domain-switch serial between) — the ~3× prize is real driver+frontend work. Only worth it on wider-IOVA hardware or if the CPU route hits a ceiling. Resume from wip/moe-saga-2026-08-18

• test_moe_par.c.