Exp 2026 07 01 Multi Domain Residence - oRKLLM/ork-driver GitHub Wiki

Experiment β€” Multi-domain residence (the proactive per-domain cap that makes >4 GiB reside)

Date: 2026-07-01 Β· Status: βœ… KEPT (default on, no knob) Β· Path: ork-driver + ggml-ork Commits: ork-driver bed4ccd (β†’ origin/main); llama.cpp-rockchip master 4f244e5c2 / diverged-history 1feaf0b7d

Result up front

7B-Q8_0 now stays fully resident on the NPU across 4 IOMMU domains (10.38 GiB incl. the full-K Bf), churn β‰ˆ 0, zero soft-resets, and prefill is ~1.45Γ— faster than our own baseline β€” correct, not fast-but-wrong.

Model / metric ork (this build) baseline rkllm verdict
7B pp256 (prefill) 57.98 36–40 73.89 +1.45Γ— vs baseline; 78% of rkllm
1.7B pp256 (prefill) 278 81 (pre-M1) 184 beats rkllm 1.51Γ—
1.7B decode 12.8 β€” 11.71 beats 1.09Γ—

Correctness: ORK PPL = 7.84 Β± 1.05 on wikitext (in the expected ~7–8 band for Qwen2.5-7B-Q8_0; a broken prefill would be NaN / >50), plus make test quant-RMS bit-exact and domain_correct bit-exact int32 across domains.

The mechanism (weights reside across domains; the active domain is swapped, zero-copy)

The rk_iommu v2 IOVA window is 32-bit (~4 GiB) per iommu_domain_id. ork-driver already exposes this: ork_npu_set_pack_domain places a weight's resident tiles in a domain; rknpu_submit stamps iommu_domain_id = w->domain; and dom_activate parks/restores per-domain scratch so a submit runs against the weight's domain. So a >4 GiB model resides across several domains and each submit zero-copy swaps the active domain β€” no data map/unmap, no streaming. tools/domain_probe.c confirms the hardware: 8.32 GB resident across 2 domains (each ~4.16 GiB, distinct IOVA), field-only and switch modes; domain_correct confirms it's bit-exact.

The bug: filling a domain to its EFAULT soft-resets + corrupts the IOMMU

Filling one domain to its ~4.16 GiB limit EFAULTs, which soft-resets the NPU and corrupts the IOMMU so the next domain fails on its first tile (measured: fill domain 0 to EFAULT β†’ domain 1 dies immediately; 18 soft-resets accrued). domain_probe avoids this only because it interleaves (both domains grow before either EFAULTs). A residence fill is sequential, so it hits the wall.

Fix β€” advance to the next domain BEFORE the EFAULT. ork_weight_domain() byte-balances weights and advances the domain cursor at a 3.0 GiB per-domain cap (below the measured ~4.16 GiB EFAULT point, with margin for the full-K Bf inflation + one-weight overshoot). No domain ever EFAULTs β†’ no soft-reset β†’ clean hand-off. This is a hardware limit, not a tuning knob. A retry-advance on EFAULT remains as a safety net. Result: 7B fills domains 0(2.90)+1(2.97)+2(2.97)+3(1.55 GiB), churn 3, 0 soft-resets.

Supporting changes

  • Residency budget = process RAM ceiling (sysconf RAM βˆ’ 1 GiB, override ORK_WCACHE_BUDGET_MB = the oRKLLM global memory limit). Large enough that a model which fits stays fully resident β†’ no eviction/churn.
  • n_domains auto ceiling (8, no knob); byte-balanced fill uses only as many as needed.
  • All-core pack: dequantβ†’int8 quant (ggml-ork) and the tile pass (ork_parallel_for in pack() / ork_stage_fill_i8) parallelized across every online core β€” the pack was otherwise bcreate-bound (synchronous ioctl), so this shortens the one-time residence fill.
  • .orkpack version token: ork_pack_format_version() = library MAJOR; stamped in the int4 blob header and the ggml-ork .orkpack footer (schema v3 + ork_fmt); a mismatch (tiling/quant change β‡’ major bump) rejects + regenerates the cache. Serve path loads pre-tiled bytes zero-copy (ork_mm_load_i8_import) into domains.

Dead ends corrected

  • The stream pool (ork_stream_pool) was the WRONG mechanism for practical models. It (RAM-hold + dma-buf map/unmap sliding window) only matters beyond 16Γ—4 GiB β‰ˆ 64 GiB resident. Left off (spool=NULL); the domain mechanism handles every real model. The add_i8_raw + self-managing-run work (ork-driver bed4ccd) stays as that >64 GiB fallback.
  • "Domains share one 4 GiB pool" was a repo-drift artifact. The board's ~/ork-driver had drifted 62 commits behind origin/main; single-file npu.c rsyncs onto the stale tree (Frankenstein) produced the false failure. Reconciling to origin/main + rebuilding is what surfaced the real (working) behavior. Lesson: never trust a board result until the checkout is confirmed at origin/main.

Open follow-ups

  • churn = 3 (target 0): 3 weights re-packed post-load β€” likely a domain that filled slightly early; minor, worth pinning.
  • 7B prefill is 78% of rkllm β€” headroom remains (prefill is partly CPU-bound: quant + resolve).
  • CPU-backend perplexity (ORK_OFF=1) as an exact-match reference is impractical (hammers all cores, starves SSH); the in-range NPU PPL + bit-exact make test/domain_correct are the correctness gate.