Heterogeneous Serving & Scheduling (design)

A captured design for serving LLM inference across the NPU, GPU (Mali) and CPU of a Rockchip SoC, and for multiplexing multiple workloads. This is a roadmap/architecture note, grounded in the measured findings on Performance — not yet implemented. It defines what belongs in ork-driver vs the inference engine, and why.

The layering: mechanism vs policy

The single most important decision is where the smarts live:

• ork-driver (and the other per-device backends) = mechanism. "Run this matmul / this prefill chunk on ≤N cores, hardware-optimally." Each backend owns the device-specific knowledge — for ork-driver that's soc->cores, the CBUF size, the K≤10752 single-submit ceiling, the per-submit-overhead crossover, K-split-vs-full-K, N-tile distribution. The caller never embeds these constants; it gets portability across RK3576/RK3588/future parts for free.
• Inference engine = policy. What to run when and where: priority queue, chunked prefill, decode/prefill interleave, core budget per request, and device routing (NPU vs GPU vs CPU). The engine knows the runtime context (in-flight requests, latency targets, power mode) the backends can't.

Concretely: ork_mm_run(...) auto-selects core count + slice strategy from (M, K, N, soc, budget); the engine sets the budget/intent (e.g. ork_npu_set_core_budget, a latency/throughput mode) but never computes slice counts. Mirrors BLAS — you set the thread budget, the library picks the blocking.

The core principle: pair opposite bottlenecks

Prefill and decode have opposite bottlenecks, which is what makes heterogeneous splitting work (see Performance for the measurements):

phase	bottleneck	best engine
prefill	compute-bound (each weight reused across M positions); low bandwidth	NPU (INT8 MACs)
decode	memory-bound (every weight read once per token); little compute	NPU multi-core; GPU as overlap fallback

Running a compute-bound task and a memory-bound task concurrently makes them contend for different resources → real overlap. NPU-prefill + GPU-decode is exactly that good pairing.

The hard constraint: one memory controller

The NPU, Mali GPU and CPU share a single LPDDR controller (~20–30 GB/s aggregate on RK3588). So heterogeneous compute does not multiply bandwidth:

• ✅ compute-bound + memory-bound (NPU prefill + GPU decode): hit different limits → genuine overlap, using the spare bandwidth prefill leaves idle (telemetry: even librkllmrt leaves RAM BW ~48% avg / ~80% peak — there is headroom).
• ❌ two memory-bound tasks (two decodes on NPU + GPU): just split the same bus and contend — little gain.

Per-engine decode speed also differs sharply (NPU INT8 decode ≫ Mali decode), so GPU-decode is a "don't block the NPU" fallback — it accepts lower tok/s for that request to keep the NPU free for prefill, it is not a faster decode path.

Building blocks already proven

• NPU decode ≈ closed runtime: 3-core + per-core full-K single-submit = 11.0 tok/s, ~96% of librkllmrt (Qwen3-1.7B int8). See Performance.
• NPU multi-core + dynamic core count, clamped to soc->cores — the per-request core budget primitive a scheduler needs (ORK_NPU_MC).
• M-chunked prefill is free: prefill is just ork_mm_run with M = chunk_size; the driver M-tiles internally, giving a natural yield point between chunks.
• A Mali GPU compute beachhead exists: oRKLLM already runs the Eagle-3 draft head on Mali via hand-written SPIR-V shaders — a starting point for a GPU matmul backend (not yet a full int8 gemm).
• CPU: the existing NEON/fp32 ops backend for light/overflow work.

Chunked prefill + priority multiplexing (engine)

The vLLM-style technique: split a long prefill into fixed M-row chunks and interleave decode steps of other requests between them, so a long prompt doesn't starve interactive requests. Pure cross-request orchestration → engine scheduler, not the driver. On these SoCs the NPU's three independent cores also allow spatial partitioning (not just time-slicing): e.g. 2 cores on a prefill chunk + 1 core on a decode stream concurrently — something a single GPU can't do cleanly.

Caveats: the budget is small (3 NPU cores total), and oRKLLM today serializes inference (single active model, one request through the pool). Realizing this is a real change in oRKLLM's pool/scheduler (priority queue, chunk-boundary preemption, per-request core/device allocation). The payoff is largest for multi-model / latency-fairness serving, less so for single-stream throughput.

Roadmap (each stage independently useful)

1. NPU decode ≈ librkllmrt — ✅ done (11.0 tok/s).
2. NPU multi-core + core-budget primitive — ✅ done (ORK_NPU_MC, clamped to soc->cores).
3. Auto-tuner in ork-driver — fold today's ORK_NPU_MC/ORK_FULLK_DEC env knobs into an internal selector: core count by N-tile count, full-K single-submit when M small & K≤10752 & int8 & fits-IOVA, else K-split. Engine can only cap it via the budget.
4. Engine scheduler — priority queue, chunked prefill, decode/prefill interleave, per-request core budget. (oRKLLM pool work.)
5. GPU + CPU backends — add the Mali GPU as a second decode backend for overlap while the NPU runs prefill; CPU for overflow/ops. Device routing by bottleneck-complementarity + availability.

Phase-aware saturation architecture (the NORTH STAR — run the largest model on the 32 GB SBC)

The goal: saturate all four resources (NPU, Mali GPU, CPU, memory) with no idle bubbles, so the biggest possible model runs on one 32 GB RK3588. The shape of the answer is phase-aware — prefill and decode have opposite bottlenecks, so the right partitioning differs by phase. Grounded in the 2026-06-26 fill-bandwidth + mixed-quant findings (dma_buf Fill Bandwidth, NF4/imatrix Mixed Precision) and the 4 GiB IOVA cap (Layer-Streaming).

1. Memory = UMA, fit-by-partition

All three engines share one LPDDR pool (UMA). Key facts that shape the layout:

• The NPU runs int8 (same-precision MAC). So a w4a8 weight resident on the NPU occupies its dma_buf as int8 — the int4 footprint win is in the source/disk and in GPU/CPU consumption (they can run int4 natively), not in NPU residency. Plan NPU residency in int8 bytes.
• The 4 GiB IOVA cap is carved out of the 32 GB (the IOMMU addresses ≤4 GiB at once — hardware).
• FIT-BY-PARTITION beats streaming for residency. A ~35B mixed model (~20 GB) fits resident across engines: the int8-tier tensors (attn/q6_K) on the NPU (≤4 GiB), the int4-bulk (FFN/q4_K) native on CPU/GPU (~28 GB headroom), plus the KV cache. Streaming layers through the 4 GiB window is the fallback — and it's bandwidth-walled for decode (every weight read once/token), so it's a prefill/verify tool, not a decode path.

2. Prefill = compute-bound → NPU-centric

Prefill reuses each weight across M positions → compute-bound. The NPU's ~6 TOPS int8 ≫ CPU NEON or Mali int8, so:

• Don't partition the matmul across slower engines — layer latency = the slowest engine, so handing matmul tiles to the CPU/GPU starves the NPU and makes the layer slower, not faster.
• CPU/GPU FEED the NPU instead: prefetch + inflate the next slices into the resident dma_buf (the Tier 9 fill chain), run the non-matmul ops (RMSNorm/RoPE/softmax/SwiGLU), gather activations. They keep the NPU's critical path matmul-only.

3. Decode = bandwidth-bound → partition for additive bandwidth

Decode reads every weight once per token → memory-bound. Here partitioning does help, because splitting tensors across engines gives additive bandwidth (each engine pulls its share from its own access pattern):

• int4 tensors → GPU/CPU native zero-copy (they run int4 directly, no inflate); int8 tensors → NPU resident.
• Only the tiny activations cross engines (cheap); weights stay put.
• Amdahl-limited to ~1.5–2× by the serial dependency chain (layers are sequential; you can't overlap layer N+1's matmul before N finishes), not the naive sum-of-bandwidths.

4. Speculation unifies the two phases

EAGLE-3 maps cleanly onto the split:

• Verify (M=K, a multi-token forward) is compute-bound → NPU (the NPU's sweet spot, the run_stream_i8 ~3× target).
• Draft (M=1) → GPU.
• verify-on-NPU ∥ draft-on-GPU — opposite bottlenecks, genuine overlap, exactly the "pair opposite bottlenecks" principle above.

5. One oracle drives engine assignment

gguf_tier_map's per-tensor {int8|int4} tier (NF4/imatrix) doubles as the engine-assignment oracle: int8-tier → NPU resident, int4-tier → GPU/CPU native. The same map that allocates precision allocates the device.

6. Why you can't bit-pack two int4 into one int8 NPU MAC

Tempting (halve the NPU residency), but impossible: a8·b8 of nibble-packed operands produces cross-terms (a_hi·b_lo + a_lo·b_hi) that collide in the int32 accumulator, and no LUT recovers it (the accumulation already mixed the products). The NPU needs one int4 per int8 lane; the codebook/LUT lives in the inflate (Tier 9b), not in the MAC. So NPU residency is fundamentally int8-byte-bounded — which is exactly why §1 keeps the int4 bulk off the NPU.

Costs / open questions

• Weight formats per device: NPU wants int8 tiled ([Ntile][Ktile][32][32]), Mali wants its own layout → either duplicate weights in RAM (bandwidth/footprint cost) or convert. Duplication competes with the NPU dual-layout's IOVA budget.
• GPU int8 gemm is real work beyond the Eagle-3 shaders.
• Scheduling overhead must stay below the per-chunk/per-token budget to be worth it.
• Measure, don't assume: verify GPU-decode-while-NPU-prefills actually overlaps (shared-bus contention) before committing — sample RAM BW + per-engine load (tools/telemetry_sample.sh pattern) under the combined workload.