Exp-2026-10-11 — ggml-ork disowned every weight whenever mmap was on

Board: RK3588 10.3.0.236, kernel 6.1.115-vendor-rk35xx-fence, governors verified (DDR 2112 MHz / A76 2400 MHz, both performance), NPU guard held, 0 faults. Stack: llama.cpp-rockchip diverged-history @ 35e76dbd0, ork-driver 1.0.168, built in ~/llama-pinned with -DGGML_ORK=ON -DGGML_OPENMP=ON, Release. Model: Qwen3-1.7B-BF16.gguf (arch qwen3), ork_bench … 128 64 128, 2 reps, warm.

Result

	before	after
tensors moved to the CPU backend	311	115
ORK_Weights model buffer size	absent	2688.46 MiB
bytes copied into anonymous RAM	—	0 (all mmap)
prefill tok/s	165.85 / 164.57	166.21 / 165.51
decode tok/s	1.10 / 1.10	1.10 / 1.10
ork_ppl 4×512, ubatch 512	1.6587 (NLL 0.506032)	1.6587 (NLL 0.506032)
ork residence	196/196 weights, 2.30 GiB, 3 domains	identical

Ownership moved; nothing else did. PPL is identical to six decimals and prefill is flat inside the run-to-run spread (the two before reps differ by more than the before→after delta).

Mechanism

The device iface set .get_host_buffer_type to the same function as .get_buffer_type. llama_model_loader::create_tensor rewrites any buft equal to the device's host buffer type to CPU when mmap is on — a rule that exists so a pinned-host staging buffer is never used as model storage. Because ork's real buft compared equal to its own host buft, that rule fired on ORK_Weights itself and handed every weight to the CPU backend.

The iface also contradicted the device's own get_props, which already reports host_buffer = false. The loader trusts the iface, not the props, so the props being right changed nothing. NULL is the honest answer and ggml_backend_dev_host_buffer_type() already handles it.

Compute was never stranded — ownership was. ork runs buffer-less via offload_op, so matmuls still reached the NPU (196/196 weights resident before the fix). What was lost is the scheduler's weight-locality signal.

The two halves have to land together

Dropping the alias alone makes things worse. Weights then arrive through ggml_backend_ork_buffer_type_alloc_buffer, which is posix_memalign plus a set_tensor memcpy of the whole model — and over a stub source (whose packed tensors are holes) that copies gigabytes of zeros into anonymous RAM.

So buffer_from_host_ptr had to change at the same time. It returned ggml_backend_cpu_buffer_from_ptr, whose buft reports CPU — so the zero-copy mmap path would have handed the weights straight back to the CPU backend anyway. It now returns a buffer carrying ORK_Weights over llama.cpp's mmap window, with a free_buffer of NULL because the mapping is borrowed and freeing it would unmap the model out from under every other view of it.

Make ownership observable, and name the mechanism

Ownership used to be inferable only from an ORK_Weights model buffer size line not being printed — a signal that is a thing not happening, which is the kind nobody notices. There is now a positive line, and it names how the bytes arrived, because "owned via a multi-GiB memcpy" and "owned via an mmap window" are very different outcomes that would otherwise print the same word:

[ORK BUFTYPE] weights on ORK_Weights via mmap (zero-copy, borrowed): 2688.46 MiB
              (cumulative: mapped 1/2688.46 MiB, copied 0/0.00 MiB)

copied 0/0.00 MiB is what makes "the memcpy trap was avoided" a measurement rather than an argument.

Caveats

• CPU_Mapped stays at 3281.97 MiB, so the two buffers sum to more than the 3.21 GiB model. Both are mmap windows over the same file (llama.cpp maps a per-context first..last range and they overlap), so this is not double RAM — free showed no anonymous growth. Inferred, not isolated.
• Validated on one model (1.7B BF16, arch qwen3). int4/W4A4 and MoE are untested under this change, and it alters routing for every model.
• Decode at 1.10 t/s is BF16-on-CPU and is a weak signal; prefill is the routing-sensitive axis.
• ork_ppl 1.6587 is not a quality number — the fixture is a repeated sentence, so it is predictable by construction. It is used here only as an equality check between two builds.

Side finding — the backend's only automatic correctness gate is dead

tests/test-ork-cache.cpp fails on pristine origin/diverged-history, verified by checking out the unmodified file, rebuilding and re-running. It is not a regression from this change.

Run 1 (X = 1.0): dst[0] = 0.000000      Expected Run 1: 64.000000
Run 2 (X = 2.0): dst[0] = 0.000000      Expected Run 2: 128.000000

What a 20-minute time-boxed look established:

• 0.0 twice is not the stale-activation-cache bug the test was written to catch — that would be 64.0 twice.
• The op is not being declined and skipped: ORK_VERBOSE shows START mul_mat_i8 on both runs and one packed weight … K=128, N=128. The backend reaches and executes the matmul path.
• ORK_MINM=1 does not change the result, so the M=1 decode gate is not the cause.
• ggml_backend_ork_graph_compute does return GGML_STATUS_FAILED when mul_mat_i8 fails — but the test never checks the return value, so a backend failure and a wrong answer are indistinguishable from its output. That is a defect in the test whatever else is true.
• The backend is not miscomputing matmuls generally on this binary: ork_ppl is bit-identical across the ownership change, 196/196 weights resolve, generation is coherent.

Not established: why dst stays zero. Leading hypothesis, unproven: the M=1 / F32-source / no-pack path inside mul_mat_i8 returns false or delegates to a route that needs a format this synthetic tensor does not have, leaving dst untouched. Distinguishing that needs an M>1 variant of the test or instrumenting mul_mat_i8's exit.

Read this as "the gate is dead", not "the backend miscomputes a 128×128 matmul". CI never runs it (LLAMA_BUILD_TESTS=OFF), which is why it has been red unnoticed. It also provokes RKNPU: soft reset, num: 6 — on the pristine build too; the driver self-heals.

What to use as a gate instead

test-backend-ops support -b ORK enumerates ORK (Backend ORK: OK, 2/2 backends passed) and is safe: support mode only calls supports_op, issues no submits, so there is no wedge risk. It gives zero MUL_MAT coverage — 0 of 1638 cases claimed, since ork's K%32 / N%16 / M≥8 and source-type gates decline them all — so it cannot gate matmul correctness. It is, however, a cheap and safe detector of supports_op changes, which is what any future N-cap or pack-policy work needs. Baseline recorded: 0/1638.

test-backend-ops in test mode (which executes ops) was deliberately not run: firing thousands of generated shapes at the NPU unsupervised, with a test already provoking a soft reset, risks a wedge whose recovery needs a physically-present operator.