Exp 2026 08 20 SDP and Int4 Envelopes - oRKLLM/ork-driver GitHub Wiki

SDP and int4 envelopes: closing the audit, and four bounds that were never measured

Companion to Exp 2026-08-20 fp16 M Envelope and the CBUF Bank Split. That entry mapped the fp16 matmul envelope; this one finishes the sweep across the SDP family and int4, so every precision now runs on a measured bound rather than an inherited one.

Landed: ork-driver main b351c14 (int16), 3b1eb96 (int8+fp16 SDP), 6033334 + bfefc64 (int4). Probes: tools/re/i16_mcap_probe.c, tools/re/sdp_mcap_probe.c, tools/re/i4_hcap_probe.c.

TL;DR

path was measured how it was wrong
int16 activations M<=8192 8176 8192 = a register FIELD WIDTH; 8184 TIMES OUT
int8 SDP M<=8192 8176 same constant, same fault; 8176 is EXACT (8177 fails)
fp16 SDP M<=8192 8188 12 rows further than int8 β€” needs its own constant
int4 batch H 16384/K, capped 16 CEIL(16384/K), capped 64 the 16 was invented; and floor≠ceil at non-dividing K

Four bounds, four different ways of being inferred rather than measured. Two were admitting fault-generating shapes; two were throttling throughput.

The SDP family: one constant, 21 copies, never measured

M <= 8192 appeared at 21 sites across i16/act.c, i16/chain.c, i16/probe.c, i8/run.c, i8/probe_sdp.c, i8/probe_prof.c, f16/perchan.c, f16/replay.c β€” every op that routes through orki_set_mul_geom. Its provenance: RK_DPU_DATA_CUBE_WIDTH (0x4030) has mask 0x1fff, so M-1 <= 8191. Someone read the field width as a capability.

A field width bounds what you can ENCODE, never what the hardware COMPLETES. Measured:

int16 act (exp)        : 8176 OK Β· 8184 rc=-1, errno=110 (submit TIMEOUT) + self-healing reset
int8  SDP (ewmul, add) : 8176 OK Β· 8177/8180/8182/8183 ALL TIMEOUT   -> 8176 is EXACT
fp16  SDP (ewmul, add) : 8188 OK Β· 8189 MISCOMPUTE

So the old guard was not merely loose β€” it admitted a shape that faults the NPU rather than refusing it cleanly. Two ops per dtype were probed so a per-op (rather than per-dtype) ceiling would have shown up; within each dtype they agree exactly.

fp16 reaches 12 rows further than int8/int16, so the fix uses two constants (ORK_SDP_MAXM 8176, ORK_SDP_MAXM_F16 8188) rather than one conservative bound β€” capping fp16 at 8176 would leave real headroom unused for no reason.

fp16 SDP frays; int8 cliffs

fp16's top end is NON-MONOTONIC:

8188 OK Β· 8189 BAD Β· **8190 OK** Β· 8191 BAD Β· 8192 BAD

8190 works but is an isolated island, so the usable bound is the contiguous maximum 8188 β€” a guard at 8190 would admit the broken 8189. Third instance of this shape today (fp16 K=128 matmul, fp16 SDP, and the int4 sweep below), and the reason every ceiling here is found by an upward scan.

int8/int16, by contrast, cliff cleanly: 8176 works, 8177 does not.

int4: a throughput bound, not a correctness one

int4 is structurally different. orki_i4_run_bchain_db already tiles M into H-row batches and refuses H<2, so there was never an unbounded-M miscompute risk. But H is rows per weight stream β€” it is directly the int4 prefill's weight-reuse factor β€” and the line computing it carried two constants, only one of which was real:

int H = 16384/K;  if (H > 16) H = 16;

if(H>16) H=16 was invented. It bound whenever K<1024, costing 2Γ— the weight streams at K=512 (H 16β†’32) and 4Γ— at K=256 (H 16β†’64).

And 16384/K was subtly wrong too β€” see the methodology section; the rule is the ceil.

Measured rule: H_max = CEIL(16384/K)

K=256  -> 64 OK, 65 bad      K=512  -> 32 OK, 33 bad      K=768  -> 22 OK, 23 bad
K=1536 -> 11 OK, 12 bad      K=2048 ->  8 OK,  9 bad      K=4096 ->  4 OK,  5 bad

Six K, every boundary bracketed at integer resolution. Note this is 16384 int4 ELEMENTS = 8192 bytes, only a QUARTER of the 32 KB CBUF bank the other dtypes get β€” the bank model from the companion entry does not generalise to int4, and predicting 65536 from it was wrong.

Method note: correctness was checked by checksum comparison across processes, not against a reference model. H only changes how M is tiled, so any valid H must reproduce a known-good H bit-for-bit β€” which sidesteps having to model int4 packing and scales entirely. Invalid H values each produce a distinct checksum, consistent with a tiling change rather than one shared failure.

A model that looked right and was not

When H=22 first appeared to exceed 16384/768, the obvious explanation was a shared budget: Wb = (131072/K) & ~63 wastes weight allocation at non-pow2 K (at K=768 it rounds 170 down to 128), and (Wb + H)*K = 147456 held exactly at K=2048 and K=512. That predicted H<=64 at K=768.

Measured: 32, 48 and 64 are all wrong at K=768. The budget is per-activation; the coincidence at the pow2 K was just Wb*K landing exactly on its own budget. Recorded because the model is seductive and someone will re-derive it.

Methodology: the four ways an inferred bound goes wrong

Every bound corrected in this campaign was inferred, and each failed differently:

  1. Cross-dtype transplant β€” fp16 inherited int8's mg_max*64.
  2. Unit confusion β€” fp16's sched=0 cap was an int8 element count applied at 2 bytes/element, so 2Γ— too loose.
  3. Field width as capability β€” the SDP 8192, above.
  4. Ladder resolution β€” the int4 floor vs ceil, below.

Ladder resolution: the error that recurs

The int4 sweep first used only powers of two {2,4,8,16,32,64} and concluded H = 16384/K, floor. Every K probed (256/512/2048/4096) happens to divide 16384, so floor and ceil coincide and the data looked conclusive. Probing between the rungs, and at non-pow2 K, showed K=768 β†’ 22 (not 21) and K=1536 β†’ 11 (not 10): plain integer division silently loses a row at every non-dividing K β€” including K=768, a real int4 down-proj shape.

This is the same error that hid the fp16 352 ceiling behind a 320/384 probe (see the companion entry). Twice in one campaign, in different code, by different people. A ladder proves nothing between its rungs; bracket every boundary at the resolution you intend to ship.

Board-ops earned here

  • Every over-ceiling int8/int16 SDP submit wedges core 0 at 100% and costs a reboot. fp16 and int4 miscompute instead, which is survivable. So: probe only points expected to PASS, bounded below an already-known-bad value, and add an op filter so one dtype's known-bad list never runs against another's ops.
  • Do not build a large reference in tiny chunks. An 8183-row reference in 8-row chunks is ~1023 allocate/submit/free cycles per op; that fragments IOVA until bcreate fails ([iova@fail], total live=11 MiB) and hard-wedges the box. Two power cycles were spent on this before the chunk size was raised to 1024 (~8 allocations). The failure was in the tooling, not the hardware.
  • Start make test only from a guard-verified-idle NPU. Two suite runs stalled at test_spine purely because they began on an NPU left busy by the preceding probes; test_spine passes standalone in ~4 ms. That looked exactly like a regression and was not.
  • Make the cap itself overridable. ORK_SDP_MAXM is -D-overridable and ORK_I4_H is an env knob, because a probe that cannot exceed the guard only ever measures the guard β€” the first int8 boundary run returned rc=-2 for every point for precisely this reason.
  • Rebuild the probe target, not just all. make does not relink tools/re/* probes; one verification ran against a stale binary and reported a pre-fix value.

What this implies for OTHER assumptions β€” a probe list

The int4 result generalises into two searchable code shapes: (A) a magic constant clamping a computed value, and (B) a computed capacity rounded down (by integer division or a mask). A sweep of src/npu/ for both turns up the following, ranked.

1. Wb = (131072/K) & ~63 β€” same line, same class, UNPROBED. Do this one first.

Sitting one line below the H bug in orki_i4_run_bchain_db. Wb is the N-tile width per weight stream, so it is to the N axis what H is to the M axis. It is floor-divided and masked down:

K=768:  131072/768 = 170.67 -> floor 170 -> &~63 -> **128**       (42 columns, 25%, discarded)
K=1536: 131072/1536 = 85.33 -> floor  85 -> &~63 ->  **64**       (21 columns, 25%, discarded)

Neither the 131072 nor the 64-column alignment was measured. If the real alignment is 16 (&~15), K=768 gets Wb=160 (+25% N per weight stream) and K=1536 gets Wb=80. Exactly the shape of the H fix, on exactly the same production shape (K=768 int4 down-proj). Probe it the same way: Wb only changes N-tiling, so a valid Wb must reproduce a known-good one bit-for-bit β€” a checksum sweep needs no reference model. Note the intermediate "shared budget" model above was wrong precisely because this rounding wastes weight budget, so measuring Wb may also explain why int4 tolerates ceil on H.

2. int4 if(H>64) H=64 β€” a boundary this campaign INTRODUCED

That clamp is where measurement stopped, not where hardware does. CEIL(16384/K) exceeds 64 for K<256 (K=128 -> 128, K=64 -> 256). Unprobed, and honest about it in the code comment β€” but it is now a known-unmeasured bound of exactly the kind this campaign exists to eliminate.

3. if (CBUF > 32768) CBUF = 32768 β€” fp16, three sites, likely vestigial

f16/run.c:171, f16/stream.c:78, f16/stream.c:169. RK3588's cbuf_elems is 57344, so this discards 43% of it on the grounds that fp16 is "validated to the 32768 tile". Since the fp16 M-tile is now orki_f16_mcap (which does not consult CBUF), this only feeds R -> 0x1010, which the int8 work already showed is a perf hint with no correctness effect. Probably dead weight rather than a lever, but it is an unmeasured constant restricting a real hardware resource.

Already checked and FINE (recorded so nobody re-probes them)

  • fp16 sched=0, 16384/K β€” uses FLOOR, and it is bracketed at four NON-dividing K: 384->42 (43 bad), 640->25 (26 bad), 768->21 (22 bad), 1536->10 (11 bad).
  • fp16 sched=1, 180224/K β€” the floor/ceil question is moot: sched=1 requires pow2 K in [128,2048), and 180224 divides evenly by all of 128/256/512/1024.
  • int8 mg_max = (base-0x1b)/slope + 1 β€” floor, bracketed at the non-dividing K=1536 (192 OK, 193 bad).

The cross-dtype inconsistency this exposes

fp16 floors and int4 ceils on the same nominal 16384-element budget, both measured, both bracketed:

fp16 sched=0, K=768: M<=21   (22 MISCOMPUTES)   = floor(16384/768)
int4 BCHAIN,  K=768: H<=22   (23 MISCOMPUTES)   = ceil (16384/768)

Since the arithmetic is identical, the "16384" cannot be the same quantity in both β€” the int4 path buys one extra row somewhere (a partial group rounded up, or a coarser allocation unit). Unexplained. Practical consequence: do not port either rounding to the other dtype. That assumption β€” that a bound measured on one datapath transfers to another β€” is the single most common defect this whole campaign corrected.

END-TO-END BENCHMARK OF THE 0x1040 LEVER: no measurable gain (2026-08-21)

The ORK_I4_DBNK=4 lever (4x the activation banks => 4x rows per weight stream) was measured end-to-end. It works mechanically and delivers nothing at the model level.

mamba2-130m-f16, ORK_QUANT=4 ORK_HADAMARD=1, P=256, fresh boot per arm, governors pinned
rep 1 (separate boots): default 394.77 -> DBNK=4 397.05 tok/s   (+0.6%)
rep 2 (paired, 1 boot): default 390.45 -> DBNK=4 391.03 tok/s   (+0.15%)
BCHAIN engaged both arms (48 dispatches); H went 11 -> 43 as designed, bit-exact.

Both deltas are noise. The 4x weight-stream reduction that is worth ~10% at the matmul level does NOT survive contact with a real model here β€” mamba2-130m is SSM-dominated and only issues 48 int4 matmuls, so BCHAIN weight streaming is nowhere near the bottleneck. => Keep the default. Do not enable DBNK rebalancing without a model where int4 matmuls actually dominate prefill.

The proper test would be a dense transformer under W4A4, but qwen2.5-7b does not fit: packed int4 it needs ~3.5 GB against the ~3.9 GB per-domain 32-bit IOVA ceiling (CREATE FAIL errno=14 size=3584KB dom=5 dom_iova=3503MB ceil=3900MB). That is the real gate on validating any int4 prefill lever.

Two traps that made this benchmark hard to get right

1. ORK_QUANT=4 does NOT engage native int4 β€” by design. ggml-ork.cpp's own header: "int4: compact int4 STORAGE + W8A8-inflate compute … int4 weights inflate int4->int8 and run on the robust int8 kernel. The win is the .orkpack STORAGE (~3.4x smaller), not the compute. Native W4A4 … is OPT-IN via ORK_MIXED_W4A4 / ORK_HADAMARD." So ORK_QUANT=4 alone never calls ork_i4_mm_run (ORK_I4_DIAG prints nothing, [bch] count 0) and any int4-lever A/B under it measures the int8 kernel. Native W4A4 needs ORK_QUANT=4 ORK_HADAMARD=1.

2. The fork VENDORS its own ork-driver copy, and it was ~4 months stale. ork_bench builds against llama.cpp/ggml/src/ggml-ork/ork-driver/, whose git HEAD read v1.0.14 while its files were stock ork-driver at 39740e6 (verified by md5 against that commit β€” no fork-specific patches, the "modified" flags are just the stale HEAD). Every end-to-end number is therefore the VENDORED driver, not ~/ork-driver. The tell is cheap: ORK_BCH_DEBUG=1 printed H=10 at K=1536 where the current rule gives 11. Before ANY end-to-end claim about a driver change, sync the fork's copy and confirm a version-distinguishing value in the trace. After syncing, H=11 (default) and H=43 (DBNK=4) confirmed the new driver was live.

Board-ops addendum

A long probing session degrades the DMA/IOVA allocator until bcreate fails for buffers as small as 512 KB with dom_iova=30-35MB against a 3900MB ceiling β€” this broke make test and every model load. sync; drop_caches does NOT clear it; a reboot does. Distinguish it from genuine exhaustion by the dom_iova figure: ~30MB used = degraded allocator, ~3500MB used = the real 4 GiB wall. (AGENTS Tier 14 D suggests caches first; caches were not sufficient in any case here.)

Cross-references