Hardware Chaining and Zero-Overhead Submissions

Date: 2026-06-23 Goal: Eliminate the inter-task interrupt overhead and kernel scheduling instability when submitting multiple matmuls.

Background

Previously, attempts to batch multiple matmuls into a single ioctl using task_number = S led to hard -EINVAL (22) errors or kernel driver timeouts (job timeout). The upstream rknpu driver's subcore_task and interrupt tracking logic for task_number > 1 is highly unstable. Attempts to bypass this by setting task_number = 1 and injecting manual PC_DATA_ADDR / PC_DATA_AMOUNT jumps inside the regcmd streams either corrupted the stack (due to incorrect jump sizing) or caused the NPU command parser to rush ahead of the MAC array execution, clobbering states.

The Breakthrough: CORE_CTRL 0x000f

While reverse-engineering how librknnrt safely avoids inter-task hangs, we discovered an undocumented behavior in the CORE_CTRL register (0x00810000). Historically, writing 0x000d to this register triggers the MAC array to begin computing. However, setting bit 1 (0x000f) triggers the MAC array and instructs the command parser to WAIT until the MAC array finishes before it fetches and executes the next command in the buffer.

Regcmd Concatenation

This wait behavior allows us to completely abandon the kernel's queueing logic and manual PC jumps. Instead, we concatenate all configurations into a single, contiguous regcmd buffer:

1. Synthesize the regcmd block for Task 0.
2. Append CORE_CTRL = 0x000f at the very end of Task 0's block (word 216).
3. Pad the remainder so the block is exactly 110 pairs (220 words).
4. Append Task 1 directly after Task 0 in the dma buffer.
5. Repeat for all intermediate tasks.
6. For the final task, synthesize its regcmd normally (108 pairs, no 0x000f override) since the kernel driver will append its own CORE_CTRL write to generate the final interrupt.

Results

We submit this massive buffer to the kernel as a single task (task_number = 1, regcfg_amount = (S - 1) * 110 + 108).

• Correctness: Output tensors match perfectly (test_chain yields C0 = 32, C1 = 64).
• Stability: Zero kernel driver -EINVAL issues, zero timeouts.
• Performance: Zero overhead. We eliminated all inter-task interrupt handling, which previously added ~130us per matmul. The NPU processes the entire chain completely natively in hardware.

Next Steps

Integrate this hardware chaining backend into llama.cpp inference pathways (particularly decode phases that issue multiple small M=1 matmuls per layer).