Fuzzing and Dynamic Tracing - oRKLLM/ork-driver GitHub Wiki

Structure-Aware Fuzzing & Dynamic Trace Instrumentation

To optimize complex tensor layouts, map register behaviors, and safely extend the ork-driver runtime (e.g., to support multi-layer hidden states or nested-tiled formats), we employ two combined validation methodologies: Dynamic Trace Instrumentation and Structure-Aware Fuzzing/Differential Testing.

These techniques enable us to observe exactly how the closed-source librknnrt.so runtime translates model weights and execution logic into raw ioctl transactions, and to systematically probe undocumented NPU behavior without bricking the system.

1. Dynamic Trace Instrumentation (The Observer)

Rather than treating the proprietary librknnrt.so as an impenetrable black box, we intercept its transactions at the boundary of the rknpu kernel driver.

A. Core Mechanism (Syscall Hooking)

By using an LD_PRELOAD interposer (such as tools/regcmd_capture.c), we intercept standard system library functions before they reach the OS:

  1. mmap / mmap64: Captured to map the virtual addresses assigned by the userspace allocator to physical memory offsets.
  2. ioctl: Intercepted specifically on the following DRM commands:
    • DRM_IOCTL_RKNPU_MEM_CREATE: Registers buffer sizes, DMA addresses (dma_addr), and object handles.
    • DRM_IOCTL_RKNPU_MEM_MAP: Captures memory offset mapping structures.
    • DRM_IOCTL_RKNPU_SUBMIT: Intercepts the raw hardware jobs containing rknpu_task descriptors, register write sequences, subcore masks, and execution timeouts.

B. Output & Diagnostic Telemetry

When active, the tracer dumps:

  • Task Descriptors: Flags, interrupts, register configuration counts (regcfg_amount), offsets, and pointer addresses.
  • Register Configuration Blocks: Clean hex logs of every (offset, value) pair programmed into the NPU's sub-blocks (0x0201 CNA, 0x0801 DPU, 0x1001 PPU).
  • Zero-Copy Verification: Tracking shared buffer pools to verify that input Hidden States (A) and Output Activations (C) avoid redundant host-side memory copying.

2. Structure-Aware Fuzzing & Differential Testing (The Probe)

Naive random-byte fuzzing is highly inefficient on structured binaries containing magic bytes, offsets, and checksums. To discover undocumented registers or optimize unknown layout shapes, we use structure-aware mutation.

A. The Schema-Driven Mutator

Instead of mutating arbitrary bits, our fuzzer tools (e.g., tools/i4_multim_fuzz.c or custom scripts) are built on a target register template:

  1. Protected Areas: We mark essential register offsets (e.g., DMA addresses, task masks, configuration lengths) as non-fuzzable/essential so the parser does not abort early.
  2. Target Mutation Zones: We select empty, unmapped, or unknown register locations and systematically sweep values (e.g., across block configurations, stride settings, or PC control flags).
  3. Automated Verification: On each iteration, the tool:
    • Writes the mutated instruction stream.
    • Submits the task via RKNPU_SUBMIT.
    • Checks the completion status (detecting hardware timeouts, page faults, or execution hangs).
    • Validates NPU outputs against a bit-exact CPU reference.

B. Defensive Blacklisting & Progress State

To ensure we do not infinitely wedge the hardware or loop on a fatal kernel state:

  • Progress Log: The current target block, register, and value-index are persisted to disk (fuzz_progress.txt) on every step.
  • Auto-Blacklisting: If a register write causes a hardware wedge, kernel oops, or timeout, it is written to fuzz_blacklist.txt on reboot. The fuzzer skips these registers on subsequent runs.
graph TD
    A[Start Fuzzer Iteration] --> B[Read Blacklist & Progress]
    B --> C[Generate Structure-Aware Regcmd]
    C --> D[Submit Job to RKNPU]
    D -->|Hardware Timeout / Wedge| E[Trigger Power Cycle]
    D -->|Success| F[Validate Output vs CPU]
    F -->|Bit-Exact Match| G[Log Optimal Register Config]
    F -->|Mismatch / Corrupted| H[Advance Value]
    E --> I[Save to Blacklist on Reboot]
    I --> B

3. The Power Cycle Fail-Safe

Fuzzing raw registers can cause the RKNPU to enter an unrecoverable "soft-reset timeout" state, effectively wedging the NPU core and locking up driver dispatches.

When the board becomes unresponsive, we use the Home Assistant MCP tool to cycle the physical smart plug connected to the board (10.3.0.236):

# Example Home Assistant command to toggle the Rock5B power line
mcp_home_assistant_hassturnoff --entity_id switch.rock5b_plug
sleep 5
mcp_home_assistant_hassturnon --entity_id switch.rock5b_plug

Upon boot, the system's cron script restarts the fuzzer, which reads the last progress index and blacklists the offending register, resuming execution seamlessly.

4. Optimization Workflows Enabled by this Toolchain

A. Tiling and Layout Verification

Using identity-matrix probing, we mutate input tensor indices and trace where they map to the NPU's internal output buffers. This was how the complex [Ntile][Ktile][16][32] nested weight format was reversed and solved.

B. Unlocking NPU Features (e.g., INT4 Multi-M)

By targeting unused registers (such as the DPU stride and PC sequencing arrays), we discovered that the INT4 mode computes multiple rows natively with a stride of 2 * m, removing the need for slow row-by-row CPU-driven submissions and boosting prefill speeds.

5. Prior Art References

For a deeper dive into the theoretical basis and industrial applications of these testing paradigms, see Prior Art on Model and Register Fuzzing.