The CUDA teardown hang - SethKitchen/ThreeMojo GitHub Wiki
The CUDA teardown hang
Fixed upstream in MAX 26.6.0. A DeviceContext destroyed before its buffers used to hang the next context's first allocation on CUDA. The reproducer in docs/max-gpu-teardown-issue/repro.mojo now exits zero both ways round. The page stays because the defenses it describes stay: they are the right order to release in whatever the runtime tolerates.
Affects
MAX 26.5.0 with Mojo 1.0.0 (ed45d567), on WSL 2 Ubuntu over an NVIDIA TITAN RTX with driver 591.86. Not reproduced on macOS with Metal. Not reproduced on MAX 26.6.0 with Mojo 1.1.0 (8189361e), on the same machine.
What happens
Destroy a DeviceContext while DeviceBuffers it created are alive. Then create a fresh context and ask it for a buffer. The second enqueue_create_buffer never returns. The process sits at zero CPU with the GPU idle, and nothing times out.
Destroy the buffers first and the context afterwards, and the same sequence works any number of times.
var ctx = DeviceContext()
var buffer = ctx.enqueue_create_buffer[DType.uint8](256)
_ = ctx^ # context first: the next context's allocation hangs
_ = buffer^
Mojo destroys a struct's fields in declaration order. GpuRenderer declared its context before its buffers, so every renderer tore its context down first. The second renderer in a process hung in its constructor. The suite builds one renderer per test, so the first test passed and the second waited forever.
How it was found
Five variants, each capped at 45 seconds:
| Sequence | Result |
|---|---|
| Renderer A drawn, B created while A is alive | Works |
| Renderer A drawn, synchronized, destroyed, then B | Hangs |
| Renderer A created but never drawn, destroyed, then B | Hangs |
| Two bare contexts, no buffers | Works |
| Context and buffers, buffers destroyed first, twice | Works |
| Context and buffers, context destroyed first, twice | Hangs |
Drawing, map_to_host and synchronize are irrelevant. Only the order of the last two destructions matters.
A second condition
With the order fixed, the suite got forty-one tests further and hung again, after a test that drew, never read back, and dropped its renderer. One synchronize() in the destructor, before the buffers are released, fixed that too. A minimal reproduction of the second condition alone did not hang, so its trigger is narrower than the description. The wait is cheap and kept.
Reproduce it
timeout 45 .venv/bin/mojo run -I . docs/max-gpu-teardown-issue/repro.mojo buffers_first # exit 0
timeout 45 .venv/bin/mojo run -I . docs/max-gpu-teardown-issue/repro.mojo context_first # exit 124 on 26.5.0, 0 on 26.6.0
docs/ is outside every Makefile glob. The reproducer is never built by make check.
What the project does
GpuRendererdeclares its context last and releases it last.__deinit__waits for the queue, releases every buffer, and then releases the context.make test-gpuruns underGPU_BUDGET, 300 seconds by default, so a hang fails instead of looking slow.set_textureswaits for the queue before it releases the buffers it replaces. Replacing a resource and tearing the renderer down are two different lifetimes.