AI Infra Debug - rFronteddu/general_wiki GitHub Wiki

A model inference endpoint is slow

A batch job uses too much GPU memory

Strategy

Baseline measurements: “Before changing anything, I measured X, Y, Z. The bottleneck appears to be A.”
Concrete improvement: For example: reduced latency, improved GPU utilization, removed a memory bottleneck, improved batching, reduced redundant work, or found a config issue.
Final writeup: 1–2 page writeup

Report

Summary:

Investigated [system/component].
Found bottleneck in [area].
Implemented/tested [change].
Result: [metric before] -> [metric after].
Remaining opportunities: [ranked list].

Method:

Baseline setup
Metrics collected
Tools used
Assumptions

Findings:

Changes made:

Risks / tradeoffs:

Recommended next steps:

Strategy

Identify Metrics:

What matters most:
- latency
- throughput
- GPU utilization
- memory
- cost
- reliability
- developer simplicity
What is the current behavior, and what would count as a useful improvement

Useful Commands:

watch -n 1 nvidia-smi: Shows GPU memory, utilization, power, running processes.
htop: Shows CPU usage.
iostat -x 1: Shows disk bottlenecks, if available.
df -h: Checks disk space.
du -sh .: Checks directory size.

For Python timing

import time

t0 = time.perf_counter()
# work here
t1 = time.perf_counter()

print(f"Elapsed: {t1 - t0:.3f}s")

For GPU timing, use CUDA synchronization

Without torch.cuda.synchronize(), GPU timing can lie because CUDA work is asynchronous.

import time
import torch

torch.cuda.synchronize()
t0 = time.perf_counter()

# GPU work here

torch.cuda.synchronize()
t1 = time.perf_counter()

print(f"GPU elapsed: {t1 - t0:.3f}s")

A data pipeline is bottlenecked

The GPU is waiting because data loading, image decoding, preprocessing, network fetch, or disk reads are too slow.

How you recognize it

watch -n 1 nvidia-smi

If GPU utilization jumps between 0% and 60% or stays low, but CPU is busy, the GPU may be starving.

Then time the stages:

for batch in loader:
    t0 = time.perf_counter()

    # batch already loaded here
    t1 = time.perf_counter()

    batch = batch.to("cuda")
    t2 = time.perf_counter()

    with torch.inference_mode():
        output = model(batch)

    torch.cuda.synchronize()
    t3 = time.perf_counter()

    print({
        "to_gpu": t2 - t1,
        "model": t3 - t2,
    })

Measure loading time:

data_iter = iter(loader)

for _ in range(10):
    t0 = time.perf_counter()
    batch = next(data_iter)
    t1 = time.perf_counter()

    print("load batch:", t1 - t0)

Common causes

num_workers is too low.
Data is read from slow remote storage.
Images/videos are decoded on CPU too slowly.
Transforms are expensive.
Batch size is too small.
Data is not prefetched.
Data is copied CPU -> GPU inefficiently.
Dataset has many tiny files.
Safe fixes

If using PyTorch DataLoader, inspect:

DataLoader(
    dataset,
    batch_size=...,
    num_workers=...,
    pin_memory=True,
    persistent_workers=True,
    prefetch_factor=2,
)

Try increasing workers (too many workers can hurt too, test 4 then 8):

num_workers=4 or 8

Use pinned memory for GPU transfer:

loader = DataLoader(dataset, pin_memory=True)

Then:

batch = batch.to("cuda", non_blocking=True)

Cache expensive preprocessing if repeated.

Bad:

for epoch in range(10):
    image = decode_and_resize(path)

Better:

decode/resize once
save preprocessed form
reuse it

If there are many tiny files they may cause metadata/read overhead. A packed format or local cache could improve throughput.

Baseline:

average data loading time: X ms/batch
average model time: Y ms/batch
GPU utilization: low / bursty

Finding:

GPU is waiting on data loading/preprocessing.

Change tested:

increased DataLoader workers from A to B
enabled pin_memory
enabled persistent_workers
tested local cache

Result:

batch loading improved from X to Y
GPU utilization improved from A% to B%
throughput improved from X samples/s to Y samples/s

A PyTorch script has poor throughput

The script works, but samples/sec is too low.

What you measure

Throughput:

import time
import torch

num_samples = 0

torch.cuda.synchronize()
t0 = time.perf_counter()

for batch in loader:
    batch = batch.to("cuda")

    with torch.inference_mode():
        output = model(batch)

    num_samples += len(batch)

torch.cuda.synchronize()
t1 = time.perf_counter()

print("samples/sec:", num_samples / (t1 - t0))

Common causes

Batch size too small.
Running one sample at a time.
Python loops around tensor operations.
Using gradients during inference.
Calling .cpu(), .numpy(), or .item() inside the hot loop.
Excessive logging.
DataLoader bottleneck.
Model not on GPU.
Input not on GPU.
Repeated model initialization.
No warmup.

Things to search for in code that if in hot path are suspicious.

.item()
.cpu()
.numpy()
print(...)
for sample in batch:
model = ...
load_model(...)

Bad:

for sample in samples:
    output = model(sample.to("cuda"))

Better:

batch = torch.stack(samples).to("cuda")

with torch.inference_mode():
    output = model(batch)

Bad:

for x in tensor:
    result.append(slow_python_function(x))

Better: use tensor operations if possible.

Safe PyTorch approaches

Inference mode:

model.eval()

with torch.inference_mode():
    output = model(input)

Batching:

# Instead of N single inferences
# Do one batched inference
output = model(batch)

Mixed precision, if acceptable:

with torch.inference_mode():
    with torch.autocast(device_type="cuda", dtype=torch.float16):
        output = model(input)

Avoid synchronizing too often. These can force GPU sync(Sometimes okay, but bad inside tight loops):

loss.item()
tensor.cpu()
tensor.numpy()
print(tensor)

What you report

Baseline throughput: X samples/sec

Findings:

script was doing per-sample inference
hot loop included .cpu() / .item()
gradients were enabled during inference
DataLoader was slower than model execution

Changes:

batched inference
added inference_mode
removed unnecessary CPU sync from hot loop
adjusted batch size

Result:

throughput improved from X to Y samples/sec

A service has bad latency under load

One request may be okay, but many concurrent users make it slow.

What to measure: latency under concurrency:

single request latency
p50 latency
p95 latency
p99 latency
requests per second
error rate

Use custom tools, wrk, hey, ab, or Locust.

Example with hey.wrk:

hey -n 1000 -c 20 http://localhost:8000/predict
wrk -t4 -c32 -d60s http://localhost:8000/predict

Common causes

Server handles requests serially.
Model inference blocks the event loop.
Too many concurrent requests overwhelm GPU memory.
No batching.
No queue/backpressure.
Too many workers each load their own model copy.
Database/object storage call is slow.
Cold starts.
Large response serialization.

If it is FastAPI/async Python, look for blocking work inside async endpoint.

Suspicious:

@app.post("/predict")
async def predict(req):
    result = slow_blocking_model_call(req)
    return result

Async does not magically make CPU/GPU work non-blocking.
Check number of workers. If each worker loads a model, this can duplicate GPU memory:

ps aux | grep python
nvidia-smi

If you see many Python processes each using GPU memory, that may be the issue.

Safe fixes

Add simple concurrency control to reduce overload and improve p95/p99, even if p50 stays similar.:

import asyncio

semaphore = asyncio.Semaphore(4)

@app.post("/predict")
async def predict(req):
    async with semaphore:
        return run_model(req)

Add batching if system supports it.
Add queue/backpressure: If too many requests arrive, queue them or reject gracefully instead of letting latency explode.
Separate model time from request overhead.

What to report

Baseline under 20 concurrent clients:
throughput: X req/s
p50: A ms
p95: B ms
errors: C%

Finding:

latency explodes after concurrency N
GPU memory saturates / CPU saturates / requests serialize / event loop blocks

Change:

added concurrency limit
reduced worker count
tested batching
moved blocking work out of async path

Result:

p95 improved from X to Y
error rate dropped from A% to B%