AI Systems Overview - rFronteddu/general_wiki GitHub Wiki

  • Measure goodput: More than FLOPS/utilization, measure ratio of time the GPU spends performing useful work vs waiting on data or other overhead. Use
    • Use NVIDIA Nsight Systems/Compute
    • PyTorch profiler
  • Prefer skillful engineering optimizations instead of brute-force spending
  • Look for order-of-magnitude impact with incremental optimizations: At scale, small-percentage efficiency gain can save millions of dollars. Avoid redundant computations and slow data pipelines.
  • Approach performance tuning with a profile-driven mindset: Use profilers to identify and mitigate the true bottlenecks (compute utilization, memory bandwidth, memory latency, cache misses, or communication/network delays).
  • Maintain a holistic view: The best performance engineers consider hardware-software codesign: sometimes algorithm changes can alleviate hardware limits, and sometimes new hardware features enable new algorithms.
  • Stay informed on the latest hardware, software, and algorithms: Modern AI hardware and software are evolving rapidly use, stay on top (MLPerf)