GPUs - BKJackson/BKJackson_Wiki GitHub Wiki

Scaling Up with GPUs

From bare metal to a 70B model: infrastructure set-up and scripts

GPU's can be programmed

• CUDA (NVDIA only)
  • Write C code that runs directly on the GPU
  • Higher-level APIs: cuBLAS, cuFFT, cuDNN, etc
• OpenCL
  • Similar to CUDA, but runs on anything
  • Usually slower :(
• Udacity: Intro to Parallel Programming http://www.udacity.com/cource/cs344
  • For deep learning use existing libraries

Credit for above: Li, Karpathy, and Johnson (2016)