Introduction to CUDA Profiling using Nvidia Nsight Compute

Relevant Links:

1. https://developer.nvidia.com/nsight-compute
2. https://developer.nvidia.com/tools-overview/nsight-compute/get-started
3. https://docs.nvidia.com/nsight-compute/NsightComputeCli/index.html

What is profiling?

When you are in the business of making efficient, fast code, you need tools that help you iron out inefficiencies and bottlenecks that might be slowing down your programs.

Code profiling will give you answers to questions like:

• How long does the execution of a portion of my code take?
• How much memory is this process occupying at runtime?
• Is there some unseen overhead causing the code to be slower than expected?
• Core usage, either on CPU or GPU

Tools that allow you to collate and interpret such details are called code profilers.

CUDA Profiling allows you to find answers to such questions and is therefore an indispensable tool. Until 2020 this tool used to be called Nvidia Visual Profiler (NVVP and NVProf). From 2020 Nvidia has moved to a new profiler Nvidia Nsight which has three components NSight Systems, Nsight Compute, Nsight Graphics.

Prerequisite for running on GCP VM

Either you can run nsight as administrator/root or execute the following commands to allow non administrators to run profiling.

• Create and open .conf file (e.g. profile.conf) in folder /etc/modprobe.d

sudo nano /etc/modprobe.d/profile.conf

• Add below line in profile.conf

options nvidia "NVreg_RestrictProfilingToAdminUsers=0"

• Close file /etc/modprobe.d/profile.conf
• run the following command:

sudo update-initramfs -u

• Reboot the system using:

sudo reboot

Profiling with Nsight Compute

Run a command in the same vein as example below.

(cuda_cl)$ ncu -o metrics python test.py > output.txt

For all metrics:

(cuda_cl)$ ncu --target-processes all --set full -o <output_name> python <your_script>.py

This command will profile the code in test.py and pipe the output to output.txt. Also a report will be generated with file name: "metrics.nsight-cuprof-report"

Make sure to add the following line at the top of your python file.

#!/usr/bin/env python

"""
.
.
.
Python Code
.
.
.
"""

Also, make sure your code is executable by running:

$ chmod u+x my_pycuda_code.py

Viewing the profiling result with Nsight Compute GUI

Note: For this you may first setup a GUI using VNC Viewer, please refer to https://github.com/eecse4750/e4750_2021Fall_students_repo/wiki/VM-GUI-Setup for instructions.

You can also install the Nsight Compute software on your local PC/Mac through this link. After you run the CLI command on the cloud and generate the output report, you can download the report onto your local system and visualize it using the already installed Nsight software.