Building and profiling with gperftools - ofi-cray/libfabric-cray GitHub Wiki

Setting up gperftools

Building libfabric for profiling with gperftools

We need to build libfabric and the application with the gcc flag -fno-omit-frame-pointer. See https://github.com/gperftools/gperftools/blob/master/INSTALL for more information.

  • Building libfabric ./autogen.sh && ./configure CFLAGS="-fno-omit-frame-pointer" --prefix="$PWD/libfabric-test" && make clean && make -j install

Building cray-tests for profiling with gperftools

  • Use the --with-profiler option during the cray-tests configure step ./autogen.sh && ./configure --prefix=$PWD/../cray-tests-build --with-libfabric=$PWD/../libfabric-test --with-pmi=/opt/cray/pmi/default --with-profiler=/path/to/gperftools-build && make -j install

Configuring an autotools application for profiling with gperftools

./configure CFLAGS=-fno-omit-frame-pointer CPPFLAGS=-I"$PWD/gperftools-build/include/" LDFLAGS=-L"$PWD/lib/" LIBS=-lprofiler

Creating an executable for the CPU profiler from gperftools

  • We need to manually compile a separate source file with a main function since pprof uses the binary to generate the profiling output. Using gnitest as the binary does not work. Here is an example of how the source code should look https://github.com/ofi-cray/cray-tests/blob/master/contrib/buddy_test.c. Note that you need to include <gperftools/profiler.h>.
  • Once you have your source code and your calls to ProfilerStart("name.prof") and ProfilerStop() where you want them and have built libfabric in $PWD/libfabric-build compile it like this on tiger: gcc -fno-omit-frame-pointer -Wall `pkg-config cray-ugni --cflags --libs` `pkg-config cray-gni-headers --cflags --libs` `pkg-config cray-udreg --cflags --libs` -I $PWD/gperftools-build/include/ -I $PWD/libfabric-cray/include/ -I $PWD/prov/gni/include/ -L$PWD/libfabric-build/lib/ -L$PWD/gperftools-build/lib/ -I $PWD/libfabric-cray -lfabric -lprofiler buddy_test.c -o buddy_test $PWD/libfabric-build/lib/libfabric.a

Running the executable

  • To profile the entire application: export CPUPROFILE=ProfileName.prof
  • export CPUPROFILE_FREQUENCY=1000000 (this can be increased to have the profiler peek at the top of the call stack more frequently)
  • Run it manually or with a job launcher

Using pprof

pprof --text a.out buddy.test

Gotchas

  • Don't forget to include gperftools/gperftools.h.
  • Wrap the section of code you're interested in in ProfilerStart("ProfileName.prof") and ProfilerStop().
  • Remember that if the job runs multiple ranks/threads/messageSizes and the ProfileName.prof is named the same for each rank/thread/messageSize the last instance to run will stomp on all the other profiles.

Notes