Profiling - openucx/ucx GitHub Wiki

UCS contains a tool to collect profiling information and save it to a file. Profiling is based on "locations", which are places in the code to collect timestamps from.
There are 3 types of locations:

• SAMPLE - simple timestamp
• SCOPE_START - mark the beginning of a nested block (code block or function call)
• SCOPE_END - mark the end of a nested block.
• REQUEST_NEW - start tracking a nonblocking send or receive request
• REQUEST_EVENT - mark an event in the lifetime of a request
• REQUEST_FREE - stop tracking a request

In addition there are several convenience macros to profile the code:

• UCS_PROFILE_CODE - Measure the time of a code block.
• UCS_PROFILE_FUNC[_VOID] - Wrap a function with profiling code which measures its time.
• UCS_PROFILE[_NAMED]_CALL[_VOID] - Profile a function call (which wasn't wrapped with UCS_PROFILE_FUNC)

When enabling profile collection, one or more of the following modes can be used:

• accum - Accumulate time and count per location.
• log - Collect all timestamps. If the log buffer is exhausted, newer records would override old ones in a circular buffer fashion.

The profiling data is saved to a file when the program exits, or when UCS catches the signal SIGHUP.
In order to read the file, the ucx_read_profile tool should be used:

$ ucx_read_profile -h
Usage: ucx_read_profile [options] <file>
Options:
      -r             raw output
      -t UNITS       select time units (sec/msec/usec/nsec)

The following profiled code is a top-level function called my_func which is profiled, and in addition it calls printf("Hello World!") and profiles that call:

UCS_PROFILE_FUNC_VOID(my_func, ()) {
    UCS_PROFILE_CALL(printf, "Hello World!\n");
}

Configure and compile again, it is recommended to use configure-prof which is just like configure-release except profiling is enabled:

$ ./contrib/configure-prof --enable-profiling --prefix=$PWD/install # etc
$ make -j 32 && make install

Run an application and collect profile:

$ UCX_PROFILE_MODE=log,accum UCX_PROFILE_FILE=ucx.prof ./app

Read profile output file:

$ ucx_read_profile ucx.prof      

   command : ./app
   host    : my_host
   pid     : 9999
   units   : usec

                     NAME           AVG         TOTAL      COUNT             FILE     FUNCTION
                   printf        15.316            15          1     profiling.c:13   my_func_inner()
                  my_func        15.883            16          1     profiling.c:12   my_func()

     0.000  my_func 15.883 {                           profiling.c:12   my_func()
     0.332      printf 15.316 {                        profiling.c:13   my_func_inner()
    15.316      }
     0.236  }

See more advanced example in ucx_profiling.c

It is possible to use the profiling infrastructure to profile code in applications or libraries using UCX. In order to do that, need to #include <ucs/profile/profile_on.h> and add the profiling macros to the code locations of interest.
In order to disable profiling by build configuration, need to #include <ucs/profile/profile_off.h> instead, which defines the profiling macros as empty stubs.