Profiling in Python (Under Construction) - shannonhouck/mayhall-lab-manual GitHub Wiki

Work in Progress

Python has several tools for profiling that I've ended up using while working on expediting Vibin's TPSCI code. I'm going to try to make a brief list of things you can do for profiling, how to do them, and commentary on their advantages and limitations.

  1. cProfile

    cProfile should usually be your first resort. To run cProfile on your program, call it as:

    $python -m cProfile -o foo.prof bar.py

    This will profile bar.py and log the results to foo.prof. The easiest way to visualize these results is with snakeviz, which you can just pip install. To visualize foo.prof, run

    $snakeviz foo.prof

    Your system will open up an interactive icicle plot where you can see the time each process is taking. cProfiling is extremely easy and helpful. Importantly, you don't need to have any idea where the problems in your code are beforehand. The main weaknesses of cProfile is that it is fairly shallow in terms of what it will track for you. If you ask it to go much deeper than function calls, it's unlikely you'll get anything. Still, it at least tells you where you should look with a more specific profiler.

  2. line_profiler

    The line_profiler tool is available at

    https://github.com/rkern/line_profiler

    The installation and usage are explained quite well in the documentation. I have had issues with getting this tool to log its results. If you come across this issue, you can just use the -v flag when you call it and print the results to stdout. This method of profiling is nice since it is inherently more detailed than cProfile. I have had trouble with using this profiler to help tweak my code, only to discover that I've been (apparently) minimizing profiler overhead rather than increasing the speed of the code.

  3. pyinstrument Sampling tool- apparently this reduces a bias in overhead toward short, frequently called functions, since it doesn't track individual calls. It doesn't seem to be compatible with snakeviz, but it spits out a pretty readable summary. Found at:

    https://github.com/joerick/pyinstrument