Profiling - mwicat/personal GitHub Wiki

timing decorator

def timing_val(func):
    @wraps(func)
    def wrapper(*arg, **kw):
        t1 = time.time()
        res = func(*arg, **kw)
        t2 = time.time()
        print(
            func.__name__, str(arg), str(kw), 'call took', (t2 - t1), 'seconds',
            file=sys.stderr)
        return res
    return wrapper

import django.core.management
django.core.management.call_command = timing_val(django.core.management.call_command)

ipython profile

one call:

https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-time

%time <statement>

multiple calls:

https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-timeit

%timeit <statement>

profiler gui (best)

sudo pip3 install snakeviz
snakeviz view.prof

profiler gui

sudo apt install kcachegrind
sudo pip3 install pyprof2calltree
pyprof2calltree -i view.prof -k

The "All Callers" tab columns should represent the following:

Incl.: The number of instructions that this function generated as a whole broken down by each caller. Because callers are a hierarchy (hence the distance column) there may be several that have the same value if your call stack is deep.

Distance: How many function calls separated is the selected line from the function that is selected in the Flat Profile panel.

Called: The number of time the Caller called the a function that ultimately led to the execution of the selected function).

Caller: The function that directly called or called another caller of your selected function (as determined by Distance).

The Callers tab is more straightforward. It shows the functions that have a distance of 1 from your selected function. In other words, these are the functions that directly invoke your selected function.

Ir: The number of instructions executed in total by the selected function after being called by this caller.

Ir per call: The number of instructions executed per call.

Count: The number of times the selected function was called by the caller.

Caller: The function that directly called the selected function.

For Events, see this page for the handbook. I suspect that if you didn't define your own types all you should see is "Instruction Fetch" and possibly "Cycle Estimation." The quick breakdown on these columns is as follows:

Incl.: Again the total instructions performed by this function and all functions it calls beneath it.

Self: The instructions performed exclusively by this function. This counter only tracks instructions used by this function, not any instruction used by functions that are called by this function.

Short and Formula: These columns are used when defining a custom Event Type. Yours should either be blank or very short (like CEst = Ir) unless you end up defining your own Types.

1.1 What is the difference between 'Incl.' and 'Self'?
These are cost attributes for functions regarding some event type. As functions can call each other, it makes sense to distinguish the cost of the function itself ('Self Cost') and the cost including all called functions ('Inclusive Cost'). 'Self' is sometimes also referred to as 'Exclusive' costs.

So e.g. for main(), you will always have a inclusive cost of almost 100%, whereas the self cost is neglectable when the real work is done in another function.

profile code

sudo pip install profilehooks
from profilehooks import profile

@profile(immediate=True, filename='view.prof', stdout=True)
def my_function(args, etc):
    pass
import cProfile

def profileit_stdout(func):
    def wrapper(*args, **kwargs):
        datafn = func.__name__ + ".profile" # Name the data file sensibly
        prof = cProfile.Profile()
            try:
                result = profiler.runcall(fn, *args, **kwargs)
            finally:
                stats = pstats.Stats(profiler)
                stats.strip_dirs().sort_stats(*sort_args).print_stats(*print_args)

        retval = prof.runcall(func, *args, **kwargs)
        prof.dump_stats(datafn)
        return retval

    return wrapper

def profileit(func):
    def wrapper(*args, **kwargs):
        datafn = func.__name__ + ".profile" # Name the data file sensibly
        prof = cProfile.Profile()
        retval = prof.runcall(func, *args, **kwargs)
        prof.dump_stats(datafn)
        return retval

    return wrapper

@profileit
def function_you_want_to_profile(...)
    ...

profile script

python -m cProfile -o script.prof script.py
pyprof2calltree -i script.prof -k

django profiler middleware

import marshal
import pstats

try:
    import cProfile as profile
except ImportError:
    import profile

try:
    from cStringIO import StringIO
except ImportError:
    from io import StringIO

from django.conf import settings
from django.http import HttpResponse
from django.utils.deprecation import MiddlewareMixin


class ProfilerMiddleware(MiddlewareMixin):

    @staticmethod
    def should_profile(request):
        return settings.PROFILE and 'prof' in request.GET

    def process_view(self, request, callback, callback_args, callback_kwargs):
        if self.should_profile(request):
            self.profiler = profile.Profile()
            args = (request,) + callback_args
            try:
                return self.profiler.runcall(callback, *args, **callback_kwargs)
            except Exception:
                return

    def process_response(self, request, response):
        if self.should_profile(request):
            self.profiler.create_stats()
            if 'download' in request.GET:
                response = self.download_response()
            else:
                response = self.stats_response(request)
        return response

    def download_response(self):
        output = marshal.dumps(self.profiler.stats)
        response = HttpResponse(output, content_type='application/octet-stream')
        response['Content-Disposition'] = 'attachment; filename=view.prof'
        response['Content-Length'] = len(output)
        return response

    def stats_response(self, request):
        io = StringIO()
        stats = pstats.Stats(self.profiler, stream=io)

        stats.sort_stats(request.GET.get('sort', 'time'))
        stats.print_stats(int(request.GET.get('count', 250)))

        return HttpResponse('<pre>%s</pre>' % io.getvalue())

statsd timer function

from statsd import StatsClient

statsd = StatsClient()

with statsd.timer('timername'):
    timed_function()
Add a custom footer
⚠️ **GitHub.com Fallback** ⚠️