Introduction

Trying to investigate an issue with jujud which is manifesting as high cpu or high memory usage by reading log files is about as productive as diagnosing an issue with your digestive tract by staring at your navel.

For Juju 2.0 and Juju 1.25.4 (or later) a profiling facility has been added to the jujud binary. This page describes the operation of the pprof facility.

No user serviceable parts inside

The pprof profiling facility is very low level. It is not a general purpose profiling or debugging tool. This page does describe how to interpret the information this facility returns, only how to collect data for later analysis.

History

pprof started life as part of the Google Perf Tools toolkit. Around 2010 the Go runtime was enhanced to collect profiling data with the recording of that data left unspecified. Slightly later in 2010 a new package, http/pprof (later renamed to net/http/pprof) was added. net/http/pprof presents a simple HTTP interface to request profiling data from a running Go program.

The profiling support added to jujud is based on this HTTP interface, modified to serve data over a local unix socket rather than tcp. Hopefully this explains the idiosyncratic user interface.

Limitations

This facility is only available on linux platforms.

This facility is only enabled for jujud binaries, what are commonly referred to as machine or unit agents. The facility is not enabled for hook commands although they share the same underlying binary via the argv[0] trick.

Prerequisites

To access the pprof facility requires issuing HTTP requests over a local unix socket. This page uses socat(1), which on the default cloud image is not installed by default. Please apt-get install socat first.

Usage

While running each jujud (version 1.25.x) binary will create a unix socket in /tmp (this can be overridden by setting $TMP or $TMPDIR before starting the process). The name of the socket follow the form

 /tmp/pprof.$PROCESSNAME.$PID

Where $PROCESSNAME is almost always jujud, and $PID is self explanatory.

If you are running juju2, then you won't find the socked file created in /tmp/ - rather, jujud creates a abstract socket, so connecting to it is a bit different. One can use lsof to get the name of the socket created:

 lsof -p `pidof jujud` | grep \@ 

Goroutine profile

To obtain a list of the running goroutines inside a jujud process, use the following command, adjusting the socket path as necessary:

echo -e "GET /debug/pprof/goroutine?debug=1 HTTP/1.0\r\n" | socat unix-connect:/tmp/pprof.jujud.5276 STDIO

or, for juju2:

echo -e "GET /debug/pprof/goroutine?debug=1 HTTP/1.0\r\n" | socat abstract-connect:jujud-machine-0 STDIO

Heap profile

To obtain a sample of the heap usage inside a jujud process, use the following command, adjusting the socket path as necessary:

echo -e "GET /debug/pprof/heap?debug=1 HTTP/1.0\r\n" | socat unix-connect:/tmp/pprof.jujud.5276 STDIO

or, for juju2:

echo -e "GET /debug/pprof/heap?debug=1 HTTP/1.0\r\n" | socat abstract-connect:jujud-machine-0 STDIO

The heap profile reports statistics as of the most recently completed garbage collection; it elides more recent allocation to avoid skewing the profile away from live data and toward garbage. The most useful information in this profile is at the end.

Connect via web browser

Instead of using SSH and the command line, it's possible to expose the pprof UNIX socket over a proxied HTTP connection, so you can connect to it with a web browser. Connect to the Juju machine and install socat as usual:

$ juju ssh 0
machine-0$ sudo su
machine-0$ apt-get install socat -y
machine-0$ socat tcp-listen:6060,reuseaddr,fork "unix-connect:/tmp/pprof.jujud.$(pgrep jujud)"

Leave the last command and its terminal running.

Then, assuming you can access the machine directly (not only via SSH), get its address from juju status, and open your browser with http://machine-0-ip-address:6060/debug/pprof. Follow the pprof package documentation - https://golang.org/pkg/net/http/pprof/ for more examples and details.

NOTE: The same setup allows you to use go tool pprof with the same URL, giving you access to the interactive pprof shell with all of its nice commands. Example session: http://paste.ubuntu.com/15471571/