SystemTap - shawfdong/hyades GitHub Wiki

SystemTap is scripting language and tool for dynamically instrumenting running production Linux. SystemTap scripts are written in the SystemTap language^[1] and are saved with the .stp extension. The system carries out a number of passes on the script before allowing it to run, at which point the script is compiled into a loadable kernel module and loaded into the kernel. The module is unloaded when the tap has finished running.

I first tested SystemTap on my CentOS 7 workstation. To deploy SystemTap, the systemtap & systemtap-runtime packages along with the set of kernel-devel, kernel-debuginfo & kernel-debuginfo-common-x86_64 packages for the running kernel need to be installed^[2]. The kernel-debuginfo & kernel-debuginfo-common-x86_64 packages are provided by the base-debuginfo repo on CentOS 7. The repo is disabled by default (see /etc/yum.repos.d/CentOS-Debuginfo.repo).

We can install the kernel packages from the disabled repo using the --enablerepo option:

# yum --enablerepo=base-debuginfo install kernel-debuginfo kernel-debuginfo-common-x86_64 kernel-devel

Alternatively we can enable the base-debuginfo repo by replacing the following line in /etc/yum.repos.d/CentOS-Debuginfo.repo:

enabled=0

with

enabled=1

then install the kernel packages:

# yum clean all
# yum install kernel-debuginfo kernel-debuginfo-common-x86_64 kernel-devel

Install SystemTap packages:

# yum install systemtap systemtap-runtime

Test SystemTap:

# stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
Pass 1: parsed user script and 101 library script(s) using 213740virt/31276res/3056shr/28840data kb, in 140usr/20sys/146real ms.
semantic error: while resolving probe point: identifier 'kernel' at /usr/share/systemtap/tapset/linux/vfs.stp:768:18
        source: probe vfs.read = kernel.function("vfs_read")
                                 ^

semantic error: missing x86_64 kernel/module debuginfo [man warning::debuginfo] under '/lib/modules/3.10.0-123.20.1.el7.x86_64/build'
semantic error: while resolving probe point: identifier 'vfs' at <input>:1:7
        source: probe vfs.read {printf("read performed\n"); exit()}
                      ^

semantic error: no match
Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 216984virt/34632res/4964shr/29988data kb, in 50usr/70sys/128real ms.
Pass 2: analysis failed.  [man error::pass2]

As it turned out, CentOS doesn't update kernel-debuginfo as promptly as the kernel itself. The latest CentOS 7 kernel was 3.10.0-123.20.1.el7.x86_64:

# uname -r
3.10.0-123.20.1.el7.x86_64

but the latest kernel-debuginfo was for kernel 3.10.0-123.13.2.el7.x86_64:

# rpm -q kernel-debuginfo
kernel-debuginfo-3.10.0-123.13.2.el7.x86_64

We, however, can still make an instrumentation kernel module for kernel 3.10.0-123.13.2.el7.x86_64:

# stap -v -r 3.10.0-123.13.2.el7.x86_64 -e 'probe vfs.read {printf("read performed\n"); exit()}' -m stap_test -p4
Pass 1: parsed user script and 101 library script(s) using 213616virt/31300res/3072shr/28716data kb, in 120usr/20sys/147real ms.
Pass 2: analyzed script: 1 probe(s), 1 function(s), 3 embed(s), 0 global(s) using 480820virt/153092res/8008shr/144536data kb, in 1650usr/350sys/2409real ms.
Pass 3: translated to C into "/tmp/stapv60dBA/stap_test_src.c" using 472940virt/151252res/7188shr/144536data kb, in 10usr/60sys/73real ms.
stap_test.ko
Pass 4: compiled C into "stap_test.ko" in 1400usr/450sys/4194real ms.

Let's boot to kernel-3.10.0-123.13.2.el7.x86_64 and retest:

# stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
Pass 1: parsed user script and 101 library script(s) using 213736virt/31388res/3056shr/28836data kb, in 130usr/10sys/146real ms.
Pass 2: analyzed script: 1 probe(s), 1 function(s), 3 embed(s), 0 global(s) using 480952virt/153056res/7992shr/144668data kb, in 1640usr/280sys/2520real ms.
Pass 3: translated to C into "/tmp/stapochJBq/stap_79767ad302afcd07cf64815822aa902b_1459_src.c" using 473072virt/151288res/7244shr/144668data kb, in 10usr/60sys/66real ms.
Pass 4: compiled C into "stap_79767ad302afcd07cf64815822aa902b_1459.ko" in 1330usr/470sys/2214real ms.
Pass 5: starting run.
read performed
Pass 5: run completed in 10usr/40sys/329real ms.

Load the instrumentation module generated earlier:

# staprun stap_test.ko 
read performed

On Hyades, we run a customized Rocks 6.1. To deploy SystemTap on Hyades, we first download the RPM packages for

• kernel-debuginfo
• kernel-debuginfo-common-x86_64
• systemtap
• systemtap-client
• systemtap-devel
• systemtap-runtime

from a CentOS 6 repository and place them in /export/rocks/install/contrib/6.1/x86_64/RPMS.

Build a new Rocks distribution:

# cd /export/rocks/install
# rocks create distro

Install the kernel packages and SystemTap on the master node:

# yum install kernel-debuginfo kernel-debuginfo-common-x86_64
# yum install systemtap systemtap-runtime

Test SystemTap:

# stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
Pass 1: parsed user script and 87 library script(s) using 196132virt/24332res/3084shr/22024data kb, in 110usr/20sys/130real ms.
Pass 2: analyzed script: 1 probe(s), 1 function(s), 3 embed(s), 0 global(s) using 426276virt/124296res/8272shr/114184data kb, in 1120usr/100sys/1667real ms.
Pass 3: translated to C into "/tmp/stapHeKy6z/stap_a159d83b47a3086ff87dcfe4362d2dac_1471_src.c" using 416460virt/120176res/6628shr/114184data kb, in 10usr/0sys/5real ms.
Pass 4: compiled C into "stap_a159d83b47a3086ff87dcfe4362d2dac_1471.ko" in 4330usr/930sys/6903real ms.
Pass 5: starting run.
read performed
Pass 5: run completed in 10usr/30sys/347real ms.

Install only systemtap-runtime on the compute nodes (target system):

# tentakel -g compute yum install systemtap-runtime

Generate instrumentation (kernel module) on the master node:

# stap -e 'probe vfs.read {printf("read performed\n"); exit()}' -m stap_test -p4

Run the instrumentation on the compute nodes (target system):

# staprun stap_test.ko 
read performed

1. ^ SystemTap Language Reference
2. ^ RHEL6 - SystemTap Beginners Guide