SystemTap - shawfdong/hyades GitHub Wiki
SystemTap is scripting language and tool for dynamically instrumenting running production Linux. SystemTap scripts are written in the SystemTap language[1] and are saved with the .stp extension. The system carries out a number of passes on the script before allowing it to run, at which point the script is compiled into a loadable kernel module and loaded into the kernel. The module is unloaded when the tap has finished running.
I first tested SystemTap on my CentOS 7 workstation. To deploy SystemTap, the systemtap & systemtap-runtime packages along with the set of kernel-devel, kernel-debuginfo & kernel-debuginfo-common-x86_64 packages for the running kernel need to be installed[2]. The kernel-debuginfo & kernel-debuginfo-common-x86_64 packages are provided by the base-debuginfo repo on CentOS 7. The repo is disabled by default (see /etc/yum.repos.d/CentOS-Debuginfo.repo).
We can install the kernel packages from the disabled repo using the --enablerepo option:
# yum --enablerepo=base-debuginfo install kernel-debuginfo kernel-debuginfo-common-x86_64 kernel-devel
Alternatively we can enable the base-debuginfo repo by replacing the following line in /etc/yum.repos.d/CentOS-Debuginfo.repo:
enabled=0with
enabled=1
then install the kernel packages:
# yum clean all # yum install kernel-debuginfo kernel-debuginfo-common-x86_64 kernel-devel
Install SystemTap packages:
# yum install systemtap systemtap-runtime
Test SystemTap:
# stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}' Pass 1: parsed user script and 101 library script(s) using 213740virt/31276res/3056shr/28840data kb, in 140usr/20sys/146real ms. semantic error: while resolving probe point: identifier 'kernel' at /usr/share/systemtap/tapset/linux/vfs.stp:768:18 source: probe vfs.read = kernel.function("vfs_read") ^ semantic error: missing x86_64 kernel/module debuginfo [man warning::debuginfo] under '/lib/modules/3.10.0-123.20.1.el7.x86_64/build' semantic error: while resolving probe point: identifier 'vfs' at <input>:1:7 source: probe vfs.read {printf("read performed\n"); exit()} ^ semantic error: no match Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 216984virt/34632res/4964shr/29988data kb, in 50usr/70sys/128real ms. Pass 2: analysis failed. [man error::pass2]
As it turned out, CentOS doesn't update kernel-debuginfo as promptly as the kernel itself. The latest CentOS 7 kernel was 3.10.0-123.20.1.el7.x86_64:
# uname -r 3.10.0-123.20.1.el7.x86_64
but the latest kernel-debuginfo was for kernel 3.10.0-123.13.2.el7.x86_64:
# rpm -q kernel-debuginfo kernel-debuginfo-3.10.0-123.13.2.el7.x86_64
We, however, can still make an instrumentation kernel module for kernel 3.10.0-123.13.2.el7.x86_64:
# stap -v -r 3.10.0-123.13.2.el7.x86_64 -e 'probe vfs.read {printf("read performed\n"); exit()}' -m stap_test -p4 Pass 1: parsed user script and 101 library script(s) using 213616virt/31300res/3072shr/28716data kb, in 120usr/20sys/147real ms. Pass 2: analyzed script: 1 probe(s), 1 function(s), 3 embed(s), 0 global(s) using 480820virt/153092res/8008shr/144536data kb, in 1650usr/350sys/2409real ms. Pass 3: translated to C into "/tmp/stapv60dBA/stap_test_src.c" using 472940virt/151252res/7188shr/144536data kb, in 10usr/60sys/73real ms. stap_test.ko Pass 4: compiled C into "stap_test.ko" in 1400usr/450sys/4194real ms.
Let's boot to kernel-3.10.0-123.13.2.el7.x86_64 and retest:
# stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}' Pass 1: parsed user script and 101 library script(s) using 213736virt/31388res/3056shr/28836data kb, in 130usr/10sys/146real ms. Pass 2: analyzed script: 1 probe(s), 1 function(s), 3 embed(s), 0 global(s) using 480952virt/153056res/7992shr/144668data kb, in 1640usr/280sys/2520real ms. Pass 3: translated to C into "/tmp/stapochJBq/stap_79767ad302afcd07cf64815822aa902b_1459_src.c" using 473072virt/151288res/7244shr/144668data kb, in 10usr/60sys/66real ms. Pass 4: compiled C into "stap_79767ad302afcd07cf64815822aa902b_1459.ko" in 1330usr/470sys/2214real ms. Pass 5: starting run. read performed Pass 5: run completed in 10usr/40sys/329real ms.
Load the instrumentation module generated earlier:
# staprun stap_test.ko read performed
On Hyades, we run a customized Rocks 6.1. To deploy SystemTap on Hyades, we first download the RPM packages for
- kernel-debuginfo
- kernel-debuginfo-common-x86_64
- systemtap
- systemtap-client
- systemtap-devel
- systemtap-runtime
Build a new Rocks distribution:
# cd /export/rocks/install # rocks create distro
Install the kernel packages and SystemTap on the master node:
# yum install kernel-debuginfo kernel-debuginfo-common-x86_64 # yum install systemtap systemtap-runtime
Test SystemTap:
# stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}' Pass 1: parsed user script and 87 library script(s) using 196132virt/24332res/3084shr/22024data kb, in 110usr/20sys/130real ms. Pass 2: analyzed script: 1 probe(s), 1 function(s), 3 embed(s), 0 global(s) using 426276virt/124296res/8272shr/114184data kb, in 1120usr/100sys/1667real ms. Pass 3: translated to C into "/tmp/stapHeKy6z/stap_a159d83b47a3086ff87dcfe4362d2dac_1471_src.c" using 416460virt/120176res/6628shr/114184data kb, in 10usr/0sys/5real ms. Pass 4: compiled C into "stap_a159d83b47a3086ff87dcfe4362d2dac_1471.ko" in 4330usr/930sys/6903real ms. Pass 5: starting run. read performed Pass 5: run completed in 10usr/30sys/347real ms.
Install only systemtap-runtime on the compute nodes (target system):
# tentakel -g compute yum install systemtap-runtime
Generate instrumentation (kernel module) on the master node:
# stap -e 'probe vfs.read {printf("read performed\n"); exit()}' -m stap_test -p4
Run the instrumentation on the compute nodes (target system):
# staprun stap_test.ko read performed