PerformanceAnalysisOs - henk52/knowledgesharing GitHub Wiki

Introduction

Table of Contents

References

Tools of interest

  • Dyninst - API that aids the development of performance measurement tools, debuggers and simulators by permitting the insertion of code into a running program.

  • strace - trace system calls of a running application.

  • memstomp - provide tools to help identify code with undefined behavior at a lower runtime cost than other tools such as Valgrind.

  • SystemTap

  • OProfile

  • Valgrind

  • elfutils

  • DWZ

  • numastat

  • netsniff-ng

    • ifpps

  • KernelShark

Kernel performance

  • perf

    • rpm -ivh perf-* *qperf?

perf

  • The design of the perf_event kernel interface uses one file descriptor per event per-thread or per-cpu.

  • irq:

  • irq_vectors

  • sched:

    • sched:sched_stat_blocked

    • sched:sched_switch

  • syscalls:sys_enter_sched_*

  • perf stat -e sched:* --pid 29077 -I 1000 -x,

    • -I 1000: print every 1 second.

    • -x, : produce comma separated list.

  • perf buildid-list -i perf.data

    • Binary identification with build-id

Enabling schedstats in Red Hat

Add 'schedstats=enable' to the kernel boot parameter list in grub.cfg

linux/Documentation/admin-guide/kernel-parameters.txt

Documented in:

A description of the perf events

  • ./tools/perf/util/parse-events.l:context-switches|cs { return sym(yyscanner, PERF_TYPE_SOFTWARE, PERF_COUNT_SW_CONTEXT_SWITCHES); }

  • ./tools/perf/util/parse-events.c: .symbol = "context-switches",

  • ./Documentation/static-keys.txt

  • ./include/linux/perf_event.h

  • ./tools/perf/design.txt: PERF_COUNT_SW_CONTEXT_SWITCHES = 3,

  • ./tools/include/uapi/linux/perf_event.h: PERF_COUNT_SW_CONTEXT_SWITCHES = 3,

  • ./tools/perf/tests/attr/test-stat-default

  • ./tools/perf/tests/attr/test-stat-detailed-1

using perf to see the off CPU reasons

  1. perf record -e sched:sched_stat_sleep -e sched:sched_switch -e sched:sched_process_exit -g -o ~/perf.data.raw sleep 20

  2. perf inject -v -s -i ~/perf.data.raw -o ~/perf.data

  3. perf report --stdio --show-total-period -i ~/perf.data

Performance engineering

  • Linpack MPI

  • SPECcpu

  • AIM 7

  • OLTP

  • DSS

  • SPECsfs NFS

  • Postmark

  • SAP - SLCS, SD

  • STAC = FSI

Benchmarks - code path coverage

  • CPU

    • linpack

    • lmbench

    • Stream ?

    • Java - SPECjbb

  • Disk IO

    • Iozone

    • aiostress - scsi

    • FC

  • filesystem

    • postmark

  • network

    • netperf

  • Advanced performance tools

  • Tuna/tuned/perf

  • Performance Co-Pilot Support

    • pmatop awesome,

    • pmcollectl

  • SPECvirt2010/2012

  • SPECvirt_sc2013

  • TPC-VMS

Hardware

  • hwloc

  • lstopo

Example

RHEL "tuned" package

# yum install tune*
# tuned-adm profile latency-performance
# tuned-adm list
Available profiles:
- latency-performance
- default
- enterprise-storage
- virtual-guest
- throughput-performance
- virtual-host
Current active profile: latency-performance
# tuned-adm profile default (to disable)

CPU Performance analysis

sysctl: cannot stat /proc/sys/kernel/sched_schedstats: No such file or directory

sysctl kernel.sched_schedstats=1 

sysctl: cannot stat /proc/sys/kernel/sched_schedstats: No such file or directory
rpm -qa| grep debug
kernel-debug-3.10.0-327.22.2.el7.x86_64

# rpm -ql kernel-debug-3.10.0-327.22.2.el7.x86_64 | grep config
/boot/config-3.10.0-327.22.2.el7.x86_64.debug

grep SCHEDSTATS /boot/config-3.10.0-327.22.2.el7.x86_64.debug
CONFIG_SCHEDSTATS=y

 cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-327.22.2.el7.x86_64.debug ... schedstats=enable

Error: The /root/perf.data file has no samples!

sysctl kernel.sched_schedstats=1

For RedHat add 'schedstats=enable' to the kernel boot parameter list in grub.cfg

 perf report -I --stdio -i ~/perf.data
Error:
The /root/perf.data file has no samples!

failed to write feature 2

See:

Describes the code added that provides the error message: 'failed to write feature' * https://lwn.net/Articles/460520/

 perf inject -v -s -i ~/perf.data.raw -o ~/perf.data
registering plugin: /usr/lib64/traceevent/plugins/plugin_xen.so
registering plugin: /usr/lib64/traceevent/plugins/plugin_scsi.so
registering plugin: /usr/lib64/traceevent/plugins/plugin_cfg80211.so
registering plugin: /usr/lib64/traceevent/plugins/plugin_function.so
registering plugin: /usr/lib64/traceevent/plugins/plugin_hrtimer.so
registering plugin: /usr/lib64/traceevent/plugins/plugin_kvm.so
registering plugin: /usr/lib64/traceevent/plugins/plugin_sched_switch.so
registering plugin: /usr/lib64/traceevent/plugins/plugin_mac80211.so
registering plugin: /usr/lib64/traceevent/plugins/plugin_jbd2.so
registering plugin: /usr/lib64/traceevent/plugins/plugin_kmem.so
overriding event (310) sched:sched_switch with new print handler
build id event received for [kernel.kallsyms]: 1d6b07240aeb5cf1ab9a5a82cf2fc0d6abba6d19
failed to write feature 2

tools

sar

sar -A -o /my_machine.sar 1

  • 1 - one second interval

  • -A - stands for:

    • -q - queue length

    • -B - paging

    • -b - storage

    • -I ALL | XALL interupt

    • -n DEV - network stats

    • -n EDEV - network errors

    • -P ALL

    • -u ALL more cpu info that -P ?

    • -w switches per seconds

Add a custom footer
⚠️ **GitHub.com Fallback** ⚠️