Process Tracing Tips and Tools - oils-for-unix/oils GitHub Wiki

Tracing Processes You Don't Control

e.g. busybox ash on an Alpine machine, for reducing bugs from regtest/aports. These techniques do NOT require recompilation, debug symbols, or source code modification

  • Shell interpreter tracing with set -x aka set -o xtrace - show shell statements executed

    • Example from Andy: I used this to discover that autotool configure scripts call rm and cat commonly, and then we implemented builtin rm and builtin cat
    • multi-process tracing with set -x; export SHELLOPTS - bash and OSH have a feature where you can export SHELLOPTS to set -x on all child processes
    • Note: bash also has an XTRACE_FD= env var to send all the traces to the same place; I don't think OSH has this yet

  • strace myprog - show syscalls

    • Samuel used this for the $PWD bug
    • It can be helpful to use -f -o FILE and then start replacing the PIDs with a human name in the file to analyze e.g. races
    • use something like -e t=chdir,file,read,write, to limit the shown syscalls (e.g. memory is usually irrelevant)
    • -v to not abbreviate
    • -s 1000 to abbreviate strings only after 1000 characters (needs to be in addition to -v
    • the -c flag creates a histogram of syscalls
    • the -f flag follows child processes
    • multi-process tracing with strace -ff -o prefix - follow forks, creates prefix.$PID files

  • ltrace myprog - show libc calls

    • has an -e flag
    • Example from Andy: I used this to debug setlocale() calls in bash versus OSH

Tracing or Debugging Processes You Can Modify

Require Symbols / Recompilation

  • gdb --args myprog --flag
  • ASAN gives nicer stack traces
  • We are using uftrace - user space function tracing
    • it traces all function call entry and exit, by putting probes in the code

Require Source Code Instrumentation

  • systemtap - we have DTRACE_PROBE() calls in the Oils C++ source code

Related