DebianDebugSymbols - hpaluch/hpaluch.github.io GitHub Wiki
Debian Debug symbols
How to install Debian Debug symbols and other information. Tested on Debian 12.
Model example: issue with GNU tar:
Installing package source
To install package source I do this:
sudo apt-get install dpkg-dev
mkdir -p ~/src
cd ~/src
apt-get source tar
# package source is now in:
ls ~/src/tar-1.34+dfsg
Installing Debug symbols
We need to enable repository with *-dbgsym packages that contains "externalized" debug symbols.
More or less we will follow: https://wiki.debian.org/AutomaticDebugPackages
In my case I had this /etc/apt/sources.list for Debian 12:
deb http://ftp.cz.debian.org/debian bookworm main contrib
deb-src http://ftp.cz.debian.org/debian bookworm main contrib
deb http://ftp.cz.debian.org/debian bookworm-updates main contrib
deb-src http://ftp.cz.debian.org/debian bookworm-updates main contrib
# security updates
deb http://security.debian.org bookworm-security main contrib
deb-src http://security.debian.org bookworm-security main contrib
To add debug symbols I examined available directories on: http://deb.debian.org/debian-debug/dists/
In my case I decided to add this line to /etc/apt/sources.list:
deb http://deb.debian.org/debian-debug bookworm-debug main contrib
Now we have to install utility package that will find *-dgbsym for our package +
its dependencies:
apt-get install debian-goodies
To install symbols for tar package and its dependencies we have to issue:
find-dbgsym-packages /bin/tar
If there is other output than:
I: All needed dbgsym packages are already installed.
Run as root:
find-dbgsym-packages /bin/tar | xargs apt-get install
# will install: apt-get install libacl1-dbgsym libc6-dbg libpcre2-8-0-dbgsym libselinux1-dbgsym
Analyzing spinning process with perf
Linux perf is easy to use set of package to evaluate performance. To install Linux perf, run:
apt-get install linux-perf
To get real-time overview of system, just run:
sudo perf top
# press 'q' to quit
To analyze specific process (in my case tar process spinning cpu):
# run as root
perf record -p `pidof tar` -g
After some time (say 30 seconds) press Ctrl-C to stop recording. Performance
data will be recorded in file perf.data
To browse data try:
# run as root
perf report -g
You can expand stacktrace with e key, example screenshot:
Samples: 334K of event 'cycles', Event count (approx.): 167377283731
Children Self Command Shared Object Symbol
- 48.18% 0.00% tar [unknown] [.] 0x742f646c6975622f ◆
- 0x742f646c6975622f ▒
43.37% find_delayed_link_source ▒
- 48.00% 47.68% tar tar [.] extract_link ▒
extract_link ▒
- 43.37% 43.07% tar tar [.] find_delayed_link_source ▒
43.07% 0x742f646c6975622f ▒
find_delayed_link_source ▒
+ 4.10% 4.09% tar libc.so.6 [.] __strcmp_sse2 ▒
+ 3.52% 0.01% tar [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
Stacktrace in GDB
Here are basic command how to get stacktrace in GDB. First install GDB:
sudo apt-get install gdb
Example session:
# I will extract .tar.zst there:
mkdir ~/tmp
# ensure that you are in tar source directory:
cd ~/src/tar-1.34+dfsg
gdb /bin/tar
# ensure that there is no error "No symbols", in my example these messages look fine:
Reading symbols from /bin/tar...
Reading symbols from /usr/lib/debug/.build-id/7e/68e3094abf707c55b0a7baaac5c3a956347b0c.debug...
# it is recommended to break at "main" function and setup other breakpoint after "main"
# is reached - otherwise somtimes not all symbols are resolved:
(gdb) break main
Breakpoint 1 at 0xae80: file ./src/tar.c, line 2773.
(gdb) run xpf /home/ansible/deb12-yocto-az/dev-user-1723647938.tar.zst -C /home/ansible/tmp
Starting program: /usr/bin/tar xpf /home/ansible/deb12-yocto-az/dev-user-1723647938.tar.zst -C /home/ansible/tmp
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Breakpoint 1, main (argc=5, argv=0x7fffffffe3c8) at ./src/tar.c:2773
warning: Source file is more recent than executable.
2773 {
# Huh, hopefully it is just warning...
# tar will fork - follow child
(gdb) set follow-fork-mode child
# try breakpoint at function of interest:
(gdb) break find_delayed_link_source
Breakpoint 2 at 0x55555556bdd0: file ./src/extract.c, line 1339.
(gdb) break extract_link
Breakpoint 3 at 0x55555556d670: file ./src/extract.c, line 1465.
# continue execution
(gdb) c
Continuing.
[Detaching after fork from child process 1297]
Here is how to get stacktrace when breakpoint is reached:
reakpoint 2, find_delayed_link_source (name=name@entry=0x5555555d83a0 "dev/poky/scripts/cross-intercept/ar")
at ./src/extract.c:1339
1339 {
(gdb) bt
#0 find_delayed_link_source (name=name@entry=0x5555555d83a0 "dev/poky/scripts/cross-intercept/ar")
at ./src/extract.c:1339
#1 0x000055555556d360 in create_placeholder_file (
file_name=file_name@entry=0x5555555d83a0 "dev/poky/scripts/cross-intercept/ar", is_symlink=is_symlink@entry=true,
interdir_made=interdir_made@entry=0x7fffffffe037, prev=prev@entry=0x0) at ./src/extract.c:1380
#2 0x000055555556d5f4 in extract_symlink (file_name=0x5555555d83a0 "dev/poky/scripts/cross-intercept/ar",
typeflag=<optimized out>) at ./src/extract.c:1540
#3 0x000055555556e9f6 in extract_archive () at ./src/extract.c:1820
#4 0x00005555555784b6 in read_and (do_something=0x55555556e720 <extract_archive>) at ./src/list.c:229
#5 0x000055555555fed8 in main (argc=<optimized out>, argv=<optimized out>) at ./src/tar.c:2845
(gdb) print name
$1 = 0x5555555d83a0 "dev/poky/scripts/cross-intercept/ar"
(gdb) info locals
dl = <optimized out>
st = {st_dev = 93824992763672, st_ino = 93824992382737, st_nlink = 93824992798977, st_mode = 1, st_uid = 0,
st_gid = 4294958896, __pad0 = 32767, st_rdev = 93824992372844, st_size = 4299262263789, st_blksize = 0,
st_blocks = 93824992782336, st_atim = {tv_sec = 93824992764008, tv_nsec = 0}, st_mtim = {tv_sec = 140737488346896,
tv_nsec = 24}, st_ctim = {tv_sec = 140737488346976, tv_nsec = 140737488346912}, __glibc_reserved = {
-5727356510224353024, 4096, 4096}}
Now we will disable that early breakpoint and continue execution:
# list breakpoints
(gdb) info b
Num Type Disp Enb Address What
1 breakpoint keep y 0x000055555555ee80 in main at ./src/tar.c:2773
breakpoint already hit 1 time
2 breakpoint keep y 0x000055555556bdd0 in find_delayed_link_source at ./src/extract.c:1339
breakpoint already hit 1 time
3 breakpoint keep y 0x000055555556d670 in extract_link at ./src/extract.c:1465
# you can delete breakpoint with number using for example "del 3"
Now other breakpoint reached:
(gdb) c
Continuing.
Breakpoint 2, find_delayed_link_source (
name=name@entry=0x5555555d8860 "dev/poky/scripts/esdk-tools/runqemu-extract-sdk") at ./src/extract.c:1339
1339 {
(gdb) bt
#0 find_delayed_link_source (name=name@entry=0x5555555df120 "dev/poky/scripts/esdk-tools/runqemu-ifup")
at ./src/extract.c:1339
#1 0x000055555556d360 in create_placeholder_file (
file_name=file_name@entry=0x5555555df120 "dev/poky/scripts/esdk-tools/runqemu-ifup",
is_symlink=is_symlink@entry=true, interdir_made=interdir_made@entry=0x7fffffffe037, prev=prev@entry=0x0)
at ./src/extract.c:1380
#2 0x000055555556d5f4 in extract_symlink (file_name=0x5555555df120 "dev/poky/scripts/esdk-tools/runqemu-ifup",
typeflag=<optimized out>) at ./src/extract.c:1540
#3 0x000055555556e9f6 in extract_archive () at ./src/extract.c:1820
#4 0x00005555555784b6 in read_and (do_something=0x55555556e720 <extract_archive>) at ./src/list.c:229
#5 0x000055555555fed8 in main (argc=<optimized out>, argv=<optimized out>) at ./src/tar.c:2845
(gdb) info locals
dl = <optimized out>
st = {st_dev = 93824992763672, st_ino = 93824992382737, st_nlink = 93824992772988, st_mode = 1, st_uid = 0,
st_gid = 4294958896, __pad0 = 32767, st_rdev = 93824992372844, st_size = 10, st_blksize = 140737352282077,
st_blocks = 0, st_atim = {tv_sec = 93824992801056, tv_nsec = 2}, st_mtim = {tv_sec = 140737488346896, tv_nsec = 24},
st_ctim = {tv_sec = 140737488346976, tv_nsec = 140737488346912}, __glibc_reserved = {-5727356510224353024,
93824992369424, 93824992772960}}
# list source
(gdb) list
1334 and does not rely on comparing file names, which may differ for
1335 various reasons (e.g. relative vs. absolute file names).
1336 */
1337 static struct delayed_link *
1338 find_delayed_link_source (char const *name)
1339 {
1340 struct delayed_link *dl;
1341 struct stat st;
1342
1343 if (!delayed_link_head)
Examine my problem with tar - I disabled all breakpoints:
(gdb) dis break
(gdb) c