Debugging Crash Dumps - SAP/project-foxhound GitHub Wiki

Introduction

This page gives some insights into how to debug Firefox (and Foxhound) with crash reports.

Building with crash dumps active:

Make sure the following line is not in your mozconfig when building:

ac_add_options --disable-crashreporter

By default the crash reporter will be there, but is disabled e.g. when building with playwright.

Getting Debug Symbols

Debug symbols are needed to get the function and file name/number out of the crash reports, so are essential for actually debugging. Either make sure you have debug symbols included, or create them separately afterwards.

Build with debug symbols

ac_add_options --enable-debug-symbols

Prevent debug symbol stripping:

Packaging via mach package will strip debug symbols, so you also need to disable this via:

ac_add_options --disable-install-strip

Create debug symbols separately

Run this command:

./mach buildsymbols

Which will produce a zip file like this: firefox-98.0.2.en-US.win64.crashreporter-symbols.zip. Unzip this somewhere on your filesystem.

More Info:

For more information see firefox docs: https://firefox-source-docs.mozilla.org/setup/building_with_debug_symbols.html

Dump Anaylsis

The crash reporter should produce dump files like this:

4feec7d3-ff98-40ac-a574-ea575b53c48b.dmp

Linux

Under Linux the dump files are creating the in following directory:

${HOME}/.firefox/Crash Reports/pending

Windows

On windows the dump files are here:

%APPDATA%\Mozilla\Firefox\Crash Reports\

Also see here for more information: https://support.mozilla.org/en-US/kb/mozillacrashreporter

Minidump Stackwalk

The dump files can be analysed with a combination of the minidump stackwalk took and the debug symbols.

Install minidump-stackwalk from here: https://github.com/rust-minidump/rust-minidump/tree/main/minidump-stackwalk

Analysis

You can analyse the dump with the command:

minidump-stackwalk --verbose=trace --cyborg=4feec7d3-ff98-40ac-a574-ea575b53c48b.dmp.json --pretty 4feec7d3-ff98-40ac-a574-ea575b53c48b.dmp --symbols-path=/path/to/debug/symbols/firefox-98.0.2.en-US.win64.crashreporter-symbols/ > 4feec7d3-ff98-40ac-a574-ea575b53c48b.dmp.log

Which should give you a file 4feec7d3-ff98-40ac-a574-ea575b53c48b.dmp.log like this:

Operating system: Windows NT
                  10.0.19042
CPU: amd64
     family 6 model 158 stepping 13
     16 CPUs

Crash reason:  EXCEPTION_ACCESS_VIOLATION_READ
Crash address: 0xffffffffffffffff
Process uptime: 42 seconds

Thread 0 MainThread (crashed)
 0  xul.dll!static JSRope::flattenInternal<JSRope::NoBarrier,char16_t>(JSRope*) [StringType.cpp : 739 + 0x0]
     rax = 0x0000b28e81ca8626    rdx = 0x00007ff8e148a59d
     rcx = 0x00002b2fcaf91ee0    rbx = 0x00000235bba0b000
     rsi = 0x7573207972746e45    rdi = 0x00002b2fcaf91ee0
     rbp = 0x00002b2fcaf00000    rsp = 0x0000007c1bdfba80
      r8 = 0x0000000000000000     r9 = 0x0000007c1bdfbb60
     r10 = 0x0000000000000001    r11 = 0x000003fc71ddd48c
     r12 = 0x00000235bba17100    r13 = 0x00002b2fcaf91ee0
     r14 = 0x000003fc71ddd461    r15 = 0x0000000000000001
     rip = 0x00007ff8df1c33e7
    Found by: given as instruction pointer in context
 1  xul.dll!JSRope::flatten(JSContext*) [StringType.cpp : 648 + 0x51]
     rbx = 0x00000235bba0b000    rbp = 0x00002b2fcaf00000
     rsp = 0x0000007c1bdfbb30    r12 = 0x00000235bba17100
     r13 = 0x00002b2fcaf91ee0    r14 = 0x000003fc71ddd461
     r15 = 0x0000000000000001    rip = 0x00007ff8df1adbca
    Found by: call frame info
 2  xul.dll!JS::taintarg(JSContext*, JS::Handle<JSString *>) [jstaint.cpp : 94 + 0x13]
     rbx = 0x00000235bba0b000    rbp = 0x00002b2fcaf00000
     rsp = 0x0000007c1bdfbb90    r12 = 0x00000235bba17100
     r13 = 0x00002b2fcaf91ee0    r14 = 0x000003fc71ddd461
     r15 = 0x0000000000000001    rip = 0x00007ff8df0463f9
    Found by: call frame info
 3  xul.dll!JS::TaintOperationFromContext(JSContext*, char const*, bool, JS::Handle<JSString *>) [jstaint.cpp : 261 + 0x20]
     rbx = 0x00000235bba0b000    rbp = 0x00002b2fcaf00000
     rsp = 0x0000007c1bdfbc10    r12 = 0x00000235bba17100
     r13 = 0x00002b2fcaf91ee0    r14 = 0x000003fc71ddd461
     r15 = 0x0000000000000001    rip = 0x00007ff8df047036
    Found by: call frame info
 4  xul.dll!TrimString(JSContext*, JS::CallArgs const&, char const*, bool, bool) [String.cpp : 2941 + 0x36]
     rbx = 0x00000235bba0b000    rbp = 0x00002b2fcaf00000
     rsp = 0x0000007c1bdfbd00    r12 = 0x00000235bba17100
     r13 = 0x00002b2fcaf91ee0    r14 = 0x000003fc71ddd461
     r15 = 0x0000000000000001    rip = 0x00007ff8df128f52
    Found by: call frame info
 5  xul.dll!str_trim(JSContext*, unsigned int, JS::Value*) [String.cpp : 2953 + 0x18]
     rbx = 0x00000235bba0b000    rbp = 0x00002b2fcaf00000
     rsp = 0x0000007c1bdfbe40    r12 = 0x00000235bba17100
     r13 = 0x00002b2fcaf91ee0    r14 = 0x000003fc71ddd461
     r15 = 0x0000000000000001    rip = 0x00007ff8df127927
    Found by: call frame info
 6  0x10be2dae649
     rbx = 0x00000235bba0b000    rbp = 0x00002b2fcaf00000
     rsp = 0x0000007c1bdfbe90    r12 = 0x00000235bba17100
     r13 = 0x00002b2fcaf91ee0    r14 = 0x000003fc71ddd461
     r15 = 0x0000000000000001    rip = 0x0000010be2dae64a
    Found by: call frame info

Script It!

Here's a handy bash script to process a bunch of files at once:

#!/bin/bash

for i in *.dmp; do
    [ -f "$i" ] || break
    echo ${i}
    minidump-stackwalk --verbose=trace --cyborg=${i}.json --pretty ${i} --symbols-path=../symbols/ > ${i}.log
done

Other Info

https://hacks.mozilla.org/2022/06/everything-is-broken-shipping-rust-minidump-at-mozilla/ https://hacks.mozilla.org/2021/05/improving-firefox-stability-on-linux/ https://firefox-source-docs.mozilla.org/contributing/debugging/debugging_firefox_with_gdb.html

⚠️ **GitHub.com Fallback** ⚠️