Analyzing SECONDO Crashes - secondo-database/secondo GitHub Wiki

This page describes how a crash of SECONDO can be analyzed. For the demonstration of a crash, the faultcrash operator is used, which is part of the AuxiliaryAlgebra. This operator forwards a stream of tuples and crashes the system with a certain probability. This operator is primarily used in SECONDO to demonstrate the error recovery capability of the distributed algebras. For example, the query query plz feed faultcrash[100] count crashes the system with a probability of (1/100) = 1% per processed tuple.

Table of Contents

Automatically Generated Stacktrace

After SECONDO has crashed, a signal handler is called, which tries to dump the current stack. To be more precise: in Tools/Utilities/Application.cpp, a signal handler is defined and called on signals like SIGINT, SIGTERM, or SIGFPE.

Default Stacktrace

The generated stacktrace is a convenient way to get a first impression which method has caused the crash. A wrapper script (i.e., bin/SecondoTTYBDB) parses the output of the stackstrace and tries to resolve the addresses of the stack to line numbers. However, this method has two significant drawbacks: (1) when the stack becomes corrupted before the crash occurs, the stacktrace can not be generated, and (2) depending on the used compiler options (e.g., -fPIC for position independent code), the addresses of the output do not resolve to the real line numbers see #6:

$ ./SecondoTTYBDB

[...]

Secondo => query plz feed faultcrash[100] count;
command 
'query plz feed faultcrash[100] count'
started at: Thu Oct 22 08:31:50 2020

noMemoryOperators = 0
perOperator = 0

 Generating stack trace ... 
 ************ BEGIN STACKTRACE ************
No stacktrace output file defined, dumping stacktrace to stdout
./SecondoBDB(_ZN7WinUnix10stacktraceEPKcS1_+0x6d)[0x55d4f31ca5b1]
./SecondoBDB(_ZN11Application20AbortOnSignalHandlerEi+0xf8)[0x55d4f31c1860]
/lib/x86_64-linux-gnu/libc.so.6(+0x3efd0)[0x7fc65d4bcfd0]
./SecondoBDB(_ZN9auxiliary14FaultLocalInfoILNS_9CrashTypeE0EE12forwardTupleEv+0x4f)[0x55d4f425dc6b]
./SecondoBDB(_ZN9auxiliary11InjectFaultILNS_9CrashTypeE0EEEiP4WordRS2_iS4_Pv+0xfe)[0x55d4f425d944]
./SecondoBDB(_ZN14QueryProcessor4EvalEPvR4Wordi+0xe33)[0x55d4f31f71f3]
./SecondoBDB(_ZN14QueryProcessor7RequestEPvR4Word+0x62)[0x55d4f31f794c]
./SecondoBDB(_Z12TCountStreamP4WordRS_iS1_Pv+0xcf)[0x55d4f335e1a0]
./SecondoBDB(_ZN14QueryProcessor4EvalEPvR4Wordi+0xe33)[0x55d4f31f71f3]
./SecondoBDB(_ZN14QueryProcessor5EvalPEPvR4Wordi+0xea)[0x55d4f31f5fb2]
./SecondoBDB(_ZN19SecondoInterfaceTTY13Command_QueryEmRmRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb+0x3cc)[0x55d4f3120ebe]
./SecondoBDB(_ZN19SecondoInterfaceTTY7SecondoERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmibbRmRiS9_RS5_S7_b+0x5100)[0x55d4f311d30a]
./SecondoBDB(_ZN10SecondoTTY11CallSecondoEv+0x34f)[0x55d4f3178399]
./SecondoBDB(_ZN10SecondoTTY12CallSecondo2Ev+0x2c)[0x55d4f317875e]
./SecondoBDB(_ZN10SecondoTTY14ProcessCommandEv+0x75c)[0x55d4f317774e]
./SecondoBDB(_ZN10SecondoTTY15ProcessCommandsEbb+0x9d)[0x55d4f3177d2f]
./SecondoBDB(_ZN10SecondoTTY7ExecuteEv+0x3d3)[0x55d4f3178cbb]
./SecondoBDB(_Z14SecondoTTYModeRK12TTYParameter+0x2cf)[0x55d4f3179400]
./SecondoBDB(main+0x1f6)[0x55d4f3175314]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7fc65d49fb97]
./SecondoBDB(_start+0x2a)[0x55d4f3114cfa]
*********** END STACKTRACE **********************

*** Signal SIGILL (4) caught! Calling default signal handler ...

Note: A stacktrace has to be read from the bottom up. On the bottom you will find the entry point in the application whereas on the top are the most recent called functions and methods.

In the example, the first three entries are the most important ones. ZN7WinUnix10stacktraceEPKcS1_+0x6d shows that the stacktrace method of the class WinUnix was run as the last method. Before this method was called, the AbortOnSignalHandler method of the class Application was called (_ZN11Application20AbortOnSignalHandlerEi+0xf8). The third entry on the stack is the most important one: _ZN9auxiliary14FaultLocalInfoILNS_9CrashTypeE0EE12forwardTupleEv+0x4f. It can be seen that the forwardTuple method of a class called auxiliary was called before the crash handler was executed. This is exactly the method of the faultcrash operator in the AuxiliaryAlgebra. In the next examples, the exact line number is determined where the crash has occurred.

It can also be seen that SECONDO has crashed with a Illegal Instruction Exception (SIGILL).

Automatically Decoded Stacktrace

When (1) the tool addr2line is installed, (2) SECONDO is started from the script SecondoTTYBDB, and (3) the relocation info can be fetched from the linker, the addresses of the stacktrace are automatically mapped to source code lines. This gives a much more clear view of where the crash has occurred. However, this only works in certain environments.

$ ./SecondoTTYBDB

[...]

Secondo => query plz feed faultcrash[100] count;
command 
'query plz feed faultcrash[100] count'
started at: Thu Oct 22 08:31:50 2020

noMemoryOperators = 0
perOperator = 0

========
SECONDO has crashed, printing stack trace....
========



Binary relocation: 0x55bae030b000

./SecondoBDB(_ZN7WinUnix10stacktraceEPKcS1_S1_+0x74)[0x55bae15d9f08]
./SecondoBDB(_ZN11Application20AbortOnSignalHandlerEi+0x107)[0x55bae15d12af]
/lib/x86_64-linux-gnu/libc.so.6(+0x3efd0)[0x7fe36e962fd0]
./SecondoBDB(_ZN9auxiliary14FaultLocalInfoILNS_9CrashTypeE0EE12forwardTupleEv+0x4f)[0x55bae2682499]
./SecondoBDB(_ZN9auxiliary11InjectFaultILNS_9CrashTypeE0EEEiP4WordRS2_iS4_Pv+0xfe)[0x55bae2682172]
./SecondoBDB(_ZN14QueryProcessor4EvalEPvR4Wordi+0xe33)[0x55bae1606bd1]
./SecondoBDB(_ZN14QueryProcessor7RequestEPvR4Word+0x62)[0x55bae160732a]
./SecondoBDB(_Z12TCountStreamP4WordRS_iS1_Pv+0xcf)[0x55bae176e168]
./SecondoBDB(_ZN14QueryProcessor4EvalEPvR4Wordi+0xe33)[0x55bae1606bd1]
./SecondoBDB(_ZN14QueryProcessor5EvalPEPvR4Wordi+0xea)[0x55bae1605990]
./SecondoBDB(_ZN19SecondoInterfaceTTY13Command_QueryEmRmRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb+0x3cc)[0x55bae1530ace]
./SecondoBDB(_ZN19SecondoInterfaceTTY7SecondoERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmibbRmRiS9_RS5_S7_b+0x5100)[0x55bae152cf1a]
./SecondoBDB(_ZN10SecondoTTY11CallSecondoEv+0x34f)[0x55bae1587df1]
./SecondoBDB(_ZN10SecondoTTY12CallSecondo2Ev+0x2c)[0x55bae15881b6]
./SecondoBDB(_ZN10SecondoTTY14ProcessCommandEv+0x75c)[0x55bae15871e2]
./SecondoBDB(_ZN10SecondoTTY15ProcessCommandsEbb+0x9d)[0x55bae15877c3]
./SecondoBDB(_ZN10SecondoTTY7ExecuteEv+0x3d3)[0x55bae1588713]
./SecondoBDB(_Z14SecondoTTYModeRK12TTYParameter+0x2cf)[0x55bae1588e1b]
./SecondoBDB(main+0x1f6)[0x55bae1584da8]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7fe36e945b97]
./SecondoBDB(_start+0x2a)[0x55bae152490a]


========
 Trying to decode the stacktrace...
========


WinUnix::stacktrace(char const*, char const*, char const*) at WinUnix.cpp:247
Application::AbortOnSignalHandler(int) at Application.cpp:330
?? ??:0
auxiliary::FaultLocalInfo<(auxiliary::CrashType)0>::forwardTuple() at AuxiliaryAlgebra.cpp:376
int auxiliary::InjectFault<(auxiliary::CrashType)0>(Word*, Word&, int, Word&, void*) at AuxiliaryAlgebra.cpp:421
QueryProcessor::Eval(void*, Word&, int) at QueryProcessor.cpp:4491
QueryProcessor::Request(void*, Word&) at QueryProcessor.cpp:4691
TCountStream(Word*, Word&, int, Word&, void*) at RelationAlgebra.cpp:2096
QueryProcessor::Eval(void*, Word&, int) at QueryProcessor.cpp:4491
QueryProcessor::EvalP(void*, Word&, int) at QueryProcessor.cpp:4151
SecondoInterfaceTTY::Command_Query(unsigned long, unsigned long&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool) at SecondoInterfaceTTY.cpp:1917
SecondoInterfaceTTY::Secondo(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, int, bool, bool, unsigned long&, int&, int&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) at SecondoInterfaceTTY.cpp:1555
SecondoTTY::CallSecondo() at SecondoTTY.cpp:577 (discriminator 2)
SecondoTTY::CallSecondo2() at SecondoTTY.cpp:629
SecondoTTY::ProcessCommand() at SecondoTTY.cpp:423
SecondoTTY::ProcessCommands(bool, bool) at SecondoTTY.cpp:481
SecondoTTY::Execute() at SecondoTTY.cpp:721
SecondoTTYMode(TTYParameter const&) at SecondoTTY.cpp:880
main at MainTTY.cpp:164
?? ??:0
_start at ??:?

Decode the Stacktrace Manually

In certain situations (e.g., when working with the Distributed Algebra and SECONDO workers), only the default stacktrace is contained in a logfile. This stacktrace needs to be decoded manually by the addr2line tool and a possible binary relocation has to be subtracted first (see the output of the stacktrace). SECONDO ships with the helper script bin/decode_stacktrace.sh to ease the calculation and the decoding of the stacktrace.

Important: This only works when the SECONDO binary on the used system is equal to the binary of the crashed SECONDO. Otherwise, the addresses are different in the binary, and wrong line numbers are displayed.

To use the tool, copy the stacktrace into a textfile and start the script with the filename as a parameter (e.g., ./decode_stacktrace.sh stacktrace.txt). Please ensure that a possible Binary relocation: is contained in the stacktrace file.

A stacktrace file (e.g., stacktrace.txt) should look as follows:

Binary relocation: 0x55a0eaea6000

./SecondoBDB(_ZN7WinUnix10stacktraceEPKcS1_S1_+0x74)[0x55a0ec18bdc6]
./SecondoBDB(_ZN11Application20AbortOnSignalHandlerEi+0x107)[0x55a0ec18316d]
/lib/x86_64-linux-gnu/libc.so.6(+0x3efd0)[0x7f0df2aacfd0]
./SecondoBDB(_ZN9auxiliary14FaultLocalInfoILNS_9CrashTypeE0EE12forwardTupleEv+0x4f)[0x55a0ed25c20b]
./SecondoBDB(_ZN9auxiliary11InjectFaultILNS_9CrashTypeE0EEEiP4WordRS2_iS4_Pv+0xfe)[0x55a0ed25bee4]
./SecondoBDB(_ZN14QueryProcessor4EvalEPvR4Wordi+0xe33)[0x55a0ec1b8a8f]
./SecondoBDB(_ZN14QueryProcessor7RequestEPvR4Word+0x62)[0x55a0ec1b91e8]
./SecondoBDB(_Z12TCountStreamP4WordRS_iS1_Pv+0xcf)[0x55a0ec320026]
./SecondoBDB(_ZN14QueryProcessor4EvalEPvR4Wordi+0xe33)[0x55a0ec1b8a8f]
./SecondoBDB(_ZN14QueryProcessor5EvalPEPvR4Wordi+0xea)[0x55a0ec1b784e]
./SecondoBDB(_ZN19SecondoInterfaceTTY13Command_QueryEmRmRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb+0x3cc)[0x55a0ec0e281e]
./SecondoBDB(_ZN19SecondoInterfaceTTY7SecondoERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmibbRmRiS9_RS5_S7_b+0x5100)[0x55a0ec0dec6a]
./SecondoBDB(_ZN10SecondoTTY11CallSecondoEv+0x34f)[0x55a0ec139caf]
./SecondoBDB(_ZN10SecondoTTY12CallSecondo2Ev+0x2c)[0x55a0ec13a074]
./SecondoBDB(_ZN10SecondoTTY14ProcessCommandEv+0x75c)[0x55a0ec1390a0]
./SecondoBDB(_ZN10SecondoTTY15ProcessCommandsEbb+0x9d)[0x55a0ec139681]
./SecondoBDB(_ZN10SecondoTTY7ExecuteEv+0x3d3)[0x55a0ec13a5d1]
./SecondoBDB(_Z14SecondoTTYModeRK12TTYParameter+0x2cf)[0x55a0ec13acd9]
./SecondoBDB(main+0x1f6)[0x55a0ec136c66]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f0df2a8fb97]
./SecondoBDB(_start+0x2a)[0x55a0ec0d665a]

This file can be now decoded by calling decode_stacktrace.sh:

$ ./decode_stacktrace.sh stacktrace.txt

========
 Trying to decode the stacktrace...
========

Executing: addr2line --demangle=auto -p -fs -e SecondoBDB 0x12E5DC6 0x12DD16D 0x296D07C06FD0 0x23B620B 0x23B5EE4 0x1312A8F 0x13131E8 0x147A026 0x1312A8F 0x131184E 0x123C81E 0x1238C6A 0x1293CAF 0x1294074 0x12930A0 0x1293681 0x12945D1 0x1294CD9 0x1290C66 0x296D07BE9B97 0x123065A

WinUnix::stacktrace(char const*, char const*, char const*) at WinUnix.cpp:247
Application::AbortOnSignalHandler(int) at Application.cpp:330
?? ??:0
auxiliary::FaultLocalInfo<(auxiliary::CrashType)0>::forwardTuple() at AuxiliaryAlgebra.cpp:376
int auxiliary::InjectFault<(auxiliary::CrashType)0>(Word*, Word&, int, Word&, void*) at AuxiliaryAlgebra.cpp:421
QueryProcessor::Eval(void*, Word&, int) at QueryProcessor.cpp:4491
QueryProcessor::Request(void*, Word&) at QueryProcessor.cpp:4691
TCountStream(Word*, Word&, int, Word&, void*) at RelationAlgebra.cpp:2096
QueryProcessor::Eval(void*, Word&, int) at QueryProcessor.cpp:4491
QueryProcessor::EvalP(void*, Word&, int) at QueryProcessor.cpp:4151
SecondoInterfaceTTY::Command_Query(unsigned long, unsigned long&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool) at SecondoInterfaceTTY.cpp:1917
SecondoInterfaceTTY::Secondo(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, int, bool, bool, unsigned long&, int&, int&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) at SecondoInterfaceTTY.cpp:1555
SecondoTTY::CallSecondo() at SecondoTTY.cpp:577 (discriminator 2)
SecondoTTY::CallSecondo2() at SecondoTTY.cpp:629
SecondoTTY::ProcessCommand() at SecondoTTY.cpp:423
SecondoTTY::ProcessCommands(bool, bool) at SecondoTTY.cpp:481
SecondoTTY::Execute() at SecondoTTY.cpp:721
SecondoTTYMode(TTYParameter const&) at SecondoTTY.cpp:880
main at MainTTY.cpp:164
?? ??:0
_start at ??:?

Using Valgrind

Valgrind is a tool to profile and debug C and C++ software. It allows us to detect memory leaks and also shows debug data when software crashes. SECONDO automatically calls Valgrind when the option --valgrind is provided. A drawback is that Valgrind performs many checks, which causes some overhead during the execution of SECONDO and significantly slows down the execution.

$ ./SecondoTTYBDB  --valgrind

[...]

Secondo => query plz feed faultcrash[100] count;
command 
'query plz feed faultcrash[100] count'
started at: Thu Oct 22 08:37:25 2020


noMemoryOperators = 0
perOperator = 0

 Generating stack trace ... 
 ************ BEGIN STACKTRACE ************
Writing stacktrace to: /tmp/tmp.Wavv1YsUFd

 *********** END STACKTRACE **********************

*** Signal SIGILL (4) caught! Calling default signal handler ...
==14132== 
==14132== Process terminating with default action of signal 4 (SIGILL)
==14132==    at 0x5EA2715: raise (raise.c:46)
==14132==    by 0x139C8AF: Application::AbortOnSignalHandler(int) (Application.cpp:311)
==14132==    by 0x81A2FCF: ??? (in /lib/x86_64-linux-gnu/libc-2.27.so)
==14132==    by 0x2438C5F: auxiliary::FaultLocalInfo<(auxiliary::CrashType)0>::forwardTuple() (AuxiliaryAlgebra.cpp:373)
==14132==    by 0x2438943: int auxiliary::InjectFault<(auxiliary::CrashType)0>(Word*, Word&, int, Word&, void*) (AuxiliaryAlgebra.cpp:420)
==14132==    by 0x13D21F2: QueryProcessor::Eval(void*, Word&, int) (QueryProcessor.cpp:4491)
==14132==    by 0x13D294B: QueryProcessor::Request(void*, Word&) (QueryProcessor.cpp:4689)
==14132==    by 0x153919F: TCountStream(Word*, Word&, int, Word&, void*) (RelationAlgebra.cpp:2095)
==14132==    by 0x13D21F2: QueryProcessor::Eval(void*, Word&, int) (QueryProcessor.cpp:4491)
==14132==    by 0x13D0FB1: QueryProcessor::EvalP(void*, Word&, int) (QueryProcessor.cpp:4143)
==14132==    by 0x12FBEBD: SecondoInterfaceTTY::Command_Query(unsigned long, unsigned long&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool) (SecondoInterfaceTTY.cpp:1915)
==14132==    by 0x12F8309: SecondoInterfaceTTY::Secondo(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, int, bool, bool, unsigned long&, int&, int&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) (SecondoInterfaceTTY.cpp:1555)
==14132==    by 0x1353398: SecondoTTY::CallSecondo() (SecondoTTY.cpp:577)
==14132==    by 0x135375D: SecondoTTY::CallSecondo2() (SecondoTTY.cpp:629)
==14132==    by 0x135274D: SecondoTTY::ProcessCommand() (SecondoTTY.cpp:422)
==14132==    by 0x1352D2E: SecondoTTY::ProcessCommands(bool, bool) (SecondoTTY.cpp:481)
==14132==    by 0x1353CBA: SecondoTTY::Execute() (SecondoTTY.cpp:721)
==14132==    by 0x13543FF: SecondoTTYMode(TTYParameter const&) (SecondoTTY.cpp:880)
==14132==    by 0x1350313: main (MainTTY.cpp:164)
==14132== 
==14132== HEAP SUMMARY:
==14132==     in use at exit: 38,668,949 bytes in 57,824 blocks
==14132==   total heap usage: 241,221 allocs, 183,397 frees, 170,810,525 bytes allocated
==14132==

Valgrind produces a more clean stacktrace with line numbers. The stack can be read from the top to the bottom to see which methods have been called before the crash has occurred. The following lines are the most important ones:

==14132==    at 0x5EA2715: raise (raise.c:46)
==14132==    by 0x139C8AF: Application::AbortOnSignalHandler(int) (Application.cpp:311)
==14132==    by 0x81A2FCF: ??? (in /lib/x86_64-linux-gnu/libc-2.27.so)
==14132==    by 0x2438C5F: auxiliary::FaultLocalInfo<(auxiliary::CrashType)0>::forwardTuple() (AuxiliaryAlgebra.cpp:373)
==14132==    by 0x2438943: int auxiliary::InjectFault<(auxiliary::CrashType)0>(Word*, Word&, int, Word&, void*) (AuxiliaryAlgebra.cpp:420)

This stacktrace contains more or less the same entries as the stacktrace of the last example. The AbortOnSignalHandler of the class Application was called before the application crashes. Before that, the forwardTuple method of the class auxiliary was executed. This method can be found in the file AuxiliaryAlgebra.cpp in line 373.

Using the GNU Debugger (GDB) and Core Dumps on Linux

To overcome the slowdown that Valgrind introduces, the GNU debugger can be used together with core dumps. Core dumps are generated on Unix systems after a software crashes. The core dumps contain the memory image of the crashed software; the call stack is a part of this memory image. To save disk space, most Unix systems don't generate core dumps per default. To store these files on disk, the command ulimit -c unlimited has to be executed. This sets the maximal the size of core dumps -c to an unlimited amount of bytes. After the command is called, the core dumps of all programs that are started from this shell are written to disk.

$ ulimit -c unlimited
$ ./SecondoTTYBDB

[...]

Secondo => query plz feed faultcrash[100] count;

*** Signal SIGILL (4) caught! Calling default signal handler ...

./SecondoTTYBDB: line 34: 14367 Floating point exception(core dumped) $runner $*

After SECONDO has crashed, a file named core was written:

$ ls -l core
-rw------- 1 nidzwetzki nidzwetzki 51212288 Okt 22 08:41 core

This file can now be loaded into the GNU Debugger (gdb). In addition, the path to the binary file that has generated this core dump has to be provided. In most cases, this is SecondoBDB. However, when a problem in the server component of SECONDO has caused the crash, the appropriate binary (e.g., SecondoListener) has to be provided as a parameter.

$ gdb SecondoBDB core
[...]
Core was generated by `./SecondoBDB'.
Program terminated with signal SIGILL, Arithmetic exception.
#0  0x000055e1bfb17c6b in auxiliary::FaultLocalInfo<(auxiliary::CrashType)0>::forwardTuple (this=0x55e1c210c280) at AuxiliaryAlgebra.cpp:376
376	             __builtin_trap();
[Current thread is 1 (Thread 0x7f6b742f37c0 (LWP 14367))]

After the core dump has been processed by gdb, two important information can be seen: (1) the method forwardTuple is shown together with the file and line number AuxiliaryAlgebra.cpp:376 that has caused the crash, and (2) the line of code that has caused the crash (__builtin_trap();) is also shown. This line executes a illegal instruction, which crashes SECONDO.

GDB provides a lot of useful commands. For example, the command bt (short for back trace) dumps the complete stacktrace.

(gdb) bt
#0  0x000055e1bfb17c6b in auxiliary::FaultLocalInfo<(auxiliary::CrashType)0>::forwardTuple (this=0x55e1c210c280) at AuxiliaryAlgebra.cpp:376
#1  0x000055e1bfb17944 in auxiliary::InjectFault<(auxiliary::CrashType)0> (args=0x55e1c2413430, result=..., message=2, local=..., s=0x55e1c24133f0)
    at AuxiliaryAlgebra.cpp:420
#2  0x000055e1beab11f3 in QueryProcessor::Eval (this=0x55e1c1ec96c0, node=0x55e1c24133f0, result=..., message=2) at QueryProcessor.cpp:4491
#3  0x000055e1beab194c in QueryProcessor::Request (this=0x55e1c1ec96c0, s=0x55e1c24133f0, result=...) at QueryProcessor.cpp:4689
#4  0x000055e1bec181a0 in TCountStream (args=0x55e1c24129c0, result=..., message=1, local=..., s=0x55e1c2412980) at RelationAlgebra.cpp:2095
#5  0x000055e1beab11f3 in QueryProcessor::Eval (this=0x55e1c1ec96c0, node=0x55e1c2412980, result=..., message=1) at QueryProcessor.cpp:4491
#6  0x000055e1beaaffb2 in QueryProcessor::EvalP (this=0x55e1c1ec96c0, node=0x55e1c2412980, result=..., message=1) at QueryProcessor.cpp:4143
#7  0x000055e1be9daebe in SecondoInterfaceTTY::Command_Query (this=0x55e1c1ebce00, list=5, resultList=@0x7ffe4780a658: 0, errorMessage="", autoTransaction=true)
    at SecondoInterfaceTTY.cpp:1915
#8  0x000055e1be9d730a in SecondoInterfaceTTY::Secondo (this=0x55e1c1ebce00, commandText="query plz feed faultcrash[100] count", commandLE=0, commandLevel=1, 
    commandAsText=false, resultAsText=false, resultList=@0x7ffe4780a658: 0, errorCode=@0x55e1c1e56948: 0, errorPos=@0x7ffe4780a644: 0, errorMessage="", 
    resultFileName="SecondoResult", isApplicationLevelCommand=true) at SecondoInterfaceTTY.cpp:1555
#9  0x000055e1bea32399 in SecondoTTY::CallSecondo (this=0x55e1c1e56730) at SecondoTTY.cpp:577
#10 0x000055e1bea3275e in SecondoTTY::CallSecondo2 (this=0x55e1c1e56730) at SecondoTTY.cpp:629
#11 0x000055e1bea3174e in SecondoTTY::ProcessCommand (this=0x55e1c1e56730) at SecondoTTY.cpp:422
#12 0x000055e1bea31d2f in SecondoTTY::ProcessCommands (this=0x55e1c1e56730, stopOnError=false, isPD=false) at SecondoTTY.cpp:481
#13 0x000055e1bea32cbb in SecondoTTY::Execute (this=0x55e1c1e56730) at SecondoTTY.cpp:721
#14 0x000055e1bea33400 in SecondoTTYMode (tp=...) at SecondoTTY.cpp:880
#15 0x000055e1bea2f314 in main (argc=1, argv=0x7ffe4780b4e8) at MainTTY.cpp:164

Note: Don't forget to delete the core file after the examination of the crash is done. This saves space and allows SECONDO to generate a new core dump on the next crash. If the core file is not generated on a Linux system, have a look at the file /proc/sys/kernel/core_pattern (e.g., cat /proc/sys/kernel/core_pattern). Some Linux distributions write the core dumps to other directories, append the process id (i.e., core.12345), or forward the core dumps to a crash handler software.

Using the low-level debugger (lldb) on macOS

macOS uses a slightly different toolchain for debugging software. To debug SECONDO on macOS, we recommend starting SECONDO directly in the low-level debugger (lldb). A crash is captured and handled by the debugger automatically, and a readable stacktrace can be generated.

To start SECONDO in the lldb, please execute the command lldb ./SecondoBDB in the bin directory of SECONDO. After the debugger is running, the path to prolog needs to be provided. This can be done by executing env LD_LIBRARY_PATH="/Applications/SWI-Prolog.app/Contents/Frameworks". As the last step, run needs to be executed to start SECONDO.

In the following example, SECONDO is started in the debugger, and a query containing the faultcrash operator is executed:

$ lldb ./SecondoBDB
(lldb) target create "./SecondoBDB"
Current executable set to '/Users/kristofnidzwetzki/secondo/bin/SecondoBDB' (x86_64).

(lldb) env LD_LIBRARY_PATH="/Applications/SWI-Prolog.app/Contents/Frameworks"                                                               

(lldb) run

[...]

Secondo => query plz feed faultcrash[100] count;
command 
'query plz feed faultcrash[100] count'
started at: Fri Dec 11 12:09:19 2020

noMemoryOperators = 0
perOperator = 0

Process 33348 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)
    frame #0: 0x00000001018bbef2 SecondoBDB`auxiliary::FaultLocalInfo<(auxiliary::CrashType)0>::forwardTuple(this=0x0000000107235bf0) at AuxiliaryAlgebra.cpp:372:14
   369 	      int randValue = rand();
   370 	      if(randValue % crashAfter == 0) {
   371 	          if(crashType == CRASH) {
-> 372 	             __builtin_trap();
   373 	             exit(1);
   374 	          } else {
   375 	             // Loop forever! 
Target 0: (SecondoBDB) stopped.

It can be seen that the resulting crash is captured and handled by the debugger. The operation that caused the crash is marked with a ->. In this example, this is line 372 in the file AuxiliaryAlgebra.cpp.

By typing in bt (short for back trace), a full stacktrace can be generated. lldb provides almost the same commands as gdb. More information about the differences can be found here.

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)
  * frame #0: 0x00000001018bbef2 SecondoBDB`auxiliary::FaultLocalInfo<(auxiliary::CrashType)0>::forwardTuple(this=0x000000010756a900) at AuxiliaryAlgebra.cpp:372:14
    frame #1: 0x00000001018b8dc7 SecondoBDB`int auxiliary::InjectFault<(auxiliary::CrashType)0>(args=0x0000000109852e40, result=0x00007ffeefbfb4e0, message=2, local=0x0000000109853808, s=0x0000000109852e00) at AuxiliaryAlgebra.cpp:415:13
    frame #2: 0x000000010016d95f SecondoBDB`QueryProcessor::Eval(this=0x000000010746c820, node=0x0000000109852e00, result=0x00007ffeefbfb4e0, message=2) at QueryProcessor.cpp:4492:29
    frame #3: 0x000000010016f5b2 SecondoBDB`QueryProcessor::Request(this=0x000000010746c820, s=0x0000000109852e00, result=0x00007ffeefbfb4e0) at QueryProcessor.cpp:4689:3
    frame #4: 0x0000000100316321 SecondoBDB`TCountStream(args=0x0000000109810640, result=0x00007ffeefbfbb20, message=1, local=0x0000000109811008, s=0x0000000109810600) at RelationAlgebra.cpp:2095:11
    frame #5: 0x000000010016d95f SecondoBDB`QueryProcessor::Eval(this=0x000000010746c820, node=0x0000000109810600, result=0x00007ffeefbfbb20, message=1) at QueryProcessor.cpp:4492:29
    frame #6: 0x000000010016c91d SecondoBDB`QueryProcessor::EvalP(this=0x000000010746c820, node=0x0000000109810600, result=0x00007ffeefbfbb20, message=1) at QueryProcessor.cpp:4143:5
    frame #7: 0x0000000100012b0b SecondoBDB`SecondoInterfaceTTY::Command_Query(this=0x0000000107582190, list=5, resultList=0x00007ffeefbfe0e0, errorMessage="", autoTransaction=true) at SecondoInterfaceTTY.cpp:1915:11
    frame #8: 0x000000010000d0ce SecondoBDB`SecondoInterfaceTTY::Secondo(this=0x0000000107582190, commandText="query plz feed faultcrash[100] count", commandLE=0, commandLevel=1, commandAsText=false, resultAsText=false, resultList=0x00007ffeefbfe0e0, errorCode=0x0000000107580ab0, errorPos=0x00007ffeefbfe0f4, errorMessage="", resultFileName="SecondoResult", isApplicationLevelCommand=true) at SecondoInterfaceTTY.cpp:1555:19
    frame #9: 0x000000010008ac1a SecondoBDB`SecondoTTY::CallSecondo(this=0x0000000107580910) at SecondoTTY.cpp:577:9
    frame #10: 0x0000000100089f5c SecondoBDB`SecondoTTY::CallSecondo2(this=0x0000000107580910) at SecondoTTY.cpp:629:12
    frame #11: 0x0000000100089c26 SecondoBDB`SecondoTTY::ProcessCommand(this=0x0000000107580910) at SecondoTTY.cpp:422:5
    frame #12: 0x0000000100088bf6 SecondoBDB`SecondoTTY::ProcessCommands(this=0x0000000107580910, stopOnError=false, isPD=false) at SecondoTTY.cpp:481:15
    frame #13: 0x000000010008b523 SecondoBDB`SecondoTTY::Execute(this=0x0000000107580910) at SecondoTTY.cpp:721:7
    frame #14: 0x000000010008be9a SecondoBDB`SecondoTTYMode(tp=0x00007ffeefbfeef8) at SecondoTTY.cpp:880:24
    frame #15: 0x00000001000851cd SecondoBDB`main(argc=1, argv=0x00007ffeefbff050) at MainTTY.cpp:164:10
    frame #16: 0x00007fff2035d631 libdyld.dylib`start + 1
(lldb) quit
Quitting LLDB will kill one or more processes. Do you really want to proceed: [Y/n] y

This is the wiki of the SECONDO project.

⚠️ **GitHub.com Fallback** ⚠️