Introduction

To deal with tricky bugs or unexpected behaviors of such a complex system as Linux kernel is never an easy task. What might help a lot is ability to capture some critical events for at least postmortem analysis.

What's important though capturing of events must be as less invasive as possible so we don't alter normal execution process a lot. Thus obviously printk() is a no go as in case of the kernel it triggers execution of way too much code as well as touches peripherals, so that more unwanted IRQs happen etc.

And our solution is to dump events just in memory buffer in form of predefined data structures. This way amount of extra instructions being executed and memory space used is minimal. That's our structure:

typedef struct {
	int cpu;
	unsigned long time;
	enum arc_event event;
	unsigned int cause;
	unsigned int  stat32;
	unsigned int pc;
	unsigned int efa;
	unsigned int extra; /* Traps: Syscall num, Intr: IRQ, Excep */
	unsigned int task;
	unsigned int sp;
} timeline_log_t;

We dump those structures in statically-allocated array timeline_log.

And now the most interesting part - how do we human beings read/interpret that binary dump? We just ask debugger to do interpretation as it knows how that structure looks like and even which definitions mean what. Just open timeline_log in MDB's Global Variables view:

How can I get it?

Unfortunately that's not a part of vanilla Linux kernel sources at least as of today and to get that functionality it's required to get Linux source tree with corresponding patches from Vineet's tree (https://git.kernel.org/pub/scm/linux/kernel/git/vgupta/arc.git/log/?h=topic-event-logger):

git clone -b topic-event-logger https://git.kernel.org/pub/scm/linux/kernel/git/vgupta/arc.git

And enable the logger selecting ARC_DBG_EVENT_TIMELINE in kernel's menuconfig.