Graph Layout - ncatlin/rgat GitHub Wiki

The current instruction (node) layout scheme (see positionVert in trace_handler.cpp) is basically just the first layout I tried that seemed relatively tidy. There are almost certainly much clearer ways of doing it, which would be most welcome.

The colour scheme is editable in the config file.

Lets look at this graph of a simple HelloWorld written in assembly

_main:
    0: mov     ebp, esp
    1: sub     esp, 4
    2: jmp stuff

• 'mov ebp, esp'(0) and 'sub esp, 4'(1) are sequential instructions which do not affect control flow. The are the yellow nodes, linked by short yellow lines (edges).

• 'jmp stuff'(2), displayed with its memory address 'jmp 0x401024' on the graph is our first control flow instruction. All jumps are displayed as red nodes.

This completes our first basic block: a sequence of sequential instructions terminated by a control flow instruction.

That takes us to line 30 in our source code, via a long diagonal yellow edge representing a jump to new code.

Next we have a 'call'(3). These are represented as purple nodes and link to their targets with vertical purple edges.

writestd:
    ; hStdOut = GetstdHandle( STD_OUTPUT_HANDLE)
    4: push    -11
    5: call    _GetStdHandle@4
    [snip]

stuff:
    3: call writestd
    

We see our push as expected, but it appears that our call (5) to GetStdHandle goes via an unexpected jmp (6).

This is because incremental linking was enabled, causing a jump thunk to be inserted for access to library code.

New we have the library code itself, represented as green nodes/edges referred to as externs.

2x GetStdHandle (0: STDOUT) +1saved

The '2x' prefix shows that there were 2 executions of GetStdHandle

The symbol 'GetStdHandle' is retrieved from its DLL (kernel32.dll, you can enable extern paths in the text options)

After that is our basic argument display. Argument 0 was the 'STDOUT' enum. At the moment this only supports a few arguments for a few functions that have been interesting to me for one reason or another. This is partly because displaying every argument of every DLL call would make a mess, but mainly because I just haven't gotten round to it yet. See Issue #3

Finally it ends with '+1saved', indicating that another argument was recorded, but not displayed for the sake graph cleanliness.

Now let's look at returns

We are not instrumenting and drawing the code inside GetStdHandle, because that would be slow and make a huge mess.

Instead the graph resumes at 'mov ebx, eax '(8) close to the instruction that called the external. Here rgat is sacrificing a clean layout for the sake of giving you a better idea of the structure of the code being executed.

rgat maintains an internal call stack which tries to pair calls (purple nodes/edges) with rets (orange nodes/edges). As these orange edges(usually...) lead to previously executed code, they are curved inside the sphere. New code (call/jump edges) go on the outer shell. This is how rgat attempts to solve the problem of horrible spiderweb control flow graphs

Not shown are white edges, which link to old code from instructions other than 'ret'.

An exception handled by a handler within instrumented code looks like this:

'eax' contains 0 but node 1080 tries to treat it as an address for a write operation. An exception occurs and execution moves to the start of the exception handling code with a cyan line marking the transition.