Guide to collecting instrumentation information from qemu in userspace emulation - ruijiefang/llvm-hcs GitHub Wiki

Userspace qemu intercepts exit syscalls to run the exit handlers set by the target program; this causes qemu's own the exit handlers set by compiler-rt for writing collected instrumentation data not to be run. We can see how qemu handles syscalls of this type in linux-user/i386/cpu_loop.c's cpu_loop function:

        case EXCP_SYSCALL:
            /* linux syscall from syscall instruction */
            ret = do_syscall(env,
                             env->regs[R_EAX],
                             env->regs[R_EDI],
                             env->regs[R_ESI],
                             env->regs[R_EDX],
                             env->regs[10],
                             env->regs[8],
                             env->regs[9],
                             0, 0);
            if (ret == -TARGET_ERESTARTSYS) {
                env->eip -= 2;
            } else if (ret != -TARGET_QEMU_ESIGRETURN) {
                env->regs[R_EAX] = ret;
            }
            break;

We can see that when the target program raises a syscall exception (e.g. exit or exit_group), qemu invokes a syscall within its own emulation functionality and executes that syscall; qemu's own exit handlers are not executed (since eventually that syscall gets translated into an actual exit syscall in our host CPU, and the entire program, along with the userspace emulator, exits without invoking the emulator's own exit handlers).

The solution is simple: We test if the syscall is an exit syscall, and if so, we simply return from the cpu_loop function:

            /* ruijief: capture exit syscall here and call our own
             * exit handler instead of the target program's exit handler */
            if ((env->regs[R_EAX]== __NR_exit_group) 
                || (env->regs[R_EAX] == __NR_exit)) { /* return from loop */ return; }
            /* ruijief: otherwise, do a normal syscall. */

Here is the final code: https://github.com/ruijiefang/qemu-x86_64-linux-benchmark/blob/master/linux-user/i386/cpu_loop.c#L229. Now when the target program exits, qemu's cpu loop returns and causes qemu's main function to exit, executing qemu's own exit handlers. A side effect is the target program's exit handlers will not be properly executed in this case.

Note, that we're using <linux/unistd.h>'s syscall macros for __NR_exit_group and __NR_exit, since both host and target in our case is x86_64/i386. To make userspace emulation exit this way in qemu for other targets we need to use the macros provided in qemu's header files for the target platform.