Distcc based distributed compilation enabling steps on edk2 - shijunjing/edk2 GitHub Wiki

Terminology:
My localhost is the machine to run the edk2 build, and I call it client machine.
Other machines in the network which accept the distcc jobs are the work servers, and I call them server machines.
Host machines = client machine + server machines.

0. Install distcc
$ sudo apt-get install distcc distccmon-gnome distcc-pump

1. Update the /etc/default/distcc configures in all host machines (both client and servers) as below. Then check distcc service status:
$ sudo vim /etc/default/distcc
STARTDISTCC=“true”
ALLOWEDNETS=“127.0.0.1 10.239.219.0/24” # add your network segment here. Only local network segment make sense.
LISTENER=""

$ sudo service distcc restart
$ sudo service distcc status
distcc.service – LSB: simple distributed compiler server
Loaded: loaded (/etc/init.d/distcc; generated)
Active: active (running) since Thu 2018-12-06 16:52:48 CST; 16h ago
Docs: man:systemd-sysv-generator(8)
Process: 3486 ExecStop=/etc/init.d/distcc stop (code=exited, status=0/SUCCESS)
Process: 3492 ExecStart=/etc/init.d/distcc start (code=exited, status=0/SUCCESS)
Tasks: 7 (limit: 4915)
CGroup: /system.slice/distcc.service
|- 3500 /usr/bin/distccd —pid-file=/var/run/distccd.pid —log-file=/var/log/distccd.log —daemon —allow 127
|-11518 /usr/bin/distccd —pid-file=/var/run/distccd.pid —log-file=/var/log/distccd.log —daemon —allow 127
|-11657 /usr/bin/distccd —pid-file=/var/run/distccd.pid —log-file=/var/log/distccd.log —daemon —allow 127
|-11783 /usr/bin/distccd —pid-file=/var/run/distccd.pid —log-file=/var/log/distccd.log —daemon —allow 127
|-11863 /usr/bin/distccd —pid-file=/var/run/distccd.pid —log-file=/var/log/distccd.log —daemon —allow 127
|-11983 /usr/bin/distccd —pid-file=/var/run/distccd.pid —log-file=/var/log/distccd.log —daemon —allow 127
`-12079 /usr/bin/distccd —pid-file=/var/run/distccd.pid —log-file=/var/log/distccd.log —daemon —allow 127

Dec 06 16:52:48 ub2-uefi-b02 systemd¹: Starting LSB: simple distributed compiler server…
Dec 06 16:52:48 ub2-uefi-b02 distcc³⁴⁹²: * Starting Distributed Compiler Daemon: distccd
Dec 06 16:52:48 ub2-uefi-b02 distcc³⁴⁹²: …done.
Dec 06 16:52:48 ub2-uefi-b02 systemd¹: Started LSB: simple distributed compiler server.

2. In the client machine, append the host mahcines in the /etc/distcc/hosts file as below:
(I use total 4 Ubuntu 18.04 VMs here)
$ sudo vim /etc/distcc/hosts
// Add a list of hostnames in one line, seperated by spaces, here.
// +zeroconf
localhost
10.239.219.159
10.239.219.160
10.239.219.211

3. Create the soft links for edk2 build bins, and map the compilers to distcc
$ mkdir distcc_ln/ && cd distcc_ln/

$ ln -s /usr/bin/distcc clang
$ ln -s /usr/bin/distcc clang++
$ ln -s /usr/bin/distcc gcc
$ ln -s /usr/bin/distcc g++
$ ln -s /usr/bin/ar ar
$ ln -s /usr/bin/gcc-ar gcc-ar
$ ln -s /usr/bin/llvm-ar llvm-ar
$ ln -s /usr/bin/objcopy objcopy

$ export CLANG38_BIN=~/wksp_efi/distcc/distcc_ln/
$ export GCC5_BIN=~/wksp_efi/distcc/distcc_ln/
$ export GCC49_BIN=~/wksp_efi/distcc/distcc_ln/

4. Open the GTK monitor for distcc in another terminal
$ distccmon-gnome

5. Build edk2 OVMF and check the distccmon-gnome window
$ source edksetup.sh
$ build -p OvmfPkg/OvmfPkgIa32X64.dsc -t CLANG38 -a IA32 -a X64 -b NOOPT -n 50 -DDEBUG_ON_SERIAL_PORT -DSMM_REQUIRE=TRUE -DSECURE_BOOT_ENABLE=TRUE -DNETWORK_IP6_ENABLE=TRUE -DHTTP_BOOT_ENABLE=TRUE -DTPM2_ENABLE
(note I set -n 50, not 5 here )

Current edk2 build multi-thread bugs which block distributed compilation performance:

• Current edk2 build is only multi-thread on modules level, not on files level. The current parallel granularity is too coarse. This is not good to fully make use of existing tool of distributed compilation, which are usually designed for files level distribution. In my testing, most of C files cannot be dispatched by distcc. The distcc is designed to distribute files compilation paralleled by “-j n”, but current edk2 module parallel option “-N n” != files parallel option “-j n”, and limited the build as single thread in module internal.
• Besides above single thread bug in module internal, I found the module level multiple process Scheduler is very slow. I suspect there are also some bugs in the module level multiple process Scheduler. In my testing (see below the distccmon-gnome views which monitor the compiler invoking by distcc and the running history on distributed machines, all my VMs are same and have 4 cores), the modules are not fully paralleled to build, and it seems only beginning several scheduler iterations can really launch paralleled modules. It is clear that distcc cannot well assign edk2 compiling task to other machines. I need more time to debug distcc to understand what makes distcc unhappy to assign tasks to other machines.

This is a small module scope ovmf build with distcc enabled. You can see the compiler is not invoked in parallel in most time. Looks very bad.
$ build -p OvmfPkg/OvmfPkgIa32X64.dsc -t GCC5 -a IA32 -a X64 -b NOOPT -n 50

This is a bigger module scope ovmf build with distcc enabled. And it looks worse!
$ build -p OvmfPkg/OvmfPkgIa32X64.dsc -t CLANG38 -a IA32 -a X64 -b NOOPT -n 50 -DDEBUG_ON_SERIAL_PORT -DSMM_REQUIRE=TRUE -DSECURE_BOOT_ENABLE=TRUE -DNETWORK_IP6_ENABLE=TRUE -DHTTP_BOOT_ENABLE=TRUE -DTPM2_ENABLE

To better explain what is my expected performance, I apply distcc on the LLVM/CLANG build in my same 4 VMs compiler farm. Below is the result:

• Disable distcc and build in localhost machine:
  

  $ cmake ../llvm -G "Unix Makefiles" -DCMAKE_BUILD_TYPE="Release" -DLLVM_TARGETS_TO_BUILD="X86" -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_CXX_COMPILER="/usr/bin/g++" -DCMAKE_C_COMPILER="/usr/bin/gcc" -DLLVM_OPTIMIZED_TABLEGEN=ON -DLLVM_BINUTILS_INCDIR=/home/jshi19/llvm/binutils-2.31/include -DLLVM_INSTALL_UTILS=ON -DLLVM_ENABLE_ASSERTIONS=ON -DCMAKE_C_FLAGS=-DLLVM_ENABLE_DUMP -DCMAKE_CXX_FLAGS=-DLLVM_ENABLE_DUMP -DCMAKE_INSTALL_PREFIX=../releaseinstall/

  
  $ time make -j 8
  real 45m10.414s (2710.414)
  user 169m47.785s
  sys 8m41.075s

• Enable distcc distributed compilation acceleration:
  

  $ cmake ../llvm -G "Unix Makefiles" -DCMAKE_BUILD_TYPE="Release" -DLLVM_TARGETS_TO_BUILD="X86" -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_CXX_COMPILER="/home/jshi19/wksp_efi/distcc/distcc_ln/g++" -DCMAKE_C_COMPILER="/home/jshi19/wksp_efi/distcc/distcc_ln/gcc" -DLLVM_OPTIMIZED_TABLEGEN=ON -DLLVM_BINUTILS_INCDIR=/home/jshi19/llvm/binutils-2.31/include -DLLVM_INSTALL_UTILS=ON -DLLVM_ENABLE_ASSERTIONS=ON -DCMAKE_C_FLAGS=-DLLVM_ENABLE_DUMP -DCMAKE_CXX_FLAGS=-DLLVM_ENABLE_DUMP -DCMAKE_INSTALL_PREFIX=../releaseinstall/ -DLLVM_ENABLE_PROJECTS="lld;clang;compiler-rt;clang-tools-extra"

  
  $ time make -j 24
  real 13m44.680s (824.68s)
  user 33m6.457s
  sys 3m1.510s
  Below is the distccmon-gnome view when build LLVM with distcc enabled. You can see the compilers are invoked and run in parallel in most time. This is very good.
  

And result: distcc normal model can accelerate the LLVM build time by (2710.414 – 824.68)/2710.414 = 69.5% !!

I do believe distcc can accelerate edk2 build by more than 50% (especially combining the compiler cache technique, e.g. ccache, https://github.com/shijunjing/edk2/wiki/Edk2-compiler-cache-enabling-steps-on-Linux). We need to make it happen!