In this page, you will find all useful details for an user to go further in this project use. If you want to learn more or to help, we advise you to read the developers documentation afterward.

All the information below can be found in our User manual.

Table of content :

• Python script entry point : kernel_generator.py
• Sample generation workflow
• Database structure : what do we retrieve?
• Advanced use case : Compilation inside Docker Tuxml

• This standalone script is a way for the user to use this project with ease, without any need to go through all the project and understand everything.
• This script provides to the user a simpler usage of our docker image, by managing all by himself the fetch and building, providing also a way to use different versions of the Linux kernel (this part is still in development and we can’t assure that every single Linux kernel version 4.x.x will work at the moment), while also providing a way to use the stable (prod) or the latest (dev) version of the image.
• This script also gives an easy way to the user to test a specific Linux config file and fetch its compilation logs, to compile a whole bunch of random config or to simply check if the project should not crash on your system.

$ ./kernel_generator.py --help
usage: kernel_generator.py [-h] [--boot] [--checksize] [--dev] [--local]
                           [--tiny] [--config CONFIG] [--seed SEED]
                           [--linux4_version LINUX4_VERSION] [--logs LOGS]
                           [-s] [--unit_testing] [-n NUMBER_CPU]
                           [nbcontainer] [incremental]

positional arguments:
  nbcontainer           Provide the number of container to run. Have to be
                        over 0.
  incremental           Optional. Provide the number of additional incremental
                        compilation. Have to be 0 or over.

optional arguments:
  -h, --help            show this help message and exit
  --dev                 Use the image with dev tag instead of prod's one.
  --local               Don't try update the image to run, i.e. use the local
                        version.
  --tiny                Use Linux tiny configuration. Incompatible with
                        --config argument.
  --boot                Optional. Try to boot the kernel after compilation if
                        the compilation has been successful
  --checksize           Optional. Compute additional size measurements on the
                        kernel and send the results to the 'sizes' table (can
                        be heavy).
  --config CONFIG       Give a path to specific configuration file.
                        Incompatible with --tiny argument.
  --seed SEED           Give a path to a specific seed options file. These
                        options will be activated before the others are
                        randomly chosen. The file will replace tuxml.config
  --linux4_version LINUX4_VERSION
                        Optional. Give a specific linux4 version to compile.
                        Note that its local, will take some time to download
                        the kernel after compiling, and that the image use to
                        compile it will be deleted afterward.
  --logs LOGS           Optional. Save the logs to the specified path.
  -s, --silent          Prevent printing on standard output when compiling.
                        Will still display the feature warning.
  --unit_testing        Optional. Run the unit testing of the compilation
                        script. Prevent any compilation to happen. Will
                        disable --tiny, --config, --linux4_version, --silent,
                        --fetch_kernel and incremental feature during runtime.
  -n NUMBER_CPU, --number_cpu NUMBER_CPU
                        Optional. Specify the number of cpu cores to use while
                        compiling.Useful if your computer can't handle the
                        process at full power.

Here are some examples to get you started:

• A simple compilation: kernel_generator.py (nothing else !)
• With unit testing : kernel_generator.py –unit-testing
• With a custom .config file: kernel_generator.py –config <path/to/.config>
• Running 100 compilations on 2 cpu cores: kernel_generator.py 100 -n 2 or kernel_generator.py 100 –number_cpu 2
• Running 4 compilations on 16 cpu cores with kernel version 4.15.1: kernel_generator.py 4 -n 16 –linux4_version 15.1

Work in progress

In order to work, the TuxML project needs an access to the internet, and more specifically to the database server holding all the data.

Above is a graph showing those data. As you can see the only things concerning your hardware is your machine’s CPU, RAM and Storage type (HDD, SDD ...) in the hardware_environment table. As for the software environment we save on which Os and Kernel version it was done (in the software_environment). The rest of the fields should stay consistent across all rows because those data are gathered inside the container.

One table is missing on the above graph for it had been added afterward, the 'sizes' table that contains the output of some bash commands to get extra information concerning the size of the kernel (with --checksize option). Table content :

• 'cid' (primary) referring to cid in 'compilations'
• 'size_vmlinux' -> output of 'size vmlinux' command (text)
• 'nm_size_vmlinux' -> output of 'nm --size -r vmlinux' (LONGBLOB)
• 'size_builtin' -> output of 'size */built-in.o' (LONGBLOB)

Here is some more information about these commands

Now about the data needed for the machine learning process, all of them are contained in the 4 remaining tables (compilations, boot, sizes and incrementals compilations relations), here we save everything related to the time taken to compile the configuration, the config file, different size measurements and even the compressed kernel sizes. All of this will help to predict some aspects of the kernel depending on the configuration.

(Related to issue #140) TODO: maybe add some screenshots

The goal of this tutorial is to use the docker container in order to build a specific kernel configuration, in order to do that we need to manipulate the container.

First of all we need to install the docker image locally if it isn't done yet. using this command you can download the image:

$ python3 kernel_generator.py 1 --tiny 

first we need to run the container based on the tuxml/tuxml:prod, then copy the file, you can use this command to run a container:

$ docker run -it tuxml/tuxml:prod

(to list all images use: $ docker images or $ docker image ls -> same output)

this will open you a shell inside the container, from that point you could exit the container to start it again then copy then attach. or you could simply open another terminal, list the running containers, and then copy your file in the right one.

• open another terminal
• use $ docker container ls to find your container's id or alias (funny two words names).
• then: $ docker cp <file> <container-id>:/<rest of the path>

so as an example:

$ docker cp .config objective_montalcini:/TuxML/linux-4.13.3/.config

The other method (one terminal):

• exit the shell with the "exit" command.
• restart the container, to know which one it is used $ docker container ls -a, once you find it use: $ docker restart <container-id>
• then: $ docker cp <file> <container-id>:/<rest of the path> Example:

$ docker cp .config objective_montalcini:/TuxML/linux-4.13.3/.config"

and then do what you have to do, that is installing dependencies through apt or using make on the kernel to see what happens.

(as an example if you need git: "apt install git")