Guide for Xenomai build, setup and test on top of Ubuntu&Debian (Xenomai编译安装和测试指导) - intel/linux-stable-xenomai GitHub Wiki

本文档讲述了如何基于Ubuntu 18.04/Debian 10 搭建一个xenomai测试环境, 您也可以选择等其他Linux发行版本进行实验。 Xenomai 3.x 官方的内核制作和安装文档请阅读这里: Setting_Up
这个wiki指导是官方文档的一个简化版, 重点针对 x86_64 uefi 硬件平台。

2. 环境配置

安装 Ubuntu Desktop 18.04 https://ubuntu.com/tutorials/tutorial-install-ubuntu-desktop

[可选]如果您的网络环境需要设置代理才能运行apt, 在Ubuntu 18.04上, 可以把代理的环境变量添加到/etc/environment
首先检查一下当前shell是否有有效的http_proxy, https_proxy等环境变量:

$env | grep -i proxy
http_proxy=http://xxx.abc.com:8080/
https_proxy=http://xxx.abc.com:8080/
no_proxy=localhost,127.0.0.1,

可以用如下命令直接添加相关环境变量到文件/etc/environment \

$sudo su
#printf "\n\nhttp_proxy=${http_proxy}\nhttps_proxy=${https_proxy}\nno_proxy=${no_proxy}\n\n" >> /etc/environment

安装相关工具集合，主要包括源代码下载工具，编译kernel所必须的工具和库。

sudo apt update
sudo apt install vim ssh
sudo apt install -y git build-essential libssl-dev libelf-dev iasl python nasm openssl gcc-multilib bison flex bc lz4

above package name should both work for Ubuntu 18.04 & Debian 10; Ubuntu 20.04 not tested yet.

Optional package may useful for developing but not required for kernel compile:

python python2.7 python3 python3-pip gawk wget diffstat curl unzip texinfo chrpath socat libsdl1.2-dev xterm cpio xz-utils iputils-ping whiptail

3. 源代码下载

从如下的github仓库获取完整的xenomai源代码。 https://github.com/intel/linux-stable-xenomai

mkdir workspace && cd workspace
git clone https://github.com/intel/linux-stable-xenomai
cd linux-stable-xenomai
git checkout F/4.19.59/base/ipipe/xenomai_3.1    ### use branch "RC/5.4.61/base/ipipe/xenomai_next" for kernel 5.4.x

4. 软件包构建

我们制作一个简单的bash脚本,简化了手工编译的过程:

bash build-deb.sh --help           ### help
bash build-deb.sh                  ### build - without clean previous output
bash build-deb.sh --clean-build    ### build - clean previous output before build
bash build-deb.sh --clean-all      ### clean - clean previous output then exit

if meet such error when build your own version:

*** No rule to make target 'debian/certs/debian-uefi-certs.pem', needed by 'certs/x509_certificate_list'.  Stop.

modify CONFIG_SYSTEM_TRUSTED_KEYS by:

sed -ri '/CONFIG_SYSTEM_TRUSTED_KEYS/s/=.+/=""/g' ${KBUILD_OUTPUT}/.config

如果编译成功，将在'../'目录下生成如下deb软件包。

build

5. 新内核安装

在目标板上安装新生成的内核软件包。

sudo dpkg -i linux-headers-4.19.59-xenomai+_3.1_amd64.deb
sudo dpkg -i linux-image-4.19.59-xenomai+_3.1_amd64.deb

内核的deb包会自动调用update-grub更新'/etc/default/grub'文件, 但是我们还需要手工更新kernel boot parameters.

6. 配置内核启动参数 (假设CPU是 4CORE4THREAD or 4CORE8THREAD, 把实时程序限定隔离在core 2和core 3)

sudo nano /etc/default/grub

GRUB_CMDLINE_LINUX值更新为"nosmt noefi intel_idle.max_cstate=1 pstate_driver=no_hwp clocksource=tsc tsc=reliable rcu_nocb_poll rcu_nocbs=2-3 nohz_full=2-3 nmi_watchdog=0 nosoftlockup nosmap audit=0 irqaffinity=0,1 isolcpus=2-3 console=tty1 console=ttyS0,115200"
关闭 GRUB_TIMEOUT_STYLE选项
GRUB_TIMEOUT值更为10，代表等待10秒钟

不要忘记运行update-grub配置更新，使新GRUB_CMDLINE_LINUX生效。
在本例中，core2,3将被配置为隔离态，保留给后续的时延测试程序专用。相应的在后续"run_latency.sh"的脚本内，'taskset'命令指定了时延测试程序运行于core2 或者core3.

sudo update-grub

启动时选择正确的内核版本。

reboot

进入系统以后，可以用dmesg | grep -i xenomai 检查一下当前内核是否包含Xenomai：

$ dmesg | grep -i xenomai
[    0.000000] Linux version 4.19.59-xenomai+ (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #3.1 SMP PREEMPT
[    1.047397] [Xenomai] scheduling class idle registered.
[    1.047399] [Xenomai] scheduling class rt registered.
[    1.047466] I-pipe: head domain Xenomai registered.
[    1.048794] [Xenomai] allowing access to group 1234
[    1.048821] [Xenomai] Cobalt v3.1 [DEBUG]
               *** RTnet for Xenomai v3.1 ***

7. 测试工具安装 (on kernel 4.19.x)

从xenomai.org下载'xenomai-3.1.tar.bz2'软件压缩包，完成配置、编译及程序安装过程。时延测试程序'latency'将被安装在'/opt/xenomai-3.1/bin'子目录中。
在kernel 5.4.x上, 不能使用xenomai-3.1版本, 请下载xenomai最新的master branch或者next branch.

wget https://xenomai.org/downloads/xenomai/stable/xenomai-3.1.tar.bz2
tar xvf xenomai-3.1.tar.bz2 
cd xenomai-3.1
./configure --prefix=/opt/xenomai-3.1 --with-core=cobalt  --enable-smp  --enable-pshared 
make -j4
sudo make install
sudo /opt/xenomai-3.1/bin/latency   ### successfully run means installation good!

'glmark2' 被用于xenomai时延测试时的背景程序, 如果测试板可以联网，可直接使用'apt-get'完成安装。

sudo apt-get install glmark2

如果测试板无法联网，从以下链接可获取相应的deb软件包，完成手动安装。

libpng12-0_1.2.54-1ubuntu1_amd64.deb
glmark2-data_2014.03+git20150611.fa71af2d-0ubuntu2_all.deb
glmark2_2014.03+git20150611.fa71af2d-0ubuntu2_amd64.deb
msr-tools_1.3-2build1_amd64.deb

sudo dpkg -i libpng12-0_1.2.54-1ubuntu1_amd64.deb
sudo dpkg -i glmark2-data_2014.03+git20150611.fa71af2d-0ubuntu2_all.deb
sudo dpkg -i glmark2_2014.03+git20150611.fa71af2d-0ubuntu2_amd64.deb
sudo dpkg -i msr-tools_1.3-2build1_amd64.deb

8. 启动后配置

系统启动后, 也必须对系统状态做一些配置. 总的指导原则是关闭所有的休眠和省电相关选项, 让CPU,Memory,内部总线和其他组件以固定频率运行; 关闭所有可能产生高延迟的设置。同时要注意单板散热，不能关闭Thermal监控。如果散热不佳，应降低CPU频率运行.

In intel_idle driver, lock CPU C-State, and set governor to "performance":

### if not print "intel_idle", remove "poll=idle" in kernel boot parameters
cat  /sys/devices/system/cpu/cpuidle/current_driver
### should only have state0 and 1 after apply "intel_idle.max_cstate=1"
echo 1 > /sys/devices/system/cpu/cpu0/cpuidle/state1/disable
echo 1 > /sys/devices/system/cpu/cpu1/cpuidle/state1/disable
echo 1 > /sys/devices/system/cpu/cpu2/cpuidle/state1/disable
echo 1 > /sys/devices/system/cpu/cpu3/cpuidle/state1/disable
### disable turbo freq
echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo
### set governor
echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor

Lock CPU, GPU freq and disable SAGV (SAGV's formal name is System Agent Speedstep):

### adjust and lock CPU freq (to do)
### /sys/devices/system/cpu/cpu${i}/cpufreq/scaling_min_freq
### /sys/devices/system/cpu/cpu${i}/cpufreq/scaling_max_freq
### adjust and lock GPU freq
echo 700 > /sys/devices/pci0000:00/0000:00:02.0/drm/card0/gt_boost_freq_mhz
echo 700 > /sys/devices/pci0000\:00/0000\:00\:02.0/drm/card0/gt_max_freq_mhz
echo 700 > /sys/devices/pci0000\:00/0000\:00\:02.0/drm/card0/gt_min_freq_mhz
### disable SAGV at runtime
echo 1 > /sys/kernel/debug/dri/0/i915_sagv_disable

9. 测试例运行

下载以下运行脚本至目标板:

run_latency.sh:

#!/bin/bash
cpu="2"
cycle="250"
xenomai_dir="/opt/xenomai-3.1"
time_total="345600"
####################################################################
taskset -c $cpu $xenomai_dir/bin/latency -g ./latency.histo.log -s -p $cycle -c $cpu -P 99 -T $time_total 2>&1 | tee ./latency.log

run_glmark2.sh:

#!/bin/bash
amount="8"
##################################
i="0"
while [ $i -lt $amount ]
do
	glmark2 --run-forever &
	i=$[$i+1]
done

使用'chmod +x'命令为这些脚本文件添加可执行属性。在运行脚本前运行'uname -a'确认内核版本。

新打开一个terminal窗口, 执行 ./run_glmark2.sh
新打开一个terminal窗口, 执行 sudo ./run_latency.sh

最后一列, "lat worst" 是最重要的实时性测试指标,单位是微秒。

To avoid negative value for minimum latency, set xenomai “gravity” to 0,
$echo 0 > /proc/xenomai/clock/coreclk, or write your own value to the calibration file,
$echo ${value_in_ns} > /proc/xenomai/latency

10. 可能遇到的问题

Note: if "latency" program failed to run under root, please close group rt schedule policy use: sudo sysctl -w kernel.sched_rt_runtime_us=-1

11. 深入阅读

resource-director-technology
cache-allocation-technology (CAT)
intel-cmt-cat

12. 支持

如果有任何问题, 请提交issue, 或者发邮件至Xenomai邮件列表: [email protected], 谢谢!