Xen - charlesfg/TPCx-V_setup GitHub Wiki
Xen
Instalation
PAckage Manager
I'll be using the Xen 4.4, so I'm basing the commands on that version :
sudo apt-get install -y xen-hypervisor-4.4-amd64 xen-tools libosinfo-bin
- Kernel after/before instalation of Xen
Previous: 4.4.0-45-generic #66-Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016
After: 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016
From Sources
-
Following the instruction on
-
:warning: For building xen 4.4.1 I choosed to use the ubuntu 14.04 since it has the 4.9 version of GCC that leads to less error when compiling the old XEN version :warning:
-
Adding dependencies :
- Run above script
set -o errexit
sudo apt-get -y install build-essential
sudo apt-get -y install bcc bin86 gawk bridge-utils iproute libcurl3 libcurl4-openssl-dev bzip2 module-init-tools transfig tgif
sudo apt-get -y install texinfo texlive-latex-base texlive-latex-recommended texlive-fonts-extra texlive-fonts-recommended pciutils-dev mercurial libjpeg-dev
sudo apt-get -y install make gcc libc6-dev zlib1g-dev python python-dev python-twisted libncurses5-dev patch libvncserver-dev libsdl-dev
sudo apt-get -y install iasl libbz2-dev e2fslibs-dev git-core uuid-dev ocaml ocaml-findlib libx11-dev bison flex xz-utils libyajl-dev
sudo apt-get -y install gettext libpixman-1-dev libaio-dev markdown pandoc fakeroot
sudo apt-get install -y libc6-dev-i386
sudo apt-get install -y libssl-dev
- clone, configure and install
git clone git://xenbits.xen.org/xen.git
git checkout RELEASE-4.4.1
cd ~
git clone [email protected]:nuno/xeratool.git
cd xen
patch -p1 < ../xeratool/hInjector.patch
find . -name \*.\[ch\] -exec grep 'hinjector_filt' {} \; | wc -l
./configure --enable-systemd
make world
make install
sudo /sbin/ldconfig
sudo update-rc.d xencommons defaults 19 18
sudo update-rc.d xendomains defaults 21 20
sudo update-rc.d xen-watchdog defaults 22 23
dpkg-divert --divert /etc/grub.d/08_linux_xen --rename /etc/grub.d/20_linux_xen
sudo update-grub
awk '/menuentry/ && /class/ {count++; print count-1"****"$0 }' /boot/grub/grub.cfg | grep -i "Xen"
# update grub with the apropriate index (in this case 3)
sudo sed -i 's/GRUB_DEFAULT=[0-9]/GRUB_DEFAULT=3/' /etc/default/grub
sudo update-grub
# reboot
Checking Xen Instalation
egrep '(vmx|svm|hypervisor)' /proc/cpuinfo
xl dmesg
Configure lvm
# Install lvm
sudo apt-get install lvm2
# Check which block to use
lsblk -o "NAME,MAJ:MIN,RM,SIZE,RO,FSTYPE,MOUNTPOINT,UUID"
# sdc/1 sdd/1
sudo pvcreate /dev/sdc1
sudo vgcreate vg0 /dev/sdc1
sudo lvcreate -n tpc-tenant-disk -L 30G vg0
sudo lvcreate -n tpc-tenant-swap -L 4G vg0
sudo mkswap /dev/vg0/tpc-tenant-swap
Xen Tunning
The Dom0 VM needs tunning since it will load a lot of resources by default. For instances using a host with 96GB of Ram and 24 CPU the Dom0 was getting, by default, 39GB of Ram and 24 vcpus which was not needed. It might a good idea dedicate a cpu core especially for systems running IO intensive guests.
- add the line for
/etc/default/grub, then run update-grub and reboot
GRUB_CMDLINE_XEN_DEFAULT="dom0_mem=2048M,max:2048M dom0_max_vcpus=4 dom0_vcpus_pin"
- Prevent the dom0 to ballooned down the memory if no memory is available for creating new guests
sed -i '/autoballoon/s/^#\?autoballoon.*/autoballoon=0/' /etc/xen/xl.conf
- Exclude the pinned vcpus for being used from the guest VMs
sed -i 's/^\(vcpus.*\)/\1\ncpus="all,^0-3"/g' tpc*.cfg
Afert rebooting and starting all the VMs check for the allocated cpus and memory available for dom0
# show list only 2GB allocated and exclusive allocation for the vcpus
free -h && xl vcpu-list
root@oxum:/var/tpcv/tpc_repo/xen_install# free -h && xl vcpu-list
total used free shared buffers cached
Mem: 1,9G 345M 1,5G 1,2M 16M 79M
-/+ buffers/cache: 248M 1,6G
Swap: 95G 0B 95G
Name ID VCPU CPU State Time(s) CPU Affinity
Domain-0 0 0 0 -b- 14.7 0
Domain-0 0 1 1 -b- 9.6 1
Domain-0 0 2 2 r-- 10.1 2
Domain-0 0 3 3 -b- 9.9 3
tpc-drive 1 0 17 -b- 10.5 4-23
tpc-drive 1 1 4 -b- 8.5 4-23
tpc-g1a 2 0 13 -b- 9.7 4-23
tpc-g1a 2 1 18 -b- 7.3 4-23
tpc-g1b1 3 0 14 -b- 10.0 4-23
tpc-g1b1 3 1 11 -b- 9.1 4-23
tpc-g1b2 4 0 16 -b- 10.1 4-23
tpc-g1b2 4 1 6 -b- 8.8 4-23
tpc-g2a 5 0 15 -b- 9.8 4-23
tpc-g2a 5 1 16 -b- 7.0 4-23
tpc-g2b1 6 0 15 -b- 10.1 4-23
tpc-g2b1 6 1 12 -b- 8.8 4-23
tpc-g2b2 7 0 5 -b- 10.4 4-23
tpc-g2b2 7 1 14 -b- 8.0 4-23
tpc-g3a 8 0 17 -b- 9.6 4-23
tpc-g3a 8 1 19 -b- 8.8 4-23
tpc-g3b1 9 0 7 -b- 9.5 4-23
tpc-g3b1 9 1 9 -b- 9.4 4-23
tpc-g3b2 10 0 7 -b- 8.6 4-23
tpc-g3b2 10 1 13 r-- 8.0 4-23
tpc-g4a 11 0 21 -b- 3.8 4-23
tpc-g4a 11 1 23 -b- 3.2 4-23
tpc-g4b1 12 0 8 -b- 3.5 4-23
tpc-g4b1 12 1 10 -b- 1.5 4-23
tpc-g4b2 13 0 20 -b- 1.6 4-23
tpc-g4b2 13 1 17 -b- 1.9 4-23
- https://wiki.xenproject.org/wiki/Xen_Project_Best_Practices
- https://wiki.xen.org/wiki/Tuning_Xen_for_Performance
Network Config:
- http://blog.manula.org/2012/04/manually-configuring-nat-networking-in.html
- http://unix.stackexchange.com/questions/108870/creating-a-private-network-for-internal-vms
- http://askubuntu.com/questions/119393/how-to-save-rules-of-the-iptables
- add to /etc/network/interfaces
# Xen bridge - you can also use the 192.168.0.0
auto xenbr0
iface xenbr0 inet static
bridge_ports none
bridge_stp no
address 10.0.0.1
netmask 255.255.255.0
network 10.0.0.0
broadcast 10.0.0.255
- then:
#sudo sysctl -w net.ipv4.ip_forward=1 also edit /etc/sysctl.conf
sudo sysctl -w net.ipv4.ip_forward=1
sudo iptables -A FORWARD --in-interface xenbr0 -j ACCEPT
sudo iptables --table nat -A POSTROUTING --out-interface XXXX -j MASQUERADE
sudo apt-get install iptables-persistent
sudo /etc/init.d/iptables-persistent save
sudo /etc/init.d/iptables-persistent reload
# Inside the Gest:
ifconfig eth0 10.0.0.2 up
route add default gw 10.0.0.1
Configuring the tpc-drive externally
- References:
Infos:
- tpc-drive IP: 150.164.203.70
- oxum IP: 150.164.203.68
# on tpc-drive dinamically
route add -net 10.0.0.0 netmask 255.255.255.0 gw 150.164.203.68
# OR
# statically
echo '10.0.0.0/24 via 150.164.203.68' >> /etc/sysconfig/network-scripts/route-eth0
runatall 'sed -i "/tpc-drive/ {s/10.0.0.30/150.164.203.70/g;}"' /etc/hosts
sed -i "s/tpc-drive\"/tpc-driver\"/g" vcfg.properties
Tunning TCP Small Queue
- Enabling Single Flow Throughput
echo 1048576 > /proc/sys/net/ipv4/tcp_limit_output_bytes
- Rolling back
echo 262144 > /proc/sys/net/ipv4/tcp_limit_output_bytes
Creating images
root@oxum:/home/ubuntu# lvcreate -L8G -n tpc0-centos7-disk oxum-vg
Logical volume "tpc0-centos7-disk" created
root@oxum:/home/ubuntu# lvcreate -L4G -n tpc0-centos7-swap oxum-vg
Logical volume "tpc0-centos7-swap" created
- Following this guide:
Using xen-create-image
sudo xen-create-image --hostname=test \
--memory=512mb \
--vcpus=1 \
--lvm=cfgvirt-vg \
--dhcp \
--pygrub \
--dist=trusty
# For compiling Xen
sudo xen-create-image --hostname=ubuntu14.04 \
--memory=2gb \
--vcpus=2 \
--lvm=xubuntu-vg \
--size=10gb \
--dhcp \
--bridge=virbr0 \
--pygrub \
--dist=trusty
- Then :
sudo xl create /etc/xen/testxn4.cfg
sudo xl console testxn4
- Note: The command above failed with the error:
libxl: error: libxl_device.c:1138:device_hotplug_child_death_cb: script: Could not find bridge device xenbr0. Then if checked that the configured bridge was the virbr0 (brctl show) and changed the/etc/xen/testxn4.cfgto have the vif line tovif = [ 'mac=00:16:3E:CA:B6:65,bridge=virbr0' ]
Xen References
Cloning images
- Based and helped by: https://ycnrg.org/xen-install-os-from-iso-pv/
#!/bin/bash
set -o errexit
declare -A IP_ADDR # Create an associative array
IP_ADDR[tpc-g1a]=10.0.0.31
IP_ADDR[tpc-g1b1]=10.0.0.32
IP_ADDR[tpc-g1b2]=10.0.0.33
IP_ADDR[tpc-g2a]=10.0.0.34
IP_ADDR[tpc-g2b1]=10.0.0.35
IP_ADDR[tpc-g2b2]=10.0.0.36
IP_ADDR[tpc-g3a]=10.0.0.37
IP_ADDR[tpc-g3b1]=10.0.0.38
IP_ADDR[tpc-g3b2]=10.0.0.39
IP_ADDR[tpc-g4a]=10.0.0.40
IP_ADDR[tpc-g4b1]=10.0.0.41
IP_ADDR[tpc-g4b2]=10.0.0.42
for VM in g1a g2a g3a g4a g1b1 g1b2 g2b1 g2b2 g3b1 g3b2 g4b1 g4b2;
do
echo "Cloning the tpc-${VM}";
if [ ${VM} == g1a ](/charlesfg/TPCx-V_setup/wiki/|-${VM}-==-g1b1-);
then
echo "Skiping the configured firsts VM"
continue
fi
lvcreate -L8G -n tpc-${VM}-disk oxum-vg
dd if=/dev/oxum-vg/tpc0-centos7-disk of=/dev/oxum-vg/tpc-${VM}-disk bs=512K
lvcreate -L4G -n tpc-${VM}-swap oxum-vg
dd if=/dev/oxum-vg/tpc0-centos7-swap of=/dev/oxum-vg/tpc-${VM}-swap bs=512K
partprobe
kpartx -al /dev/oxum-vg/tpc-${VM}-disk
kpartx -al /dev/oxum-vg/tpc-${VM}-swap
mount /dev/mapper/oxum--vg-tpc--${VM}--disk1 /mnt/tpc-clone
cd /mnt/tpc-clone
sed -i 's/tpc0/tpc-${VM}/' etc/hostname
# adding the tpc hosts to the hosts files
fgrep '10.0.0.' /etc/hosts >> etc/hosts
sed -i 's/10.0.0.20/${IP_ADDR[$L]}/' etc/sysconfig/network-scripts/ifcfg-eth0
cd -
umount /mnt/tpc-clone
cd /var/tpcv/tpc_repo/xen_install/
cp tpc0-centos7.cfg tpc-${VM}-centos7.cfg
sed -i 's/tpc0/tpc-${VM}/g' tpc-${VM}-centos7.cfg
sed -i 's/-centos7//g' tpc-${VM}-centos7.cfg
sed -i "s/00:16:3E:29:QQ:QQ/$(bash gen-mac.sh)/" tpc-${VM}-centos7.cfg
if [ ${VM} == g[[:digit:](/charlesfg/TPCx-V_setup/wiki/-${VM}-==-g[[:digit:)b* ]];
then
echo "Adding the dbstore space for the database VMs"
sed "s/xvdc,rw'/xvdc,rw',\n\t'phy:\/dev\/oxum-vg\/tpc_${VM}-dbstore,xvdd,rw'/" tpc-${VM}-centos7.cfg
fi
done
# if ubuntu 16.04
rm /usr/bin/gcc
ln -s /usr/bin/gcc-4.9 /usr/bin/gcc
CC=gcc-4.9 ./configure --enable-systemd
CC=gcc-4.9 make world
Issues:
xl: error while loading shared libraries:
The reason is where the library is located. Problem in configuration before build!
root@oxum:/home/ubuntu# xl info
xl: error while loading shared libraries: libxlutil.so.4.3: cannot open shared object file: No such file or directory
root@oxum:/home/ubuntu#
- http://xen.1045712.n5.nabble.com/Install-Xen-4-4-Unstable-td5719417.html
- https://codeyarns.com/2014/01/14/how-to-add-library-directory-to-ldconfig-cache/
Do not Install xen-tools from package after installing xen from source
# The overhidding of those files corrupt the xen
/etc/init.d/xendomains
/etc/xen/scripts/qemu-ifup
/etc/xen/scripts/hotplugpath.sh
/etc/default/xendomains
Other Misc References:
TODOs Desesperadores:
- Tentar o virsh novamente com o --connect qemu:///system \ (https://www.cyberciti.biz/tips/tag/xen)
for i in xl list | cut -f1 -d' ' | grep tpc-; do echo yes | ssh postgres@${i} pwd; done