Building a Development VM for the HaLVM

This guide is no longer maintained. Please see HACKING.md for the information.

Purpose For This Guide

This guide will walk you through building a virtual machine in which to develop with the HaLVM. These instructions have been written using VMWare Fusion 7.1.2 on a Mac. In the past, steps very similar to the ones I'm proposing have worked with with VMWare for Linux, VMWare for Windows, and VirtualBox. However, you may need to do things in a slightly different order. Good luck! If you have any suggestions, or if you want to add caveats for those other systems, please consider either filing a bug report or simply editing this wiki page yourself.

This guide was used to build the virtual machine distributed to those attending the "Introduction to Unikernels with the HaLVM" workshop, given at StrangeLoop 2015.

Note: This whole process will take at least a couple hours with a reasonably fast network connection and disk.

Step #1: Set up the Virtual Machine Container

Creating the New Virtual Machine in VMWare

First, let's add a new virtual machine.

VMWare Fusion really wants you to install from a disk / image, or import an existing PC. Let's not do that. Instead, select "More options ...". At the next screen, we'll "Create a custom virtual machine," and then hit the continue button. When it asks us to select the operating system select Other and then Other 64-bit. This is very important. It may be very tempting to know that we're going to install Fedora at some point, and instead pick that specific option from the Linux menu. Do not be tempted! The truth is that the OS we are going to install is Xen, not Linux, and the whole VM-within-a-VM stack tends to fall apart unless you choose Other 64-bit.

Important warning: do not install the VMware Tools or Guest Additions at any point on your new machine. This will, like failing to choose Other 64-bit, cause the VM-within-a-VM stack to fall apart.

Now we get to create a virtual disk. I'd create a new one, unless you have one you want to destroy lying around. Don't worry about the capacity; we'll adjust that later.

VMWare will then give you a summary of your new machine. Click Finish, and then give it a name. The name for the HaLVM Workshop VM is unimaginative ("HaLVM Workshop VM"), but you can name yours whatever you like.

When you hit save, it's going to start the VM, which isn't going to do anything. Shut it down, using the "Shut Down" item in the "Virtual Machine" menu.

Improving the Settings of Your New VM

OK, now let's go through and tweak a bunch of settings to make everything work better. In the Virtual Machine library, select your new Virtual Machine, and then click then click Settings (either in the right click menu, the toolbar, or from the "Virtual Machine" menu).

First thing, VMWare seems to live in the late 90s / early 2000s, and defaults to 256MB of memory for your machine. Let's bring things into the modern era, and bump that up via the "Processors & Memory" item. This will also improve your compile times. I set my VMs to use 4GB / 4096 MB, but you may give it more or less depending on how much memory your machine has available to it. However, I would strongly advise against using any less than 1GB / 1024MB.

When you're done, hit "Show All" at the top to return. Next step, network adapters.

The first thing to say about network adapters is that this is where you'll likely find the most variance from machine to machine, depending on your particular network setup. The following is one possible option. You may find, however, that other options work better for you.

That being said, we prefer to use two network adapters: one bridged to the local network, and the other private to the local machine. So select "Network Adapter" from the general settings window, and then select "Autodetect" under "Bridged Networking". This will automatically have the first network card bridge to your local network. Next, click the "Add Device" button in the top right corner. Select "Network Adapter", "Add", and then set this one to "Private to my Mac". Then "Show All" to get back to the main menu.

Again, this dual-NIC configuration isn't necessary, but we find it very convenient for debugging some network problems.

We're almost done now. Next step, click on "Hard Disk (IDE)". Give it more disk space. I tend to default all new VMs to 40GB as a nice base. Do not select less than 20GB, but feel free to pick your own amount based on your own available disk space. Click "Apply."

"Show All" gets you back to the main menu once again.

The last thing to do is figure out your source material for Fedora. The easiest way to do this is to get a DVD image from your local Fedora mirror. You can select Fedora 21 or 22, currently. Right now, we're still doing a lot of our work with 21, so we recommend that. Here's a list of mirrors for Fedora 21, and here's a list of mirrors for Fedora 22. The file you're looking for is releases/2X/Images/Workstation/iso/Fedora-Workstation-netinst-x86_64-2X.iso.

Download it. Use a local, fast mirror, because they're kind of big.

Wait. Dum de dum.

Once it's down, set it up as a CD: Click "CD/DVD (IDE)", pick "Choose a disk or disk image", and select your newly-downloaded image. Then check the "Connect CD/DVD Drive" checkbox, and you're done with the Settings. Close the window, and be glad.

Running the Fedora Installer

Now you're ready to go. Start your new VM using either the right click menu, toolbar, or menu item.

Select your language, continue. Set your date, time, and time zone if you're not presently in US Pacific. Set your keyboard if you have a non-US keyboard.

Next, click on "Installation Destination." Click on the drive. This should allow some options at the bottom about partitioning, but I have to admit I find this UI a little buggy. I frequently have to exit out of this screen (via the "Done" button in the top left corner) and then try again. Anyways, once the UI elements are enabled, you probably don't want to change them. I leave it as "Automatically configure partitioning.", and I do not check the additional space checkbox. So, at this point, click "Done" (top left corner). The warning icon over "Installation Destination" should go away. Eventually. Again, the UI is a little sluggish.

Once the option to choose packages becomes available, click it and select "Minimal Installation" on the left-hand side, and then click "Done" in the top left corner.

Finally, click on "Network & Hostname". I like to change the hostname to something inventive. Like "halvm-devel.galois.com" (if I'm working) or "halvm-devel.uhsure.com" (if I'm not). You should pick something fun, then click "Done" (top left corner, again).

Finally, click on "Begin Installation."

While it's installing, you have the option to set the root password and create a default user. You should do both of these things by clicking on the items and then clicking done. The warning icons will go away once you've set things up.

For the workshop VMs, I've set up the user as "HaLVM Developer" with the username halvm and password halvmdevel. You will have to click "Done" twice if you use the same password, because it's very weak and Fedora does not approve. Before you do, though, uou should make sure to check the "Make this user administrator" box if you want to avoid a lot of headaches.

The root password is also halvmdevel. Shhhhhhh.

Now you wait for the software to complete installing. Dum de dum.

When its done, the "Quit" button will appear on the bottom right. Click it. Just to be a pain, you now have to explicitly reboot the machine. Click on the power icon in the top right corner, which opens up a menu with another power icon. Click on that, then press Restart.

The First Boot (into Fedora)

Now Fedora will boot from your hard drive. This is great. It will boot you to a log in screen. Do so with the password you configured during installation (halvmdevel for workshop images).

Optional: Connect via SSH for Better Terminal

While you can complete the rest of this guide with the console of the VM, without a GUI desktop our options for pleasant terminal use, including copying and pasting some of the commands in the rest of this guide, are limited. Fortunately even the Fedora minimal installation comes with an SSH server ready to go, we need only figure out how to reach it. At the console, run ifconfig and look for the IP address in the inet field of the result:

$ sudo yum install net-tools
$ ifconfig
enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.XXX.YYY  netmask 255.255.254.0  broadcast 192.168.41.255
...

Then, on your host machine in a nicer terminal emulator, connect to that IP address as the halvm user:

$ ssh [email protected]

Install the Bare Minimum Utilities

From a terminal, I install sudo plus my favorite editor, vim. If you use some other editor, you should install that at this point:

$ sudo yum install sudo vim

It will ask you for your password. It may spew some "Not Found" errors, which you can ignore. Tell it that what it's going to install is OK, and let's get on with our lives.

Adam Tells You To Do A Bad Thing

Now I'm going to tell you to do something you really shouldn't do, but will make your life enormously easier. Just about all the interesting tools you're going to run when launching and testing the HaLVM will require sudo. I get very tired of typing my password all the time. So I choose to opt out.

This is a bad idea, from a security perspective. So you might not want to do this.

Use your favorite editor to open /etc/sudoers:

$ sudo EDITOR=vim visudo

Search down for "%wheel ALL=(ALL) ALL". Right below this line, add the following:

%wheel  ALL=(ALL)       NOPASSWD: /usr/sbin/xl, /usr/sbin/gdbsx

Install Xen and the HaLVM

Next, let's get Xen and the HaLVM installed.

First, let's tell our Fedora system about the HaLVM public repos. Be sure to replace 2X in the commands below with the appropriate Fedora version number:

$ sudo yum install wget
$ wget http://www.galois.com/~awick/halvm-fedora/2X/halvm-yum-repo-2X-1.fc2X.noarch.rpm
$ sudo yum install halvm-yum-repo-2X-1.fc2X.noarch.rpm

Second, let's update our system, and coincidentally learn about the HaLVM stuff.

$ sudo yum update

Now let's install the HaLVM and a slightly modified version of Xen:

$ sudo yum install gcc HaLVM-gmp halvm-xen

Why a modified version of Xen? Debugging. The stock version of Xen that comes with Fedora has certain low-level debugging capabilities disabled, as they add some bulk and slowdown to the whole system. The halvm-xen version of Xen has been modified to enable those features, which allows for us to use (with some boot flags) a very reliable output mechanism for debugging.

Before we reboot -- yes, we're about to reboot again -- let's set up that boot flag I just mentioned. Open up /etc/default/grub with your favorite editor, with sudo, and add the following line:

GRUB_CMDLINE_XEN="console_to_ring loglvl=all guest_loglvl=all"

While you're at it, find the line that starts with GRUB_DEFAULT, and change it to:

GRUB_DEFAULT='Fedora, with Xen hypervisor'

Then update the boot loader:

$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg

... and now we reboot!

Validate that Xen Works

After you've rebooted, which hopefully went smoothly, let's just triple check everything's working. Open a terminal. First, try this command:

$ sudo xl list

This gives you a list of all the Xen virtual machines running in your virtual machine. It should just list Dom0, and look something like this:

Name                               ID    Mem VCPUs    State   Time(s)
Domain-0                            0   3918     1   r----       42.7

If you get an error, then something went wrong. Maybe go back and see if you missed a step somewhere?

If that works, that means Xen is installed. Let's just triple-check the debugging options worked. Type the following command:

$ sudo xl dmesg

If the very end of this log looks like a Fedora console login prompt, then everything went fine. Next step, let's try a HaLVM example and see if it works. First, let's put the HaLVM examples somewhere more convenient:

$ cp -r /usr/share/HaLVM-2.0.0/examples halvm-examples

Then let's try a very simple one:

$ cd halvm-examples/Core/Hello
$ make run

This should run ... but it might not. Pretty regularly the first virtual machine that you try to run will fail, for no good reason. The usual error message is "failed to free memory for the domain." If that happens, just try make run again. If it is successful, you'll see something like this:

(d2) Starting 1-CPU HaLVM
(d2) init_sp: 0x00000000004aa000
(d2) self:    0x00000000004a9f6e
(XEN) grant_table.c:1249:d2 Expanding dom (2) grant table from (4) to (32) frames.
(d2) Hello
(d2) Exit called with 0

Oh. One last thing:

$ halvm-cabal update

Setting Up The Bridge

You now have a working Xen installation and a working HaLVM, and everything is good for the moment. However, you are going to run into trouble as soon as you try to build or run a HaLVM that requires networking. This is because Xen expects to attach new, virtual network devices to a Dom0 bridge called xenbr0, and we haven't told it about this.

Briefly, what we're going to do is create a new bridge called xenbr0, make sure that the STP protocol is turned off, make sure that your Internet-facing Ethernet card is attached to it, and then make sure that all of this happens on every boot.

So, to start, make sure the network service is on and will stay on through reboots:

$ sudo chkconfig network on

Now create the configuration file for the bridge:

$ sudo vim /etc/sysconfig/network-scripts/ifcfg-xenbr0

Use your editor to add the following contents to the configuration:

DEVICE=xenbr0
TYPE=Bridge
BOOTPROTO=dhcp
ONBOOT=yes
DELAY=0
NM_CONTROLLED=no

Next, we want to add a network card to our bridge. First, we need to find the name of the interface:

$ sudo yum install net-tools # if you haven't already done so above
$ ifconfig

Look for the interface with the MAC address that matches your externally-visible network interface (you can see this in the advanced settings of the VM). In my VMs, this is pretty reliably eno16777728. Edit the configuration file for that interface...

$ sudo vim /etc/sysconf/network-scripts/ifcfg-eno16777728

... and add the following lines:

BRIDGE=xenbr0
NM_CONTROLLED=no

Now restart the network service, and ping a friendly outside address to make sure you still have access to the wider Internet:

$ sudo systemctl restart network.service
$ ping galois.com

If you've done all this, you've made it through a major hurdle in setting up these machines. Pat yourself on the back.

Extending the Install

Now that you've got a basic HaLVM development environment, there are a few things you might want to do to make giving your workshop a little easier: Installing some basic stuff for the web server example, installing some basic stuff for TLS, and installing a basic linux disk to help people understand the Xen tools.

Prepping for the Web Server

First, this is basically a revisit of this wiki page. It's probably more likely I'll update that page than this one, so you might want to check there if anything goes wrong. That being said:

Pull and install HaNS:

$ sudo yum install git
$ git clone https://github.com/GaloisInc/HaNS
$ cd HaNS
$ halvm-cabal install

Then pull and install network-hans:

$ git clone https://github.com/GaloisInc/network-hans
$ cd network-hans
$ halvm-cabal install

Then pull and install Adam's version of HTTP. Which should be pushed upstream at some point:

$ git clone https://github.com/acw/HTTP
$ cd HTTP
$ halvm-cabal install

Then install blaze-html, which might just install:

$ halvm-cabal install blaze-html

Prepping for TLS

$ git clone https://github.com/vincenthz/hs-certificate
$ cd hs-certificate/x509
$ halvm-cabal install
$ cd ../x509-store
$ halvm-cabal install
$ cd ../x509-validation
$ halvm-cabal install
$ cd ../..
$ git clone https://github.com/acw/hs-tls
$ cd hs-tls/core
$ halvm-cabal install -f-network -fhans

Creating a fedora boot disk