Creating a development environment with Libvirt, KVM, and TheForeman

This blog post has been in my to-write pile for nearly 4 months now. I have two laptops at home, both of which are capable of running a few virtual machines. If you missed the news, I now work on Foreman full time, so obviously I want to use Foreman to manage my virtual machines. So it seems like the perfect opportunity to give you a blog post about getting Libvirt set up on a host of your choice (in this case, my laptop).

Let's start by discussing my assumptions. Firstly, my personal laptop runs Archlinux, but since my Foreman server will be a virtual machine (Debian Squeeze) this won't matter too much. Second I'm assuming you're going to use the NAT network inside libvirt, so discussions of bridging will not be found here (although it's easy to do). In summary, I will use:

Libvirt host: Archlinux, 64bit (192.168.212.1)

Foreman VM: Debian Squeeze, 64bit (192.168.212.2)

NAT network (specifically 192.168.212.0/24)

We will look at the setting up Libvirt with TFTP, bootstrapping a Foreman box, installing the proxy, and finally building a test vm from Foreman.

Special note: This blog relies on a feature only present in the nightly version of Foreman (specifically, host identification via tokens). As such, we'll be using the nightly packages.

Step 0 - Libvirt

This is step zero, because without libvirt we would be a bit stuck :). We start by installing libvirt and the kvm modules:

$ yaourt -S libvirt qemu-kvm

We now need to set up a couple of things. Firstly, the kernel module:

$ modprobe kvm_intel # or kvm_amd

(You might want to make that permanent on boot. I'm not always going to want my laptop to run VMs, so I haven't.)

I do not need my libvirt service advertising itself, so I disable mDNS:

$ echo "mdns_adv = 0" >> /etc/libvirt/libvirt.conf

We also need to give our day-to-day user the right permissions. Archlinux uses PolKit for that:

$ gpasswd -a greg kvm

$ cat > /etc/polkit-1/localauthority/50-local.d/50-org.libvirt.unix.manage.pkla <<EOF

[Allow a user to manage virtual machines]

Identity=unix-user:greg

Action=org.libvirt.unix.manage

ResultAny=yes

ResultInactive=yes

ResultActive=yes

EOF

For brevity, I will assume your partition structure can handle the space requirements - VM image files go in /var/lib/libvirt/images. I put 100G of storage into that area via a Logical Volume.

Now we can start libvirt. I use Systemd, so

$ systemctl start libvirtd.service

If all goes well, you will have libvirt in your processes (ps ax|grep libvirt).

Step 1 - Configuring Libvirt

We need to tweak Libvirt a bit for our purposes - specifically, it starts a dnsmasq process to handle DNS and DHCP, which is fine, but we need to add the bootp IP for our eventual Foreman server

First we need to edit the default network. I'd love to use 'virt-manager' for this step, as I find it easier than editing XML by hand, but 'virt-manager' cannot add the bootp option we need. So, we use 'virsh'. First we stop the network from running, and then edit it:

$ virsh
net-destroy default
net-edit default

Now you should see something like:

<network>
<name>default</name>
<uuid>2f2a01fe-fe6c-4f17-9f6d-2fb9d0ce6bc5</uuid>
<forward mode='nat'/>
<bridge name='virbr0' stp='on' delay='0' />
<mac address='52:54:00:27:1A:97'/>
<ip address='192.168.212.1' netmask='255.255.255.0'>
<dhcp>
<range start='192.168.212.2' end='192.168.212.254' />
</dhcp>
</ip>
</network>

We need to alter the range and add the bootp directive. Recall that Foreman will be static on 192.168.212.2, so we should start our DHCP range from 3. Make it look like this:

Save and exit. Finally we can restart the network:

net-start default

You should then get something like this:

$ ifconfig virbr0

virbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 metric 1

inet 192.168.212.1 netmask 255.255.255.0 broadcast 192.168.212.255

You will also notice a dnsmasq process, bound to 192.168.212.1:

$ ps ax|grep dnsmasq
13972 ? S 0:00 /sbin/dnsmasq --strict-order --bind-interfaces --pid-file=/var/run/libvirt/network/default.pid --conf-file= --except-interface lo --listen-address 192.168.212.1 --dhcp-range 192.168.212.3,192.168.212.254 --dhcp-leasefile=/var/lib/libvirt/dnsmasq/default.leases --dhcp-lease-max=252 --dhcp-no-override --dhcp-boot /pxelinux.0,,192.168.212.2

Step 2 - Building the Foreman box

So, at this point we have Libvirt, but no Foreman. Time to fix that! We start by downloading the install media of your choice (I used the Debian netinst CD). Once you have it, you can create a new VM in virt-manager (I called mine Foreman - I know, I am crazy :P). Simply click the "New VM" button in virt-manager, point it at the install CD, and get a basic install up and running. I am going to assume you know how to do a manual install of your favourite OS :). Bear in mind that we want to use a static IP for this server (192.168.212.2).

That will take you a while so I am going to go grab a coffee....

...

Done yet? Good. We can set up Foreman then.

We can refer to the excellent Foreman-installer puppet modules as discussed on [[theforeman.org][http://theforeman.org/projects/foreman/wiki/Using_Puppet_Module_ready_to_use]. We are going to use an All-In-One install.

To do this, we need to ensure we will be using a relatively recent version of Puppet. At time of writing, Puppet 2.6.2 is in the Squeeze repo (too old) and Puppet 3 is not yet supported by Foreman. So we'll use th Puppetlabs repo and install a specific version of Puppet:

$ echo "deb http://apt.puppetlabs.com squeeze main" > /etc/apt/sources.list.d/puppetlabs.list
$ wget -q -O - http://apt.puppetlabs.com/pubkey.gpg | sudo apt-key add -
$ apt-get update
$ apt-get install -y puppet-common=2.7.19-1puppetlabs2 puppet=2.7.19-1puppetlabs2 puppetmaster-common=2.7.19-1puppetlabs2 puppetmaster=2.7.19-1puppetlabs2

We will also need git

$ apt-get install git-core

Then we will need the installer

$ git clone --recursive https://github.com/theforeman/foreman-installer.git

Now we need to review the parameters in these modules. There are a lot of them, but for my purposes, most of the defaults are sane, so I will only list the ones I am changing for my setup:

foreman/manifests/params.pp

$ssl = false # ssl is not really need on my dev setup

$use_testing = true

foreman-proxy/manifests/params.pp

$use_testing = true

puppet/manifests/params.pp

$git_repo = 'true' # This is personal preference only, not required

Now the fun bit - installing Foreman:

$ echo include puppet, puppet::server, foreman, foreman_proxy | puppet apply -v --modulepath /root/foreman-installer/

This will pull in and configure foreman, foreman-sqlite3 (good enough for my needs, but you may prefer mysql or postgres - set these up after the module finishes configuring foreman for you), but I will also need foreman-libvirt to manage my libvirt setup:

$ apt-get install foreman-libvirt

$ touch /usr/share/foreman/tmp/restart.txt

If we now navigate to our VM (http://192.168.212.2 in my case) then we have Foreman up and running. Tadah!

Step 3 - Configuring Foreman and Foreman Proxy

Let us start by making foreman show up in its own hosts page. Fortunately that is quite easy :)

$ puppet agent -v --noop -o --no-daemonize

Now we need to add the proxy. Go to 'More->Smart Proxies' and click 'New Proxy'. You will probably need to add an entry to /etc/hosts to make this resolve, as Foreman (192.168.212.2) isn't in DNS. Don't just the IP here, as the server in the URL string is given to all the clients as the name of the Puppet Master

Name: Dev Proxy

URL: http://foreman.my.domain.com:8443/

Submit, and you should see a proxy with Puppet, PuppetCA, and TFTP features. So far, so good. You should also go to the Domains page and check the first puppet run created your vrtual domain. You might as well also edit it and give it a readable name in the blank box.

Lets move on to DNS/DHCP. We're aren't really going to manage these, but we still need to give some details to Foreman. Go to the More->Subnets page, add a new subnet, and fill it in something like this:

Name: Libvirt Network
Network: 192.168.212.0
Netmask: 255.255.255.0
Gateway: 192.168.212.1
Primary DNS: 192.168.212.1
Secondary DNS: 8.8.8.8
From / To / VLAN: [leave blank]
Domain: [tick the domain of your machine]
Dhcp: [None]
Tftp: [Your Proxy Name]
Dns: [None]

Nearly there! We need to add a compute resource. Go to 'More->Compute Resources' and click 'New Compute Resource'. Give it a name and select 'libvirt' as the provider. It will ask for a connection to the libvirt system, and this can be tricky to set up. Some people have managed to use the puppet certificates for SSL/TLS communication with the livbirt host, but I could not get it to work, so I am using restricted-access SSH keys:

su - foreman -s /bin/bash

ssh-keygen # accept default file locations, no passphrase

# copy publickey to libvirt host (192.168.212.1 in my case)

ssh root@192.168.212.1 # if this works you are good to go

You will also need the 'nc' binary on the libvirt host - on Archlinux it is actually nc.openbsd, so I had to create a symlink. The URL is 'virsh -c qemu+ssh:///root@192.168.212.1/system' - the Test Connection button is quite silent when everything works, so press it! If nothing happens, your libvirt connection is fine. Save the resource, and then select it and go to the 'Virtual Machines' tab - you should see your foreman box listed.

Finally, we need to enable the Token-based installs. Go to More->Settings, select the Provisioning tab, and set Token_duration to something other than zero. I used 60 minutes.

Step 4 - Building a VM

Ok, home straight. We can now create Virtual Machines, but we need to test that the whole system hangs together. Go to 'Hosts' and select 'New Host'. You should then be able to create a Host using one of the OSes that Foreman comes pre-installed with (Ubuntu 12.04 for example). Foreman will ask you for an IP (because we're not managing DHCP in Foreman) but you can put anything you like here (I used 1.2.3.4). When the host first runs puppet and uploads it's facts to Foreman, it will replace 1.2.3.4 with the IP it got from Dnsmasq.

When you Submit the host, you should see a set of progress bars as Foreman creates the VM on libvirt, and the after you are redirected to the Host page, you can click the Console button to watch the install happen. Awesome!

So, now you have a Foreman VM which can create other VMs. You also have a puppetmaster on there, which will accept code pushed to "puppet@192.168.212.2:puppet.git" so now you can test modules in a proper client/server virtual environment. Enjoy!

Step 5 - Extra Exercises

There are a few extra things you could do, if you want to play around some more:

Use a different database - mysql and postgres are supported
Add a different compute resource - EC2 is a good choice since you can get free machines
Add bridged networking - sometimes those VMs need to be on the main network...
Control DHCP/DNS with Foreman - this is tricky as you have to disable the built in dnsmasq server and set up bind9 and isc-dhcp-server on Foreman.

Go forth and create virtual machines!