It's been a while since I last wrote a blog, but that doesn't mean I've been slacking off. In the next series of blog posts, we'll be looking at some of the new things in Foreman 1.2. But today, I want to make a small diversion...

I recently got some new hardware in the house, courtesy of my employer, so I decided it was time to play with one of the other virtualization technologies out there. As existing readers will know, I'm a big fan of Libvirt and KVM. I'm in no way disatisfied with Foreman's ability to manage Libvirt - but there's little point in running it on 2 machines. I won't learn anything new that way.

So, what to use?

My choice was really between two systems - oVirt and OpenStack. I had originally planned to run oVirt when I ordered the hardware, but delivery issues meant it took a while to reach me. In the meantime, I was asked to help out with some OpenStack/Foreman integration, and was quickly intrigued by it's capabilites. I've been using it now for a couple of weeks, and it's pretty great. However, I haven't seen a huge amount of literature on OpenStack and Foreman, so I decided to explain how I set it all up to you.

What's OpenStack?

There's a ton of blogs about OpenStack at the moment - with so many companies putting time and effort into the codebase, and even more using it in production, it's hard to not hear about it. So I'll be keeping my introduction to OpenStack itself brief.

OpenStack is a tool for building a private cloud at large scale - if you're the sort of organisation that wants the compute power & flexibility of Amazon EC2, but on your premises, OpenStack is the way to go. It deals with co-ordinating real hardware as a cloud, and deploying virtual machines to those nodes, along with the associated tasks of storage, image management, networking, capacity balancing, and so on.

However, despite the goal of a massively scalable computing system, it's quite possible to run Openstack on a single machine. It simply means that services which would normally run on a dedicated system each, all run on the same box. Not optimal perhaps, but certainly usable.

Stack 'em up

So, I've settled on OpenStack - how do I get rolling?

DevStack was written for people in my position - those wanting to try out OpenStack, but lacking a decent pool of hardware to run it on. DevStack is a script which takes in a configuration file and then sets up a complete installation of OpenStack ready for use.

(Aside: now that I'm more familiar with OpenStack, I may well try out some of the more other installers at a later date. DevStack is a great starting point while you're getting used to the terminology though :P)

DevStack requires an Ubuntu or Fedora base install - you'll not be surprised to hear that I opted for a 12.04 LTS install. Once that was complete, I started to follow DevStack's execellent all-in-one install guide. I'll not bore you with repeating it here, but I will post my localrc file (which is the configuration input for DevStack)

greg@amethyst:~/devstack$ cat localrc | egrep -v "^\s*#|^\s*$"
ADMIN_PASSWORD=<random string>
MYSQL_PASSWORD=<random string>
RABBIT_PASSWORD=<random string>
SERVICE_PASSWORD=$ADMIN_PASSWORD
HOST_IP=172.20.10.35
LOGFILE=$DEST/logs/stack.sh.log
LOGDAYS=2
SWIFT_HASH=<random string>
SWIFT_REPLICAS=1
SWIFT_DATA_DIR=$DEST/data
SERVICE_TOKEN=<random string>

I've stripped the comments to keep it short, and most of it is specific to my network - but as you can see, it's a pretty short config file for such a big project. Once we're done with the config, we can run the installer

./stack.sh

This should run to completion, but it does take quite a while :P. We can now access Openstack on our machine, as a webservice (so, in my case, at http://amethyst/) using 'admin' and the password from the localrc file.

Err, so?

It's all very well saying 'it's running!' but what do we do with it? Well, Openstack is primarily an image-based system, so the next step is either to download or build some images to use.

If you've already logged in, you'll have seen that Images & Snapshots is a menu entry in the navigation sidebar. Going there, we have Create Image, which will ask for a source URL (or file to upload). A few distros do provide images ready for use (see the OpenStack documentation on obtaining imagesfor more):

Image DIY

Sadly, my two favorite distros (Debian and Arch) do not provide a prebuilt image for OpenStack. Debian have started a project to build a Wheezy or Jessie image from a script (see this thread for the discussion) which should eventually end up as a package available to install on Debian which will build an OpenStack image.

You can download this script from their git repo (git://anonscm.debian.org/openstack/openstack-debian-images.git), and I've also forked a copy of it myself. I then used it as a base for doing the same steps for Archlinux, which has been moderately successful. The only thing I haven't yet (at time of writing) managed to do is to have the root filesystem expand to the full size of the disk on boot (which the Debian images do via cloud-initramfs-growroot). I'm sure I'll get it working though. You can find my current versions on my GitHub page).

If all else fails, you can use Libvirt (you knew I'd get that in here somewhere, right?). The OpenStack documentation has walkthroughs for both CentOS and Ubuntu, using an interactive install in virt-manager, which work fine for any distros where you can't get a prebuilt image (I did this for Squeeze, as the above Debian script doesn't work on Squeeze).

Once you have an image, you can upload it to OpenStack using the Create Image button. You should also be able to test lauching an instance and make sure it boots. Hooray! We have a working virtualization platform! Well, almost....

Networking Fun

The only thing that wasn't set up out of the box (as far as I can remember, after three weeks of using OpenStack) was the floating IPs. What's that? Let me explain...

Those of you who have used Libvirt in any serious endeavours will be aware that there are broadly two networking approaches. The first is to create a network bridge between the physical network device on the libvirt host, and the virtual machines. This has the effect that all VMs are on the public-facing network, and firewalling is the job of the VM. Alternatively, one can create a virtual NAT network inside Libvirt and have the VMs attach to that - which makes them hard to get to. If you want to reach a NATed VM from your laptop, you either need to do some SSH tunnelling, or set up some static port-forwards on your Libvirt host. Neither option is particularly awesome, although bridging works OK if you have control of the 'public' network too.

OpenStack (and other platforms, for that matter) approach this by having a pool of 'floating IPs' on the public network (in this case public means my house network, not the public internet). These IPs are reserved for use by OpenStack, so I had to reduce the pool of my DHCP server to avoid clashes. These can then be assigned to a VM from the OpenStack console. This gives you the best of both worlds - you still get the security of being on an isolated network (you have to configure the Security Groups if you even want to be able to ping your hosts on the assigned floating IP), but you can then access the VM on the public IP, eliminating the need for SSH tunnels (which can be awkward if you're testing webservices, for example).

I added 49 IPs to my OpenStack instance using the following inline bash script:

for n in `seq -w 151 199`; do nova-manage floating create --ip_range 172.20.10.$n; done

Your mileage may vary, as I'm writing this from memory (and .bash_history), but you will need some floating IPs for the Foreman stage.

You'll also need to allow SSH access (22 TCP) to your VMs using the Security Groups. I used a simple rule of '22 TCP from 0.0.0.0/0' (i.e everywhere) on the 'default' security group.

Stack The Deck

So, we have OpenStack up and running, and after ~150 lines of blog, you're wondering when I'll get to the Foreman bit - well, that's now ;)

I'm going to assume a few things about your setup:

  • Foreman is already configured with an appropriate Architecture
  • Foreman already has an Operating System that matches the distro image you uploaded to OpenStack
  • Foreman and OpenStack are on different machines
  • Both are on the same network (in my case 172.20.10.0/24)
  • Foreman controls DNS for this network (optional)

The first step is to add your OpenStack API credentials to Foreman. Go to More -> Provisioning -> Compute Resources and add a new one. Give it a Name of course, and select OpenStack as the type (slightly obvious, but if you don't see OpenStack, ensure you have the additional compute packages installed on your Foreman server). The URL is the API endpoint, which will be something like http://amethyst:5000/v2.0/tokens (set the hostname as appropriate). The Username/Password will be 'admin' and the admin password from your localrc. Then you can press Load Tenants and pick the one which you uploaded your distro image(s) to. Hit Test Connection and (assuming it's fine - remember that Test Connection silently succeeds) save the CR.

Now we need to map the image on OpenStack to the Operating System on Foreman. Click the new CR and select 'New Image'. Select the appropriate image from the dropdown, and fill in the rest of the boxes (be careful with 'Username' as it varies from distro to distro - Ubuntu use 'ubuntu', Debian use 'debian', and my Arch image uses 'root'). Save the image.

We're almost there, but we also need a Finish template to run on the machine when Foreman provisions it. You could use any existing Finish template,but I'm going to add an OpenStack specific one (I also created an 'openstack' Puppet environment, to completely isolate my test area, but this is optional). Under More > Provisioning > Provisioning Templates I created a new template, containing just

mkdir -p /root/.ssh
echo "<my ssh key>" >> /root/.ssh/authorized_keys

Assign this template to the appropriate OS, and also restrict by environment if you're using one. You can always extended this further, using exising templates for inspiration (such as installing packages, running Puppet, etc). You can also check out Ohad's EC2 blogpostfor more ideas - OpenStack works very similarly to EC2, and those scripts should work for OpenStack too.

Deal the hand

So, after a lot of UI setup, we're ready to try it!

Go to Hosts -> New Host, select your OpenStack CR from Deploy On, give it the usual things like Name, Host Group, Puppet Environment, etc. You'll notice Network only has a choice of Domain now (which matters for your DNS records). On the Operating System tab, after selecting the correct Architecture and Operating System, you should be able to select the Imageyou created above, and load the templates to make sure the new template is being used. The Virtual Machine tab allows us to select how many resources we should give to the new VM, and importantly we must select the Floating IP network (otherwise Foreman won't be able to reach the VM). You should also select the Security Group which has SSH access (otherwise Foreman won't be able to log in).

Once we've done all that, Submit the Host, and you should see Foreman spin up the VM on OpenStack, SSH in, and run the template.

Magic! :)

Conclusion

OpenStack plays very well with Foreman, as they split the required duties - OpenStack deals with all the resource management, quota management, security, and networking, and simply tells Foreman the IP it chose to assign to the host. Foreman can then do it's usual job of managing DNS, certificates, and providing Puppet on the host with ENC data (obviously DHCP and TFTP are not required in this scenario).

As a result, I now have the joy of booting my most commonly used distributions in ~1min rather than the ~20min it takes to do a PXE install on Libvirt, with DNS, and (if requested) a full Puppet run already done. Of course, I still have my Libvirt host of course, and can do a PXE install if I need a specific setup on the host, but so far I've had little need for that.

While this blog is quite long, the above took me less than a day to figure out. OpenStack is fairly intuitive (assuming you've used some other virtualization before), and of course, once you get to Foreman, you're getting the same consistent interface you're used to. It's all very smooth, really.

As ever, if you try this yourself, do let me know how you get on!