Emerald Reverie

Gwmngilfen's blog - Tech, cooking, walking and other randomness from the heart of Scotland

Managing SSH Host Keys in a Reliable Way

I’ve been managing my virtual machines using Foreman for close to 2 years now, and that’s brought me a huge set of benefits in terms of how I test new code (or changes to existing code), and new packages. That’s just awesome :)

But repeated rebuilds of a machine lead to one small niggling problem. One which bites you on every rebuild. One which doesn’t stop you working, but requires a few extra keypresses after every rebuild, and possibly at every login.

Not got it yet? Does this look familiar?

[greg:~]$ ssh test2
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
Please contact your system administrator.
Add correct host key in /home/greg/.ssh/known_hosts to get rid of this message.
Offending RSA key in /home/greg/.ssh/known_hosts:217
RSA host key for test2 has changed and you have requested strict checking.
Host key verification failed.

(If it doesn’t, the answer is changed SSH host keys, but the rest of this blog probably won’t make much sense :P)

Yeah. Irritating, isn’t it? Every time, every rebuild, a few seconds wasted. It adds up. There’s got to be a way to make this go away, right?

The ‘Traditional’ Way

The way most Puppet users might approach this problem would be to use one of Puppet’s greatest features: exported resources. This feature allows you to exchange information between hosts that the puppet master doesn’t know a priori.

This is a quick solution, from a code perspective, and making sure all the machines in your infrastructure have each others keys is really simple (example courtesy of Puppet Labs’ documentation:

class ssh {
# Declare:
@@sshkey { $hostname:
type => dsa,
key => $sshdsakey,
# Collect:
Sshkey <<| |>>

The problem with this is two-fold. Firstly, it’s a quick win in code, but only if you’re already using exported resources. See, to use them, you have to be using a database backend for Puppet itself. Given the database backend is otherwise optional, not everyone does. If I want to write a nice solution for everyone to use, this isn’t going to work.

Second, it’s also slow. When a machine changes it’s host keys, it’s going to take two puppet runs for the systems to catch up - the first is when the new VM uploads it’s new key to the puppet database, and the second when my laptop retrieves the new key and updates it’s key list.

So that’s actually slower than just deleting the changed key from my known_hosts. No good.

The ‘Foreman’ Way

My next thought was to look at how Foreman can be used to replace exported resources. This has been covered in other blogs (The Foreman Blog covered this a while back).

So, since the facts of a host are uploaded to Foreman we could replicate the above code by doing a Foreman search for all the $sshkey facts and writing them to a file. Pretty neat.

However, this doesn’t help either. Not everyone uses Foreman (even if I think they should :P), and we still have the 2 run problem which makes it slower to fix the problem than to suffer it. We’re not getting closer…

Inspiration Via Services

The solution came to me when working on a module for backups. I needed a way to allow the backup system to SSH onto the backup targets to initiate rsync. Ideally, I didn’t want to store the private key in puppet, since I was planning to publish the repo. But I also want the module to work without an end user having to manually add an admin ssh key for the service.

I Googled around and came across this: Github: fup/puppet-ssh

This is a function to generate keys as required on demand, and store them on the Puppet master, and make it possible to read them back (both the private and public parts) for use in Puppet manifests.

Bingo! With this function I can create a key the first time it is requested, but thereafter it will be re-read from the keystore dir on the puppetmaster. Since the puppetmaster isn’t the VM being rebuilt, when I recreate my vm, it’ll get the same SSH key as last time it was built.

The function didn’t quite fit my needs, as I want to keep some types of keys separated by environment (development backup servers should have access to production machines, for example), so I forked the function and extended it a little. You can find the result at Github: GregSutcliffe/puppet-modules, but let’s take a quick look at how the module works.

The Code

Most of the ssh module is fairly tedious - make sure it’s installed, manage the config file, start the service, yadda yadda…. The only interesting bit is the key handling. Lets take a snippet straight from the repo:

$rsa_priv = ssh_keygen({name => "ssh_host_rsa_${::fqdn}", dir => 'ssh/hostkeys'}) 
$rsa_pub = ssh_keygen({name => "ssh_host_rsa_${::fqdn}", dir => 'ssh/hostkeys', public => 'true'})

What’s happening here? Well, a couple of things. Firstly the dir parameter is my extension to Fup’s original function - it allows me to specify where to store the keys on the puppet master. Otherwise, we’re asking the function to read (and if required, create) a key named ssh_host_rsa_myvm.fqdn.com and read both the private and public parts into appropriately named variables.

    file { '/etc/ssh/ssh_host_rsa_key':
owner => 'root',
group => 'root',
mode => 0600,
content => $rsa_priv,
file { '/etc/ssh/ssh_host_rsa_key.pub':
owner => 'root',
group => 'root',
mode => 0644,
content => "ssh-rsa $rsa_pub host_rsa_${::hostname}\n",

Here we take those variables and apply them to the server in question. Thus, when this module runs, it will overwrite the auto-generated ssh key with the one from the puppetmaster. As such, whenever I log into the machine, it will always have the same key (and the same fingerprint) so my known_hosts file is happy.


Security Caveat

There is a minor security issue - all the keys generated by the function live on the puppet master. Technically, if they got into the wrong hands, that could be bad. However, as the machine which hands out configuration data to your infrastructure, if someone compromises it to the point where they can read those keys, it’s already game over. Plus, if they leak some other way, regenerating all the keys of your infrastructure is only a rm -rf /etc/puppet/ssh away ;)

So there you have it - consistent SSH host keys for your machines every time! Better, it works for everyone, regardless of database backends or other external stores of data. It’s also fast - since I do a small Puppet run as part of the provisioning of my machines, the host key is already set when it comes up at first boot. All my requirements met. Wonderful :)

Installing Foreman on Archlinux

So as many of you know, I use Archlinux on a lot of my hardware - but you’ll have noticed that I always use Debian for my Foreman servers. What gives?

The Problem with Versions

Archlinux presents two major problems to Foreman. Firstly, the current version of Puppet in the AUR is 3.0.1. Admitedly, I made that problem for myself, since the AUR PKGBUILD of Puppet is owned by me… However, Arch is all about latest-and-greatest so it’s the right thing to do for Puppet.

Fortunately, the latest development code of Foreman supports Puppet 3, so we’re ready to go right?


See, Arch also packages the latest and greatest Ruby as well (1.9.3p327). Foreman cannot (yet) deal with Ruby 1.9. So, we’re dead in the water. Or are we?

Community to the Rescue!

There is a pull request from the excellent user @witlessbirdproviding support for Ruby 1.9.3. It’s messy, as many of the problems stem from the gems we depend upon not being 1.9 compatible, but he’s forked them all and updated our Gemfile to point as his forks. So in theory it all works. He’s awesome :)

So I decided to set up Foreman on Arch. Just to see if it runs ;)

Getting Down and Dirty

So here we go. I’ll assume you have a freshly installed Archlinux box (assuming you’re playing along at home). I’ll also assume the ever-popular AUR helper ‘yaourt’ is installed, and with that you’ve installed Puppet 3.

Lets start by cloning the source. There’s no packages for Foreman on Arch yet, and we need to apply some 1.9 fixes anyway.  @witlessbird’s changes required some merge conflicts, so I’ve forked his repo so I could rebase the changes against. Get it this way:

cd /usr/share
git clone --recursive https://github.com/GregSutcliffe/foreman -b193

We also need bundler:

yaourt -S ruby-bundler

Now we set up run-time dependencies:

cd foreman
cp config/settings.yaml.example config/settings.yaml
cp config/database.yml.example config/database.yml
echo ":puppetconfdir: /etc/puppet" >> config/settings.yaml
yaourt -S libxml2 libxslt libmysqlclient postgresql-libs libvirt

Get the gems:

bundle install --path vendor/

Set up the db

bundle exec rake db:migrate

Start the server

bundle exec rails s

And lo! Foreman starts on port 3000. Wonderful. But we’re not done being awesome yet.

Enter the (e)ngin(e)

A better webserver needs to be added, since Webrick sucks. I decided to give Nginx a try. Nginx has the downside that modules have to be compiled in, but on Arch, Passenger has already been compiled in. So we just install it:

yaourt -S passenger nginx

In nginx.conf we have a basic passenger definition:

http {
passenger_root /usr/lib/passenger;
passenger_ruby /usr/bin/ruby;

server {
listen 80;
server_name topaz.elysium.emeraldreverie.org;
root /usr/share/foreman/public;
passenger_enabled on;
rails_env production;

So now we can start Nginx

systemctl start nginx
systemctl enable nginx

Yay! Foreman on port 80. Win!

Proxies, proxies, everywhere!

Of course, we’ll also need a foreman-proxy to manage puppet for us. We could just use my new foreman-proxy-git AUR package. But where’s the fun in that? Much more fun to run it in Nginx with Foreman! Lets clone the code:

cd /usr/share
git clone https://github.com/theforeman/smart-proxy.git foreman-proxy

Now to fix up the proxy…

yaourt -S ruby-sinatra sudo
server {
listen 8443;
server_name foreman-proxy;
root /usr/share/foreman-proxy/public;
passenger_enabled on;
rails_env production;

Of course, we still need to replicate a lot of what the installer would do on other distros:

echo -e "foreman-proxy ALL = NOPASSWD : /usr/bin/puppet
Defaults:foreman-proxy !requiretty" >> /etc/sudoers.d/foreman-proxy
groupadd -r foreman-proxy
useradd -r -g foreman-proxy -d /usr/share/foreman-proxy -s /sbin/nologin -c "Foreman Proxy deamon user" foreman-proxy
chown -R /usr/share/foreman-proxy

There are a few necessary gems for the foreman-proxy user as well:

su - foreman-proxy -s /bin/bash
gem install sinatra net-ping

That’s it! Restart nginx to pick up the config, and try hitting the proxy for a request:

wget -q -O - http://localhost:8443/features

Final notes

Finally, a small dependency bug I discovered - Foreman requires Python to display the noVNC libvirt console. So we also need to do:

yaourt -S python

That’s it! However, just for fun, I decided to set Foreman up running on PostgreSQL 9.2, just for extra bleeding-edge-ness. I leave it as an excerise to the reader to do this, as it’s fairly standard database setup - Foreman has support Psql for quite a while.


Debian Packaging, the Transparent Way

So I’ve been managing the Debian packages for about 8 months now, and every so often I get asked if there’s anything people can do to help. I have to answer “Not really” because the way we’re building the Debian packages is somewhat arcane.

At least, it was. This blog is to tell you all about how it’s now much more open.

Packaging in the open

The first piece of the puzzle starts with our foreman-rpms repo (github.com/theforeman/foreman-rpms)
in this repo we store all the packaging data for the foreman packages, both deb and rpm. In the case of Debian, you’ll see a structure like:


For example, the foreman-proxy nightly package for squeeze is at:


Now, the eagle-eyed and quick-witted among you will already have thought “That’s quite a lot of code duplication”. You’re absolutely correct, but this method allows us to package each distro in it’s own way, if need be. At the moment, it’s all very similar, but it might not stay that way. It also makes it trivial to add in more distributions later (say, quantal or wheezy).

So, now you know where our package data is, you’ll want to know; how does it get used? Well, here we turn to the king of automation - Jenkins.


You can find our Jenkins server at ci.theforeman.org. In there, you will find a set of interesting packaging jobs, specifically:


There’s two types of jobs here. Let’s start with the “individual” jobs. These are what they sound like - they take a set of parameters about what package the are building and build the required package(s). For example, if I feed it the options

    arch = 64bit
    distro = squeeze
    type = nightly

Then it checks out the foreman-rpms repo, and changes to the appropriate directory in the repo, according to those options. It then runs ‘build.sh’ found in the packaging directory, which uses Pbuilder to build the package for the requested settings. At the end of the job (if it built successfully) the package is uploaded to our Debian repository at deb.theforeman.org

The matrix jobs, as you can guess, call the individual jobs many times over - once for all the possible combinations of the options. Thus we can have the matrix job track the Foreman git repo, and rebuild the devel packages whenever a commit is made.

How to get involved

Now that things are more open, it’s easy to get involved. Here’s a workflow you could

1. Get a build box

You don’t have to go through the trouble we did of configuring pbuilder. Just build a
machine with the OS you’d like to run the packages on

2. Install basic packages

You’ll need ‘build-essential’ and ‘devscripts’ as a minimum (well, on Squeeze
anyway :P)

3. Check out the repo

Fork the foreman-rpms repo and check it out to your buildbox. If you’re not trying
to update an existing distro, copy the closest match to a new repo.

4. Build the package

You don’t need to use ‘build.sh’ for this (that’s crafted towards pbuilder and other
small automated task we need to do in the final packages). Just run “debuild -us -uc”
and see if it builds. If it doesn’t either

  a. Install the missing build-dependencies, or
  b. Fix the broken packaging

5. Test

You can either test on your buildbox, or for best results, on another machine with
the same OS installed. Hopefully the package will install and work ;)

6. Contribute

Commit your changes to your repo and send us a pull request on foreman-rpms. We’ll
test your changes and merge if it looks good.


Hopefully that demystifies our package building architecure a bit. If nothing else,
this blog serves to remind me how we set it all up when I can’t remember in six
months time ;)


If anyone is interested in replicating the whole chain, you can get our pbuilder
puppet configuration from our infrstructure repo at github.com/theforeman/foreman-infra

This repo contains the puppet code we use to build the various parts of our
infrastructure, including the Debian build slaves for Jenkins. The only thing you won’t
find there is the GPG key to sign the packages, for obvious reasons ;)

Creating a Development Environment With Libvirt, KVM, and TheForeman

This blog post has been in my to-write pile for nearly 4 months now. I have two laptops at home, both of which are capable of running a few virtual machines. If you missed the news, I now work on Foreman full time, so obviously I want to use Foreman to manage my virtual machines. So it seems like the perfect opportunity to give you a blog post about getting Libvirt set up on a host of your choice (in this case, my laptop).

Let’s start by discussing my assumptions. Firstly, my personal laptop runs Archlinux, but since my Foreman server will be a virtual machine (Debian Squeeze) this won’t matter too much. Second I’m assuming you’re going to use the NAT network inside libvirt, so discussions of bridging will not be found here (although it’s easy to do). In summary, I will use:

Libvirt host: Archlinux, 64bit (
Foreman VM: Debian Squeeze, 64bit (
NAT network (specifically     
We will look at the setting up Libvirt with TFTP, bootstrapping a Foreman box, installing the proxy, and finally building a test vm from Foreman.

Special note: This blog relies on a feature only present in the nightly version of Foreman (specifically, host identification via tokens). As such, we’ll be using the nightly packages.

Step 0 - Libvirt

This is step zero, because without libvirt we would be a bit stuck :). We start by installing libvirt and the kvm modules:

    $ yaourt -S libvirt qemu-kvm

We now need to set up a couple of things. Firstly, the kernel module:     

    $ modprobe kvm_intel   # or kvm_amd     

(You might want to make that permanent on boot. I’m not always going to want my laptop to run VMs, so I haven’t.)

I do not need my libvirt service advertising itself, so I disable mDNS:        
    $ echo “mdns_adv = 0” >> /etc/libvirt/libvirt.conf                           

We also need to give our day-to-day user the right permissions. Archlinux uses PolKit for that:

    $ gpasswd -a greg kvm
    $ cat > /etc/polkit-1/localauthority/50-local.d/50-org.libvirt.unix.manage.pkla <<EOF
    [Allow a user to manage virtual machines]                                      

For brevity, I will assume your partition structure can handle the space requirements - VM image files go in /var/lib/libvirt/images. I put 100G of storage into that area via a Logical Volume.

Now we can start libvirt. I use Systemd, so                                    
    $ systemctl start libvirtd.service                                           
If all goes well, you will have libvirt in your processes (ps ax|grep libvirt).

Step 1 - Configuring Libvirt

We need to tweak Libvirt a bit for our purposes - specifically, it starts a dnsmasq process to handle DNS and DHCP, which is fine, but we need to add the bootp IP for our eventual Foreman server

First we need to edit the default network. I’d love to use ‘virt-manager’ for this step, as I find it easier than editing XML by hand, but ‘virt-manager’ cannot add the bootp option we need. So, we use ‘virsh’. First we stop the network from running, and then edit it:

    $ virsh
    net-destroy default
    net-edit default

Now you should see something like:

      <forward mode=’nat’/>
      <bridge name=’virbr0’ stp=’on’ delay=’0’ />
      <mac address=’52:54:00:27:1A:97’/>
      <ip address=’’ netmask=’’>
          <range start=’’ end=’’ />

We need to alter the range and add the bootp directive. Recall that Foreman will be static on, so we should start our DHCP range from 3. Make it look like this:

    <ip address=’’ netmask=’’>
        <range start=’’ end=’’ />
        <bootp file=’/pxelinux.0’ server=’’ />

Save and exit. Finally we can restart the network:

   net-start default

You should then get something like this:

    $ ifconfig virbr0                                                          
    virbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500  metric 1     
           inet  netmask  broadcast     

You will also notice a dnsmasq process, bound to

    $ ps ax|grep dnsmasq
    13972 ?        S      0:00 /sbin/dnsmasq –strict-order –bind-interfaces –pid-file=/var/run/libvirt/network/default.pid –conf-file= –except-interface lo –listen-address –dhcp-range, –dhcp-leasefile=/var/lib/libvirt/dnsmasq/default.leases –dhcp-lease-max=252 –dhcp-no-override –dhcp-boot /pxelinux.0,,

Step 2 - Building the Foreman box                                              

So, at this point we have Libvirt, but no Foreman. Time to fix that! We start by downloading the install media of your choice (I used the Debian netinst CD). Once you have it, you can create a new VM in virt-manager (I called mine Foreman - I know, I am crazy :P). Simply click the “New VM” button in virt-manager, point it at the install CD, and get a basic install up and running. I am going to assume you know how to do a manual install of your favourite OS :). Bear in mind that we want to use a static IP for this server ( 
That will take you a while so I am going to go grab a coffee….               

Done yet? Good. We can set up Foreman then.

We can refer to the excellent Foreman-installer puppet modules as discussed on [[theforeman.org][http://theforeman.org/projects/foreman/wiki/Using_Puppet_Module_ready_to_use]. We are going to use an All-In-One install.

To do this, we need to ensure we will be using a relatively recent version of Puppet. At time of writing, Puppet 2.6.2 is in the Squeeze repo (too old) and Puppet 3 is not yet supported by Foreman. So we’ll use th Puppetlabs repo and install a specific version of Puppet:

    $ echo “deb http://apt.puppetlabs.com squeeze main” > /etc/apt/sources.list.d/puppetlabs.list
    $ wget -q -O - http://apt.puppetlabs.com/pubkey.gpg | sudo apt-key add -
    $ apt-get update
    $ apt-get install -y puppet-common=2.7.19-1puppetlabs2 puppet=2.7.19-1puppetlabs2 puppetmaster-common=2.7.19-1puppetlabs2 puppetmaster=2.7.19-1puppetlabs2

We will also need git

    $ apt-get install git-core

Then we will need the installer

    $ git clone –recursive https://github.com/theforeman/foreman-installer.git

Now we need to review the parameters in these modules. There are a lot of them, but for my purposes, most of the defaults are sane, so I will only list the ones I am changing for my setup:
      $ssl              = false     # ssl is not really need on my dev setup     
      $use_testing = true                                                                             

      $use_testing = true
      $git_repo = ‘true’  # This is personal preference only, not required     

Now the fun bit - installing Foreman:
    $ echo include puppet, puppet::server, foreman, foreman_proxy | puppet apply -v –modulepath /root/foreman-installer/

This will pull in and configure foreman, foreman-sqlite3 (good enough for my needs, but you may prefer mysql or postgres - set these up after the module finishes configuring foreman for you), but I will also need foreman-libvirt to manage my libvirt setup:
    $ apt-get install foreman-libvirt                                            
    $ touch /usr/share/foreman/tmp/restart.txt                                   

If we now navigate to our VM ( in my case) then we have Foreman up and running. Tadah!

Step 3 - Configuring Foreman and Foreman Proxy                                 

Let us start by making foreman show up in its own hosts page. Fortunately that is quite easy :)

    $ puppet agent -v –noop -o –no-daemonize

Now we need to add the proxy. Go to ‘More->Smart Proxies’ and click ‘New Proxy’. You will probably need to add an entry to /etc/hosts to make this resolve, as Foreman ( isn’t in DNS. Don’t just the IP here, as the server in the URL string is given to all the clients as the name of the Puppet Master

    Name: Dev Proxy
    URL: http://foreman.my.domain.com:8443/                                    

Submit, and you should see a proxy with Puppet, PuppetCA, and TFTP features. So far, so good. You should also go to the Domains page and check the first puppet run created your vrtual domain. You might as well also edit it and give it a readable name in the blank box.

Lets move on to DNS/DHCP. We’re aren’t really going to manage these, but we still need to give some details to Foreman. Go to the More->Subnets page, add a new subnet, and fill it in something like this:

    Name: Libvirt Network
    Primary DNS:
    Secondary DNS:
    From / To / VLAN: [leave blank]
    Domain: [tick the domain of your machine]
    Dhcp: [None]
    Tftp: [Your Proxy Name]
    Dns: [None]

Nearly there! We need to add a compute resource. Go to ‘More->Compute Resources’ and click ‘New Compute Resource’. Give it a name and select ‘libvirt’ as the provider. It will ask for a connection to the libvirt system, and this can be tricky to set up. Some people have managed to use the puppet certificates for SSL/TLS communication with the livbirt host, but I could not get it to work, so I am using restricted-access SSH keys:

    su - foreman -s /bin/bash
    ssh-keygen  # accept default file locations, no passphrase
    # copy publickey to libvirt host ( in my case)
    ssh root@  # if this works you are good to go

You will also need the ‘nc’ binary on the libvirt host - on Archlinux it is actually nc.openbsd, so I had to create a symlink. The URL is ‘virsh -c qemu+ssh:///root@’ - the Test Connection button is quite silent when everything works, so press it! If nothing happens, your libvirt connection is fine. Save the resource, and then select it and go to the ‘Virtual Machines’ tab - you should see your foreman box listed.

Finally, we need to enable the Token-based installs. Go to More->Settings, select the Provisioning tab, and set Token_duration to something other than zero. I used 60 minutes.

Step 4 - Building a VM                                                         

Ok, home straight. We can now create Virtual Machines, but we need to test that the whole system hangs together. Go to ‘Hosts’ and select ‘New Host’. You should then be able to create a Host using one of the OSes that Foreman comes pre-installed with (Ubuntu 12.04 for example). Foreman will ask you for an IP (because we’re not managing DHCP in Foreman) but you can put anything you like here (I used When the host first runs puppet and uploads it’s facts to Foreman, it will replace with the IP it got from Dnsmasq.

When you Submit the host, you should see a set of progress bars as Foreman creates the VM on libvirt, and the after you are redirected to the Host page, you can click the Console button to watch the install happen. Awesome!

So, now you have a Foreman VM which can create other VMs. You also have a puppetmaster on there, which will accept code pushed to “puppet@” so now you can test modules in a proper client/server virtual environment. Enjoy!

Step 5 - Extra Exercises

There are a few extra things you could do, if you want to play around some more:

  • Use a different database - mysql and postgres are supported
  • Add a different compute resource - EC2 is a good choice since you can get free machines
  • Add bridged networking - sometimes those VMs need to be on the main network…
  • Control DHCP/DNS with Foreman - this is tricky as you have to disable the built in dnsmasq server and set up bind9 and isc-dhcp-server on Foreman.
Go forth and create virtual machines!

Updated Archlinux ISO Details

Sigh. Nothing stays still in the tech world, does it? :)

The Archlinux Releng team have drastically altered the method for installing Archlinux since I wrote my previous article. In particular, they have dropped AIF and resorted to simple bash commands for installation.

This actually makes our life for Foreman much easier, but I thought I should write a small blog post to tell you how to use it.

Installation Media

Not much changes here, except now you can use the default release images, from https://www.archlinux.org/download/

You will still need to loop-mount the iso for NFS or HTTP access and copy out the vmlinuz and archiso.img files though.

The /dev/vda problem seems to be gone now, hooray! Default-created libvirt guests from Foreman should work fine now, assuming you keep the “sed -i mkinitcpio.conf” change in the Finish template.

The script=… parameter in your PXElinux template remains the same, but at present there is a bug in the current release which prevents it working. The installer will boot to a root shell, at which point you have to manually install wget and run the foreman script:

    pacman -Sy wget
    wget -q -O /tmp/startup_script http://foreman/unattended/provision
    bash /tmp/startup_script

Once this bug has been fixed and should be released in the next stable iso. Or you can use https://wiki.archlinux.org/index.php/Archiso#Configure_our_live_medium to build your own ISO with wget in it - worked fine for me, with no cusomizations.

New Templates for New Times

The only real changes are what data we feed to Arch - it used to be AIF profile data, now it’s just a bash script. The installation is roughly 2 parts:

    1) Partitioning/mounting the disk and running pacstrap
    2) Chrooting into the install for everything else

So, as is fairly standard for Foreman, we can use a ‘provision’ type template for (1) and a ‘finish’ for (2). Let’s see an example:


# Simple script to provision new style Arch install

### Disk setup / mount
<%= @host.diskLayout %>
### End disk setup

# Base install
/usr/bin/pacstrap /mnt base

# Install Grub
/usr/bin/pacstrap /mnt grub-bios

# Fstab
/usr/bin/genfstab -p /mnt >> /mnt/etc/fstab

### Create finish script
/usr/bin/wget <%= foreman_url(“finish”) %> -O /mnt/bin/foreman.finish
/bin/chmod +x /mnt/bin/foreman.finish
/usr/bin/arch-chroot /mnt foreman.finish
rm /mnt/bin/foreman.finish

# Finish up
umount /mnt
/usr/bin/wget –quiet –output-document=/dev/null –no-check-certificate <%= foreman_url %>

# Reboot
/sbin/shutdown -r now


/bin/echo “<%= @host.shortname %>” > /etc/hostname

/bin/ln -snf /usr/share/zoneinfo/Europe/London /etc/localtime

/bin/echo -e ‘LANG=”en_GB.UTF-8”\nLC_COLLATE=”C”’ > /etc/locale.conf
/bin/echo -e ‘en_GB.UTF-8 UTF-8’ > /etc/locale.gen

/bin/echo ‘KEYMAP=”uk”’ > /etc/vconsole.conf

/bin/sed -i ‘s/MODULES=.*/MODULES=”virtio_blk virtio_pci virtio_net”/’ /etc/mkinitcpio.conf
/usr/bin/mkinitcpio -p linux

/sbin/modprobe dm-mod
/usr/sbin/grub-install –target=i386-pc –recheck /dev/vda
/bin/mkdir -p /boot/grub/locale
/bin/cp /usr/share/locale/en\@quot/LC_MESSAGES/grub.mo /boot/grub/locale/en.mo
/usr/sbin/grub-mkconfig -o /boot/grub/grub.cfg

/bin/echo ‘root:<%= root_pass %>’ | /usr/sbin/chpasswd -e

# Other stuff

pacman -S –noconfirm openssh
/bin/sed -i ‘s/DAEMONS=(\(.*\))/DAEMONS=(\1 sshd)/’ /etc/rc.conf

# Yaourt repo
echo ‘[archlinuxfr]
Server = http://repo.archlinux.fr/$arch
SigLevel = Never’ >> /etc/pacman.conf

pacman -Sy –noconfirm yaourt base-devel

# Install stuff with yaourt
yaourt -Sy –noconfirm puppet

cat > /etc/puppet/puppet.conf << EOD
<%= snippets “puppet.conf” -%>
/usr/bin/touch /etc/puppet/namespaceauth.conf 
/usr/bin/puppet agent –config /etc/puppet/puppet.conf –onetime –tags no_such_tag –server <%= @host.puppetmaster %> –no-daemonize

Partition Table

# Blank the disk
/bin/dd if=/dev/zero of=/dev/vda count=1 bs=512

# Single partition
/bin/echo -e “n\np\n1\n\n\nw\n” | fdisk /dev/vda

# Format partition
/sbin/mkfs.ext4 /dev/vda1

# Mount partition
/bin/mount /dev/vda1 /mnt

Wrap up

So there you have it. You can install extra packages, configure services, etc at the end of the Finish script as per normal. Enjoy!

Setting Up a Home Office

It’s been a while, so I thought I’d write about what’s been keeping me busy this last few weeks. In short, I’m about to start working from home, so that’s required me to really think about how I work, and how I can improve my productivity.

Location, Location, Location

As with buying property, the location is critical. Until recently, my study has been on the ground floor of my house, and that room has only one tiny window in the corner. It’s also fairly difficult to heat in the winter.

I’d initially started by looking into daylight lamps and electric heaters to combat this. Then I went up into our loft space to get some boxes to start clearing the study, and I had a revelation. The loft is brighter, warmer, and has the added advantage of being physically separate from the rest of the house. That means at the end of the day, I close the hatch to the loft, and I’m “home”. Work stays in the office, as it should. I also great a great view with that daylight.

(Anyone pointing out that I do my hobby for a living, and tend to work in the lounge will be shot :P)

So, I have a comfy place to work. Now what?

Surviving IKEA

Now I need furniture. As it happens, I have a large desk that I bought some 5(ish) years ago, which has survived several house moves, in which it’s been dis- and re-assembled each time. You’ve got to hand it to IKEA - their furniture can last. I’ve also got a quality office chair with decent back support.

So the basics are covered. What else? Well, I’m trying not to go crazy here. It would be possible to spend a huge amount of money on things I don’t actually need. So, I’m getting a decent laptop stand and a whiteboard for sketching thoughts, and the rest can wait.

So what’s interesting here? Well, I thought I’d give a nod to my favourite room design tool, XFig. I needed to be sure that my desk would fit, especially as it’s a modular system. I needed to design which parts of the desk would work.

XFig helps here by having 2 features that work well together. The first is the ability to edit the nodes of a object (say, the corners of a rectangle). This means I can put one corner at 0,0 on my grid, and the other at wherever my tape measure says it should be. The second is the ability to group objects into a single entity which be dragged, rotated, duplicated directly.

Thus, a complex object like a L-shaped desk with modular components can be built up and added to the plan of the room. Here’s the design I came up with:

Not all the desk/storage space listed on the right as been purchased yet, but it will be :)

I’m also happy that I have spare space around the ladder hatch - at some point, I can make that a comfy space to have a coffee and a think.

So how does it look?

Not too shabby, and with plenty of room for personalisation, when I know what flourishes I want.

Task Management

So, I have a place to work, but I also mentioned my efficiency. I’ve been on the hunt for a good task manager for a while now, and I have some stringent requirements:

  • Must be date focused. I need to know what I need to do today. Too many systems are note or content focused, about grouping things into workbooks with no due dates.
  • Must be able to sync to my phone. I get many ideas when I’m at my desk, so a single-machine solution isn’t acceptable.
  • Scriptable API - I have a number of useful hooks in my window manager (XMonad) that can be used to add notes without navigating away from my current task.
  • (Optional) Calendar integration
  • (Optional) RSS Feeds of task lists
I tried Evernote and barely got started with it before dropping it. I then tried SpringPad, which I’ve been using now for a while. SpringPad has a lot going for it, especially with it’s ability to recognise webpage types and add the appropriate type of item to the workbook you’re editing.

However, about 1 month after I joined SpringPad, they removed the “Alerts” bar which showed me my upcoming (and overdue) tasks. Bummer. So it became an idea repository, and the hunt was on for a new task manager.

This week, however, I found something that satisfies all of the above. Tracks is a lovely bit of software, which is an implementation of the GTD (Getting Things Done) methodology. I’ve been using it for a couple of days now, and I love it. 

The install is currently problematic - after 4 attempts which ended up in either db:migrate errors or asset problems, I gave up and used their BitNami installer. In general I don’t like these pre-packaged installers, as I already have an Apache and a Mysql server in the house. However, in this case, I’m left with no choice. I will revisit the install in a short while, now that I know I want to continue using Tracks.

There is one thing it doesn’t come with - an Android app. However, the developer behind Shuffle has done extensive work, and made it possible for Shuffle to sync with Tracks. Shuffle doesn’t perfectly mirror Tracks, and I’m not sure I’d want to use it as the primary task manager, but for making tasks on the move and uploading them to Tracks afterwards it’s perfect.

AS a side note, I learned something interesting about Chromium - if you launch a page like “chromium –app=http://mytracksbox/” then you get application-like window, with no location bar, bookmarks etc. Makes it feel like a stand-alone task management application. Perfect!

Wrap up

So, I’m all set. I have a way to get things done, and a place to get them done in. The next few weeks will be interesting :)

A Cornucopia of Techy Bits!

So, a combination of a short holiday and a new laptop have conspired to keep me away from my text editor for a short while. The holiday was a walking trip, carrying our own gear, along the coast of Scotland. It was truly fabulous, but already covered at length on my wife’s blog so I’ll not repeat her here.

As for the new laptop, well…. there’s nothing like a new machine to make you re-evaluate your working environment. So I thought I’d share with you some of the tweaks I’ve been making over the last week or two.

Operating System and Hardware Tweaks

So the laptop I’ve gone for is a Dell XPS 17” (L702x). It’s a beefy piece of gear, with more than enough horsepower to run a Foreman testing environment on libvirt. As you might expect, I have Archlinux installed upon it. The majority of the hardware worked out of the box, as one would expect. There are a couple of things though:

  • The special keys next to the power button send out key symbols which are duplicates of other keys, so I can’t use them. No big deal.
  • The card reader won’t actually read cards unless you rescan the SCSI bus - I guess it’s because it’s on the same bus as the eSATA port which would also need a rescan. Again, not a deal breaker.
The only complex piece of hardware is the Optimus video setup…

Configuring Optimus

No, my laptop is not a Transformer, awesome though that would be. Optimus refers to nVidia’s new dual-video-card solution for laptop power management. Simply put, the laptop ships with two video cards - an Intel i915 for day-to-day work, and an nVidia 555GM for any heavy 3D crunching (such as, say, Diablo 3 :P). It’s a really clever solution, because the nVidia card is powered down when not in use, so when coding on a train journey, you can get much greater battery life.

However, this is not “two video cards” like a desktop would be - there’s no physical access to the nVidia, the DisplayPort and HDMI connectors go to the Intel. The nVidia is hidden ‘behind’ the Intel and you have to use Optimus to get to it.

Linux support for Optimus isn’t perfect yet, but it’s pretty close. The driver package is called Bumblebee (http://bumblebee-project.org) and it works by running the nVidia card on a second (headless) X server (on DISPLAY :8) and then shipping the frames to the Intel card via VirtualGL. When the application exits, so does the X server, and the nVidia card shuts down to give you better power saving.

I won’t go into the install process, since the Bumblebee and Archlinux wikis cover it all very well. Once set up, simply prefix a command with ‘optirun’ to start programs on the nVidia card.

I guess there’s not much to learn here, but as I was wary of buying an Optimus laptop, I thought I’d share with you all that it works fine, so don’t be worried about it :)

The Joy of Window Managers

Now I can run applications on either video card, I’ll need something to manage those applications. Enter the window manager…

Old friends will know me as a long time KDE user, but recently I’ve been on a minimalist, tiling window manager kick. For the last few months I’ve been using Awesome, and I’d definitely recommend it to people new to tiling WMs as it has excellent defaults.

However, I’ve recently switched to XMonad for my window manager of choice. It’s been around along time, and as such is much more stable than Awesome. The config file is more complex (it’s written in Haskell), but there are many examples to follow scattered across the web.

You can find my config here and my XMobar status bar config is in the same repo. I particularly like the Tabbed window layout for console work, especially if I’m working on unrelated tasks. The Full window layout is good for gaming - just fire up “optirun wine <game>”, and hit Mod-b to allow it to overlap the status bar. Fullscreen gaming, easy. Of course there are a galaxy of other layouts too, detailed in the XMonad docs.

I also found this excellent post series which I intend to work though sometime soon: Pimp Your Xmonad. Looks handy.

Vim Overload

I’ve been using Vim as my primary editing environment for 10 years, and I still learn things about it all the time. I’ve dabbled with other editors, but I always come back to Vim.

I’m not going to bore you with all my settings, as there are thousands of posts out there already on configuring every aspect of Vim. What I will do is link my personal Vim configuration (github.com/vim.git) and also point you to this blog post where I stole most of my ideas from.

Just to whet your appetite though - my top 3 picks from the modular side of Vim would have to be Pathogen, Tabular, and NERDTree. I use them all the time.

More to come?

So that’s my laptop taken from unboxing to being usable. But what’s that I hear you ask? Where’s the Foreman setup? Fear not! I have a blog post in mind about building a libvirt/KVM based Foreman development environment, and guess what the host is going to be… :)

Until next time!

Automated Archlinux Installs With Foreman - Part 1

The Foreman developers have just merged a small patch which allows the UI to detect and correctly display Archlinux hosts. This means it’s now possible to configure Foreman to do automated installs of Archlinux, just as you would for Debian or RedHat. This makes me happy, as people who know me might guess!

The problem

The thing is, Archlinux isn’t easy to net-install. It pains me to say it (because I’m a big fan of Arch), but their approach to automated installs just doesn’t fit with how super-smooth the rest of the distro is. Here’s a rough rundown of the steps you have to perform:

* Install  TFTP/DHCP (Foreman has us covered there :P)
* Download the Archlinux netinst iso (~200Mb, bleah)
* Loop-mount it and copy out the kernel and initrd
* Set up an network mount of the loop-mounted ISO
* Set up the templates for PXE and AIF (Arch Installer Framework) in Foreman

So yeah, we have to download a 200Mb CD, unpack it, and copy out the initrd/kernel. Not cool, guys (although if any of the ArchLinux guys want to point me at a better way of doing this, I’ll happily update the Arch Wiki in return :P). This is compared to Debian, where you can simply wget a 6Mb initrd and 2Mb kernel from ftp.debian.org, feed a preseed, and watch it fly.

It’s not all bad though, and it does work, as you will shortly see :)


Ok, I’m going to assume you have a working Foreman install - Puppet, TFTP, DHCP, etc. If you’ve not got that far, then worry not! I plan to blog about the more basic setup in a few weeks, or you can checkout @fitzdsl’s excellent blog post here

I also assume you have a machine (virtual or otherwise) that you want to install Archlinux on - no surprises, eh? :)

Getting started

Commandline configuration

So, we need the CD. Unfortunately, the Arch Releng guys don’t make a symlink to the latest CD that always works, so you have to go to http://releng.archlinux.org/isos/ and pick a date (latest is probably best), and then grab the netinstall CD. We’re also going to need a directory to serve it from, so we’ll loop mount the ISO. For example:

The last two lines are because Foreman will attempt to download the Archlinux initrd/kernel when you click “Build” on your (yet-to-be-created) Arch guest. However the Foreman codebase doesn’t yet know *how* to get those files, so we need to get ahead of Foreman and put the files there ourselves.

Finally, we need to arrange a way for the Arch installer to download the rest of it’s ISO image. There are three ways to achieve this: NBD, NFS, or HTTP. I used to use NBD, but the version of nbd-client in the current Archlinux installer is incompatible with the version of nbd-server in Debian Squeeze. So, that leaves two methods, and I’ll detail both.

*Note* in both cases, we’re serving the contents of the iso via loop-mount. If that bothers you, you could copy it out into a ‘real’ filesystem. Adjust the paths in the rest of the document if you do.


This is my preferred option as it doesn’t require downloading the whole CD into memory on your guest (at ~200Mb, it would struggle to cope on a small VM). The setup is simple enough:

Remember to adjust your network range to suit your setup.

*Note* I had some issues initially with the installer just hanging when mounting the NFS share. These went away before I could find out what was happening, but if you hit this you can always use the HTTP version below.


This is easier, in some ways, since we already have apache2 running for Foreman. However, since Foreman occupies port 80, we’ll need to configure another one. I’ll use 23575:

End of Part 1

That’s all for the configuration side - you now have a server which is capabale of building Archlinux installs. Join me in Part 2, where we’ll see how to configure Foreman to take advantage of the new configuration!

Automated Archlinux Installs With Foreman - Part 2

Welcome back! In Part 1, we got our server set up to build Archlinux machines. Now we need to configure Foreman to make use of it! Let’s get started…

UI configuration

Operating System setup

Firstly we need the Operation System. If you already have some Archlinux clients, this might already be done, but head over to the Operating System page:

and either edit the existing Archlinux OS, or create a new one if need be:

Then edit the page and edit the details to match this:

Don’t worry about partition tables, mirrors, or templates yet - we haven’t created them! Obviously, you may have a different list of architectures - tick what’s right for you.

Save the OS, and move on to Installation Media (just above Operating Systems, under More). Add a New medium (mine is “Archlinux mirror”), give it the URL “http://ftp.archlinux.org/core/os/$arch”, and associate it with your OS type:
Actually, the value for “Path” is just a note for ourselves - due to the fact that Foreman will interpret the $variables and Arch wants the mirror with the literal ‘$’ symbols as above, we can’t actually use this path - we’ll be putting the real mirror directly into the installation template. Save that, and we’ll move on to templates.


We’re going to need a total of 5 files to finish setting up our Arch installer - a PXElinux file, an install script, an AIF profile, a partition table, and a finish script . Be *very* careful with backslashes - AIF uses a lot of shell variables, and we need to be sure that Foreman doesn’t try to interpret them.

For all five of these files, you’ll need to go to the Association tab and tick “Archlinux 1.0”  to associate the template with Archlinux. You can call them what you like, but if you don’t use my suggested names, be sure to update the templates where it’s referenced.

Firstly, we’ll add a partition table to More -> Partition Tables:

As per the Troubleshooting notes, this is all set up for a machine installing to /dev/sda - you can find more examples on the Arch Wiki

Then we’ll go to More -> Provisioning Templates and create the following four files:

Be sure to check through those files for IP addresses that need to be altered for your environment - there are quite a few. In the PXELinux template, change the ONTIMEOUT parameter to arch_http if you’re using the HTTP boot method from page 1.

Now go back to the Operating System page, and edit “Archlinux 1.0”. Ensure the correct Partition Table and Installation Mirror is ticked on the first page. Now go to the Templates tab, and assign the templates to the OS:

Build a Host

We’re finally there! You should now be able to create a new host, assign the correct OS, media, and partition table, and watch it build. Enjoy your tasty Archlinux machines!

Troubleshooting / Notes

I’ve found a number of issues with the present installer ISOs, which have been reported to the Arch Releng team. Most I have been able to work around in the installation templates, but one serious bug remains with GRUB - I cannot make a Virtio-based disk install correctly at all. As such, if you are testing on a virtual machine, I recommend editing that machine and using a SATA based disk which appears as /dev/sda instead of /dev/vda.

Any issues, please do report them in the comments and I’ll update accordingly.

Inaugral Post!

So it seems that finally started to have enough things to say to start a blog. My interest are varied, and I don’t have enough material for a blog on a single one of them, so you’re going to see blogs on topics like:

  • Linux / DevOps
  • Cooking / Baking
  • Walking
  • Politics
  • Generally awesome things

You’ll also get anything else that pops into my head. I’ll try to tag things properly, but no promises :)

So, here’s my new blog. In the time-honoured tradition of programmers, I think this is appropriate:

echo "Hello World"