Debian Packaging, the Transparent way
So I've been managing the Debian packages for about 8 months now, and every so often I get asked if there's anything people can do to help. I have to answer "Not really" because the way we're building the Debian packages is somewhat arcane.
At least, it was. This blog is to tell you all about how it's now much more open.
Packaging in the openThe first piece of the puzzle starts with our foreman-rpms repo (github.com/theforeman/foreman-rpms)
in this repo we store all the packaging data for the foreman packages, both deb and rpm. In the case of Debian, you'll see a structure like:
For example, the foreman-proxy nightly package for squeeze is at:
Now, the eagle-eyed and quick-witted among you will already have thought "That's quite a lot of code duplication". You're absolutely correct, but this method allows us to package each distro in it's own way, if need be. At the moment, it's all very similar, but it might not stay that way. It also makes it trivial to add in more distributions later (say, quantal or wheezy).
So, now you know where our package data is, you'll want to know; how does it get used? Well, here we turn to the king of automation - Jenkins.
JenkinsYou can find our Jenkins server at ci.theforeman.org. In there, you will find a set of interesting packaging jobs, specifically:
There's two types of jobs here. Let's start with the "individual" jobs. These are what they sound like - they take a set of parameters about what package the are building and build the required package(s). For example, if I feed it the options
arch = 64bit
distro = squeeze
type = nightly
Then it checks out the foreman-rpms repo, and changes to the appropriate directory in the repo, according to those options. It then runs 'build.sh' found in the packaging directory, which uses Pbuilder to build the package for the requested settings. At the end of the job (if it built successfully) the package is uploaded to our Debian repository at deb.theforeman.org
The matrix jobs, as you can guess, call the individual jobs many times over - once for all the possible combinations of the options. Thus we can have the matrix job track the Foreman git repo, and rebuild the devel packages whenever a commit is made.
How to get involvedNow that things are more open, it's easy to get involved. Here's a workflow you could
1. Get a build boxYou don't have to go through the trouble we did of configuring pbuilder. Just build a
machine with the OS you'd like to run the packages on
2. Install basic packagesYou'll need 'build-essential' and 'devscripts' as a minimum (well, on Squeeze
3. Check out the repoFork the foreman-rpms repo and check it out to your buildbox. If you're not trying
to update an existing distro, copy the closest match to a new repo.
4. Build the packageYou don't need to use 'build.sh' for this (that's crafted towards pbuilder and other
small automated task we need to do in the final packages). Just run "debuild -us -uc"
and see if it builds. If it doesn't either
a. Install the missing build-dependencies, or
b. Fix the broken packaging
5. TestYou can either test on your buildbox, or for best results, on another machine with
the same OS installed. Hopefully the package will install and work ;)
6. ContributeCommit your changes to your repo and send us a pull request on foreman-rpms. We'll
test your changes and merge if it looks good.
ConclusionHopefully that demystifies our package building architecure a bit. If nothing else,
this blog serves to remind me how we set it all up when I can't remember in six
months time ;)
EpilogueIf anyone is interested in replicating the whole chain, you can get our pbuilder
puppet configuration from our infrstructure repo at github.com/theforeman/foreman-infra
This repo contains the puppet code we use to build the various parts of our
infrastructure, including the Debian build slaves for Jenkins. The only thing you won't
find there is the GPG key to sign the packages, for obvious reasons ;)