Emerald Reverie

Gwmngilfen's blog - Tech, cooking, walking and other randomness from the heart of Scotland

Walking the Arran Coastal Path - Day 7

| Comments

  • Walk distance: 8.3 miles
  • Walk time: ~4.5 hours
  • GPX file
  • KML view on Google Maps





Goatfell from SannoxGoatfell from Sannox

So we come to the final day, and we still hadn’t made a decision about the towering mass of Goatfell. The plan throughout the week had been to do Goatfell on the last day. There were two reasons for this; one, to do it when we were supposedly at out fittest, and two, to look back from the highest point of Arran on everything we had walked over the course of the week.

Walking the Arran Coastal Path - Day 6



Day 6 - Lochranza to Sannox





The Arran Alps
So, the penultimate day, taking in the north coast of Arran as well as a famous scientific landmark. Better yet, our target for the day was, in fact, the very cottage we were staying in! As such, we could leave the heavy weight in our room, and take just one light pack with food, water, and first aid kit. Joy!

The weather was again bright and clear, and we were expected high temperatures. After a superb breakfast from out hosts, we were driven back to Lochranza to start the day. We stopped to grab a picture of the so-called Arran Alps; a view of Goatfell and it’s lower siblings from the north. We then started off around Loch Ranza and picked up the northern trail to Hutton’s Unconformity.

Ex-scientist ponders the Unconformity
Hutton’s Unconformity is a place on the north coast of Arran where two different rock strata meet, but at almost 90 degrees. Along with a number of other similar formations around Scotland, this helped Hutton disprove the geological theories of the day, and in doing set some of the principles of modern geology (including establishing it as a proper science). That it has a splendid view of northern Kintyre and Bute does nothing to spoil it’s significance.

The next section of the walk is extremely difficult to describe - but this time, in a good way. If all walks were like this, everyone would be doing it. The temperature was mid-twenties, with a light sea breeze. The terrain was easy walking, with splendid views across the sea and into the island. We took a break
just before the An Scrior rockfall, and leaned in the shade tossing pebbles into
the sea. I could have stayed there all day, it was bliss.
An Scrior

An Scrior was not as difficult to cross as the rocks around Black Cave, as there was a reasonable path above it, and we soon passed into a verdant coastline. After lunch, the heat and the horseflies both became brutal - we estimate it was 32C in the shade by 3pm. We drank copious amounts of water, but it was never going to be enough in those temperatures, and before we reached the forest just north of Sannox, there was no shade at all. We took a few stops, but the flies were not about to give us any peace, so we soldiered on through some of the remotest parts of Arran.

It’s 5 miles to the nearest road from here
I hope you like privacy…
Once in the trees, out of the direct sun, it became once again an extremely pleasant walk. We were tired, but the light in the trees was a lovely bonus. We returned to our B&B (which conveniently has a garden gate on the Coastal Way itself) at around 4.30pm. Given it was our last night on the island, I treated myself to steak for dinner :)

That left one more day, and the big question was still unanswered. Goatfell loomed large on the horizon as the sun was setting….





Walking the Arran Coastal Path - Day 5



Day 5 - Pirnmill to Lochranza










Clisham Lodge, Pirnmill
So after an excellent rest day (involving the joy of buses!), and our delayed-but-not-forgotten meal at the Lagg Hotel, it was time to shoulder the bags again. Over half-way distance-wise, and with the longest day done, we were in good spirits. Doubly so, since this was the shortest day of the walk. As such, we breakfasted in the excellent Clisham Lodge in good cheer.

The plan called for a short road stretch to Catacol, then over the headland on the old Postman’s Path to Lochranza. The added joy is that Lochranza is home to Arran’s own whisky distillery, and given the short distance, we aimed to be arrived in time for a tour and a taster before being picked up by our host for the evening at 4pm. A grand plan.
The Apostles of Catacol

It was not such a sunny morning as others on the walk, overcast and somewhat humid. The sea across to Kintyre was unbelievably still; it barely rippled. If it weren’t for the midges flying around, it would have been possible to think time itself had stopped. The road was easy going when fresh from breakfast, and we were at the base of the Postman’s Path in Catacol before 12. Catacol’s only noteworthy fact is that it’s cottages are called the Apostles, for reasons that were explained to me at the time, but now escape me. They were good resons, I assure you :)

Looking back the way we had come
After a short break we started the ascent, and it’s on such short sharp climbs that I realised how heavy my bag was - the knees really protested. But it was over quickly, and we ambled along above the road as we rounded the headland. The view was only spoiled the the horseflies once again going crazy - several new bites were obtained.

We dropped down into Lochranza itself by just after 1pm, and as we rested on a bench, we discovered Mrs ER had picked up a big collection of ticks from the close vegetation as we passed. Ticks are not pleasant critters, and in rare cases can transmit the extremely nasty Lyme’s Disease, so we immediately broke out the tools and removed them. Gladly, they were so quickly
discovered that some hadn’t even latched on yet, and were easy to remove.

Lochranza Castle
Vowed to do a more thorough check later that evening, we set off on the last mile to the distillery, picking lunch from a cafe along the way. Worth the wait - the Arran distillery was lovely. I’ve been on numerous tours, and the process is no mystery to me now, but the bonus tasting of Arran whisky (unpeated, fairly sweet, with lots of spices - good with Christmas Pudding, we think) as well as Arran Gold (a whisky-based cream liqueur, like Baileys, but considerably nicer) was really the point of the entry price ;)

Finally we were collected by our host, as we were staying in nearby Sannox for the evening. Darven Cottage’s owners were incredibly welcoming, and we felt right at home. After a shower and a meal, we hit the sack, feeling ready for
the north part of the island the following day….






Walking the Arran Coastal Path - Day 4



    Day 4 - Blackwaterfoot to Pirnmill









    Setting off towards King’s Cave
    A night’s rest had done a hell of a lot to restore my spirits, but as we an excellent breakfast in Blackwaterfoot, and contemplated the OS map in front of us, it was hard not to be a little worried. We had approximately 11 miles to go (an underestimate, as it turned out), and about half of that was by road. I’m no fan of road walking - I’m slightly overweight as it is, and with a heavy pack, the unyielding surface tires my feet out very quickly. Still, our accommodation was booked, and for once so was our evening meal - 7.15pm at the Lighthouse Restaurant, so we had to get rolling whether we liked it or not. 


    Looking back to Drumadoon Point


    The first 4 miles or so were some of the nicest of all. The path from Blackwaterfoot around Drumadoon Point and out to King’s Cave was of excellent quality (we were experts in this now, after the nonexistant paths of the previous day). 




    King’s Cave
    King’s Cave itself was fairly impressive (allegedly it was even used as a schoolroom in the past), and it gains it’s name from a story concerning King Robert the Bruce. Pretty much everywhere in Scotland has a story about the Bruce though :)



    From there we climbed into the woods at Torr Righ, following a track around the edge that gave some great views over the moorlands back to Goatfell on the far side of the island. We then joined the road and strolled down into Machrie for lunch.



    *How* much food?
    After some cold drinks and some serious munching (Mrs ER’s ‘Arran platter’ was impressive!) and a good bit of ‘feet-up’ time, it was time to tackle the road section. We had 5 miles to do to reach Imachar Point, where we would diverge from the road again. It was tough, and both of us were struggling at various points. We even had to resort to singing at times to keep moving. There’s not much else to say - road walking is fairly dull, even if the scenery was quite nice.

    After a short break (read: collapse) at the start of Imachar Point, we proceeded. On the plus side, it was only about 3pm - road walking is fast, so we still had 4 hours to cover the last few miles. On the down side, we were
    very tired by this point.

    Imachar Point - yet more bracken…
    The last section was varied - we started out on good grass/rock paths, but that quickly went to (yet more) shoulder-high bracken, and then eventually to a shingle beach. This was the last straw for my belaboured feet, as discussed yesterday, thumping my weight down on loose rocks is not fun.

    We limped into Pirnmill around 5.30pm, and somehow found the energy to stand up long enough for a shower. There was much self-congratulation over dinner - the hardest day was done, no other day was as long, we had a rest day coming up, and a short walking day after that. Bliss!

    Walking the Arran Coastal Path - Day 3

    Day 3 - Lagg to Blackwaterfoot








    Farewell, Lagg!
    This day was going to be good. Some of the wilder terrain on Arran, not too hilly, just a long way from towns and villages. Nice to get even further away from things. In addition, the distance we estimated to be 8 miles, which after the epic march of the evening before was a welcome relief.

    Since it was going to be a short day, we had a slightly later breakfast, and were underway by 10am. The main issue of the day was going to be lunch - there were no convenient towns to stop in en route, so we tried to get supplies in Lagg. Sadly the village store seemed to be closed. Not to worry, there was apparently another store en route to the coast path.

    Wading through bracken
    On yet another sunny day (in Scotland! Unbelievable) we set off, sad to leave such an awesome spot. On subsequent trips to Arran, I think the Lagg Hotel will be a strong contender for accomodation. The first section of the walk was road-based, and we strolled up out of Lagg and along a couple of pleasant miles to Sliddry, when the second store was said to be.

    While Sliddry does indeed have a store, it seems it’s a farming supply store rather than a food shop. Good job we still had the provisions we bought on day 1! Heading down a lane, we found ourselves back on the coast proper, and with good paths underfoot, we set off around the southwest corner of Arran.

    Wilder parts of the Arran Coast
    The paths didn’t stay good for long, and a few times we lost our way and started to make it up a bit. We’d heard on previous days that we “were going the wrong way round” - apparently nearly all walkers go anti-clockwise around the island. Here it made a difference, as the Coastal Way arrows only served to tell us when we had regained the path, rather than stopping us from losing it in the first place.


    Lunch was taken on some appropriately large and comfortably shaped rocks. We figured we had about 3 miles to go to Blackwaterfoot, and it was only 1pm. Easy work after the previous day. However, we hadn’t reckoned on the terrain.

    Can you see the path? This was a highway
    compared to what we had just done
    Our notes said “Uneven rocks underfoot, with constricting vegetation. Progress will be slow”. That was one hell of an understatement. The bracken was easily 5ft high, and concealed a large number of brambles which proceeded to hook onto our clothes at every opportunity. All this vegetation was growing on an uneven boulder scramble, which meant you couldn’t always see where your feet were coming down, and as such you had to test your footing carefully. The saving grace was that the rocks were marked with blobs of paint to signify the course of the path - which zigzagged up and down the side of the coast as we progressed northwards. At one point we lost the path entirely, and might have done the rest of the walk as a boulder scramble nearer the sea, but the one solitary other walker we saw that day gave us a point to aim for (“He’s got to be on the path right?”). Given the abysmal nature of the terrain, we arrived at Preacher’s Cave, 1 mile south of Blackwaterfoot, by 3pm. 2 hours for less than 2 miles - not the greatest speed ever, but we
    were glad to be out of the hell zone.

    Preacher’s Cave
    Preacher’s Cave itself was impressive (so called as sermons were held in it in past times), but weren’t in the best of moods to appreciate it. My feet were not happy at all - with my rucksack I was weighing around 17stone, and having that jolting down on one unstable rock after another was not fun. We took a good long rest, and then ambled the last mile into Blackwaterfoot itself.

    Another good meal at the Kinloch Hotel, followed by a pint in our B&B (Blackwaterfoot Lodge), as well as a chat with the owners of said B&B helped set things to rights.


    Finally, Blackwaterfoot in sight!
    We went to bed in good spirits, but I was worried - the next day was to be 11 miles, over half of which was by road (very hard on my already aching feet), and today had not been as restful as expected. If that wasn’t bad enough, we had a dinner reservation for 7.15pm, so we had a deadline as well. It was going to be a tough day…




    Walking the Arran Coastal Path - Day 2

    Day 1 - Whiting bay to Lagg






    Another glorious day, and another large breakfast - required though, as we set off on our longest day, both by distance and time. The issue was that we had two tidal sections to do - first, the boulder field at Dippen Head (which has an optional inland route if it’s a problem) and the coast at Black Cave (which doesn’t). Black Cave is impassable for 2 hours either side of high tide, and with that being at 4pm, we had to aim to arrive at either 2pm or 6pm. Figuring 2pm was unrealistic, we set upon a plan to arrive later and take our time.

    Glenashdale Falls
    Packs were settling in, and we were in pretty high spirits, so we started with a couple of miles inland to see the Glenashdale Falls - Arran’s own Niagra. Truly impressive, and a good way to warm up for the day, with wide easy trails that are firm underfoot. After a short break to take some pictures we headed back to the coast, within sight of where we’d left it.





    South to Dippen Head from Largymore
    Next up was a stretch of coastal walking from the edge of Whiting Bay down to Largymore Point, at which we had to decide about Dippen Head. Given some of the horror stories about taking 3 hours to cross it, and so on, we decide to take the road route and avoid it. We trudged over the top and down into the Kildonan Hotel, which was a very welcome sight indeed. Mrs ER commented when we left that she was sure the short lane leading to it was 3 times longer on the way in than when we left…

    There was a strong possibility that passing Black Cave at 6pm would get us to our hotel after they stopped doing food, so we decided to have a pretty sizable lunch - if all else failed, we had our emergency provisions in our packs.

    Seals enjoying the afternoon sun
    After food and a nap on the grass in Kildonan, we set off for Black Cave at 4pm. We saw loads of seals - apparently there’s a colony of 40-50 just off Kildonan. Our timing was perfect, and we clambered over the boulder field on the approach to Black Cave, arriving spot on at 6pm. Sadly, our tide forecast was not so good - it still wasn’t passable.

    We had information that the remainder of the walk should take around 1.5 hours, and that food stopped at 9pm in the Lagg Hotel, our eventual goal. But we were tired, and carrying heavy bags, so we assumed it would take longer. However, we had no option but to wait, so we rested and watched the tide. It definitely was going out, no doubt about it - but very slowly…

    The rocks we had to cross…
    By 7pm, I couldn’t wait any longer. There were enough rocks visible now for me to attempt to hop my way over the wet section - something I’d spent years doing as a kid. So I had a go. There were a few slippery moments, but the impassable section was only about 15 metres wide, and I soon figured out a route across. I then ferried both packs over, as Mrs ER wasn’t so confident, and she came along last.

    We could now actually see the cave - and it’s pretty huge. We didn’t admire it for long though - we had miles to go to Lagg, and if we wanted any food, we had less than 2 hours to get there (it was now 7.15pm). We set off at a surprisingly high pace.



    The awe-inspiring Black Cave
    I don’t know what happened next. The last 4km of this walk are a blur. I remember not really being able to see at one point because the sun was setting in front of me and my eyes were watering from the light. We were exhausted, yet we set one of the highest paces of the walk - arriving into Lagg at 20.40, less than the stated 1.5 hours. Astonishing.

    We paid the price though. Because of the big lunch, and the big exertion, we weren’t hungry. We’d pushed every limit to arrive in time to get a meal, and we weren’t hungry! We settled for an ice cold beer, and a dessert each. It was heavenly, and we spent the ‘meal’ discussing exactly how crazy we are, and how we managed to keep that pace up for over an hour.


    The Lagg Hotel
    We went straight to bed afterwards. It was a comfortable bed, but it could have been a floorboard for all that I would have noticed. I was asleep in seconds, knowing that the next day was considerably shorter… what a fool I was…. but more on that later ;)






    Walking the Arran Coastal Path - Day 1

    Day 1 - Brodick to Whiting bay









    Rothwell Lodge
    (Apologies for the text spacing, it seems necessary to insert a lot of blank lines to make the photos line up properly in Blogger. If anyone’s got tips for doing it better, let me know)

    The first day of our epic walk started with sunshine blazing in through the window - it was clear, blue, and going to be hot. I started the day with an epic walker’s breakfast (porridge and a cooked breakfast :P) and some lively chat with our host at Rothwell Lodge.



    Is this for real?
    We set off through the town of Brodick, grabbing supplies from the supermarket as we went. We had places planned to eat lunch and dinner for almost every day of the walk, but it’s a good idea to carry provisions in case of emergency. Looking back across Brodick to Goatfell looked like something out of a fantasy novel - it really was quite impressive.





    “Lunch is that-a-way”
    We left Brodick by road as there’s a small diversion due to erosion of the coastal path at the moment, but soon enough we were back on the coast and heading for Clauchland Point. Progress was slow over rocks and boulders, but as we reached the point and started to turn towards Lamlash, we had splendid views over to Holy Island. There were plenty of dragonflies about - they always fascinate me, for an insect they seem to big to fly.




    Holy Isle from Clauchland Point
    Lamlash was reached around 1pm, but my rucksack was started to really hurt my shoulders by then. This did not bode well - having issues with the pack half a day into a 7 day walk is not good. I grumbled my way over the last mile through Lamlash to the Old Pier tearoom, where ginger beer and ice cream was had (it was starting to really heat up by this point in the day). It also gave me a chance to examine my pack after my shoulders had relaxed a little, and I determined the the height of the adjustable back was probably just a little short, putting nearly all the weight on my shoulders, and very little on my hips. Easily fixed.

    Forestry tracks - no shade!

    We now headed inland, up to Dyemill and the Forestry track over the headland towards the Glenashdale Falls. It was blisteringly hot - 2pm is the heat of the day, and forestry tracks rarely have much shade, as the trees will be cleared back to a good 10ft either side of the track. Also, we were climbing uphill, so we had to take it slow.


    Holy Isle from the southwest


    Once the track levelled out, we could start looking for our turning left off the main track, down into Whiting Bay. This lead us down the hillside, with great evening views of the south side of Holy Isle, before passing a few houses, and heading north up the road to our second accommodation, the Burlington Hotel.






    The Burlington
    After the first full day of walking, we were pretty glad to see the place - it was nearly 6pm at this point - so we grabbed a fast shower while our host booked a table for us at the Trafalgar resturant 3 doors down. Having a meal within staggering distance was a welcome relief. The meal was great, the owner (Wolfi) being something of a local legend.

    We hit the sack early, as we had a big day ahead of us next - the only day with any real tide-dependant sections. I was tired, sure, but not exhausted, and pretty pleased with the first day’s walk. The next promised to be interesting…

    Walking the Arran Coast

    When I started this blog, I promised that this would be about tech, walking and baking. I’ve been quite good about blogging tech things, but there hasn’t been much of the other two. Time to fix one of those.

    I spent last week on a walking holiday, around the coast of Arran, an island in the Firth of Clyde, off the coast of Ayrshire, southwest of Glasgow. It’s postition is fortuitous - being on the western coast, it sits in the Gulf Stream, and yet is sheltered from the western Atlantic storms by the Kintyre peninsula. There are palm trees on some parts of the island :). It also lies across the Highland Boundary Fault - the tectonic line which gives Scotland a lot of it’s mountains. As such, it’s northern part is mountainous, and yet it’s southern half is rolling farming land. Not bad for an island 30 miles long. These features have earned Arran the nickname “Scotland in miniature”. It’s earned this not just for it’s geography, but it’s produce as well - Arran is known for it’s cheese, beer, and whisky.

    Arran has had a coastal path for a while - it actually has the domain “www.coastalway.co.uk”. We’d heard how nice Arran was from several friends, and we’d already had one holiday abroad this year, so we decided to go walk around it. There are a number of companies that will organise this kind of thing for you (and we have used them in the past) - they book accommodation, make dinner reservations, move your bags from one place to the next, all you have to do is turn up and walk. However, they’re quite pricey, so my darling wife spent a heap of time organising the B&Bs herself, and we decided to carry our packs ourselves.

    Since this was a one week walk, I’m planning to write up each day separately - I sent myself a email every day with my thoughts on the day, which I’ll expand with photos, GPX/KML data so on. For now I’ll give the total stat block for the walk, and my thoughts on arrival… stay tuned over the next few days if this interests you.

    (Technical note: Sadly, my Garmin GPS can’t store a week’s data, and I only discovered this when I got home, so I only have accurate GPS data for the last day. The rest has been recreated roughly, from memory. It won’t be far wrong though :P)

    Day 0 - Arrival in Brodick

    • Total walk distance: 71.63 miles
    • Total walk time: 7 days
    • GPX file
    • KML view  on Google Maps

    Friday was a crazy day. I had only got back from a business trip to Europe the day before, with a list of tasks I wanted to accomplish before my holiday. I also had to pack my rucksack for an 8day trip, lock up the house, and be in a taxi by 4.30pm. Somehow, I don’t know how, it all got done.

    I met up with Mrs ER on the train, and we switched trains in Glasgow Central - it was ridiculously hot and humid there, I felt sticky even without moving. Fortunately the train to Ardrossan harbour was air conditioned, and I could start to relax.
    The sea was very calm as we sailed over from Ardrossan to Brodick, and it was easy to grab a few snaps of the approaching island (and thus what we had let ourselves in for). To give you an idea of what we were in for, on the right is a picture of the whole GPX track laid out on a contour map (each day is a different colour).
    The approach was spectacular in the sunset, with the sun going down behind the right shoulder of the highest peak on Arran, Goatfell. The plan called for walking up that on the last day, on the way down to the ferry home. It looked pretty imposing from out on the ferry (although the light levels didn’t allow for the best photos).





























    Sadly, the chip shop was closed when we got off the ferry (9.30pm - that’s island life for you :P). So, we made do with snacks from the local supermarket, and headed to our first B&B for the night (the lovely Rothwell Lodge). It had been a crazy day of work and travel, and sleep was most welcome before the first serious day of walking.

    Stacking the Odds in Your Favour - Using OpenStack With Foreman

    It’s been a while since I last wrote a blog, but that doesn’t mean I’ve been slacking off. In the next series of blog posts, we’ll be looking at some of the new things in Foreman 1.2. But today, I want to make a small diversion…

    I recently got some new hardware in the house, courtesy of my employer, so I decided it was time to play with one of the other virtualization technologies out there. As existing readers will know, I’m a big fan of Libvirt and KVM. I’m in no way disatisfied with Foreman’s ability to manage Libvirt - but there’s little point in running it on 2 machines. I won’t learn anything new that way.

    So, what to use?

    My choice was really between two systems - oVirt and OpenStack. I had originally planned to run oVirt when I ordered the hardware, but delivery issues meant it took a while to reach me. In the meantime, I was asked to help out with some OpenStack/Foreman integration, and was quickly intrigued by it’s capabilites. I’ve been using it now for a couple of weeks, and it’s pretty great. However, I haven’t seen a huge amount of literature on OpenStack and Foreman, so I decided to explain how I set it all up to you.

    What’s OpenStack?

    There’s a ton of blogs about OpenStack at the moment - with so many companies putting time and effort into the codebase, and even more using it in production, it’s hard to not hear about it. So I’ll be keeping my introduction to OpenStack itself brief.

    OpenStack is a tool for building a private cloud at large scale - if you’re the sort of organisation that wants the compute power & flexibility of Amazon EC2, but on your premises, OpenStack is the way to go. It deals with co-ordinating real hardware as a cloud, and deploying virtual machines to those nodes, along with the associated tasks of storage, image management, networking, capacity balancing, and so on.

    However, despite the goal of a massively scalable computing system, it’s quite possible to run Openstack on a single machine. It simply means that services which would normally run on a dedicated system each, all run on the same box. Not optimal perhaps, but certainly usable.

    Stack ‘em up

    So, I’ve settled on OpenStack - how do I get rolling?

    DevStack was written for people in my position - those wanting to try out OpenStack, but lacking a decent pool of hardware to run it on. DevStack is a script which takes in a configuration file and then sets up a complete installation of OpenStack ready for use.

    (Aside: now that I’m more familiar with OpenStack, I may well try out some of the more other installers at a later date. DevStack is a great starting point while you’re getting used to the terminology though :P)

    DevStack requires an Ubuntu or Fedora base install - you’ll not be surprised to hear that I opted for a 12.04 LTS install. Once that was complete, I started to follow DevStack’s execellent all-in-one install guide. I’ll not bore you with repeating it here, but I will post my localrc file (which is the configuration input for DevStack)

    greg@amethyst:~/devstack$ cat localrc | egrep -v "^\s*#|^\s*$"
    ADMIN_PASSWORD=<random string>
    MYSQL_PASSWORD=<random string>
    RABBIT_PASSWORD=<random string>
    SERVICE_PASSWORD=$ADMIN_PASSWORD
    HOST_IP=172.20.10.35
    LOGFILE=$DEST/logs/stack.sh.log
    LOGDAYS=2
    SWIFT_HASH=<random string>
    SWIFT_REPLICAS=1
    SWIFT_DATA_DIR=$DEST/data
    SERVICE_TOKEN=<random string>

    I’ve stripped the comments to keep it short, and most of it is specific to my network - but as you can see, it’s a pretty short config file for such a big project. Once we’re done with the config, we can run the installer

    ./stack.sh

    This should run to completion, but it does take quite a while :P. We can now access Openstack on our machine, as a webservice (so, in my case, at http://amethyst/) using ‘admin’ and the password from the localrc file.

    Err, so?

    It’s all very well saying ‘it’s running!’ but what do we do with it? Well, Openstack is primarily an image-based system, so the next step is either to download or build some images to use.

    If you’ve already logged in, you’ll have seen that Images & Snapshots is a menu entry in the navigation sidebar. Going there, we have Create Image, which will ask for a source URL (or file to upload). A few distros do provide images ready for use (see the OpenStack documentation on obtaining imagesfor more):

    Image DIY

    Sadly, my two favorite distros (Debian and Arch) do not provide a prebuilt image for OpenStack. Debian have started a project to build a Wheezy or Jessie image from a script (see this thread for the discussion) which should eventually end up as a package available to install on Debian which will build an OpenStack image.

    You can download this script from their git repo (git://anonscm.debian.org/openstack/openstack-debian-images.git), and I’ve also forked a copy of it myself. I then used it as a base for doing the same steps for Archlinux, which has been moderately successful. The only thing I haven’t yet (at time of writing) managed to do is to have the root filesystem expand to the full size of the disk on boot (which the Debian images do via cloud-initramfs-growroot). I’m sure I’ll get it working though. You can find my current versions on my GitHub page).

    If all else fails, you can use Libvirt (you knew I’d get that in here somewhere, right?). The OpenStack documentation has walkthroughs for both CentOS and Ubuntu, using an interactive install in virt-manager, which work fine for any distros where you can’t get a prebuilt image (I did this for Squeeze, as the above Debian script doesn’t work on Squeeze).

    Once you have an image, you can upload it to OpenStack using the Create Image button. You should also be able to test lauching an instance and make sure it boots. Hooray! We have a working virtualization platform! Well, almost….

    Networking Fun

    The only thing that wasn’t set up out of the box (as far as I can remember, after three weeks of using OpenStack) was the floating IPs. What’s that? Let me explain…

    Those of you who have used Libvirt in any serious endeavours will be aware that there are broadly two networking approaches. The first is to create a network bridge between the physical network device on the libvirt host, and the virtual machines. This has the effect that all VMs are on the public-facing network, and firewalling is the job of the VM. Alternatively, one can create a virtual NAT network inside Libvirt and have the VMs attach to that - which makes them hard to get to. If you want to reach a NATed VM from your laptop, you either need to do some SSH tunnelling, or set up some static port-forwards on your Libvirt host. Neither option is particularly awesome, although bridging works OK if you have control of the ‘public’ network too.

    OpenStack (and other platforms, for that matter) approach this by having a pool of ‘floating IPs’ on the public network (in this case public means my house network, not the public internet). These IPs are reserved for use by OpenStack, so I had to reduce the pool of my DHCP server to avoid clashes. These can then be assigned to a VM from the OpenStack console. This gives you the best of both worlds - you still get the security of being on an isolated network (you have to configure the Security Groups if you even want to be able to ping your hosts on the assigned floating IP), but you can then access the VM on the public IP, eliminating the need for SSH tunnels (which can be awkward if you’re testing webservices, for example).

    I added 49 IPs to my OpenStack instance using the following inline bash script:

    for n in `seq -w 151 199`; do nova-manage floating create --ip_range 172.20.10.$n; done

    Your mileage may vary, as I’m writing this from memory (and .bash_history), but you will need some floating IPs for the Foreman stage.

    You’ll also need to allow SSH access (22 TCP) to your VMs using the Security Groups. I used a simple rule of ‘22 TCP from 0.0.0.0/0’ (i.e everywhere) on the ‘default’ security group.

    Stack The Deck

    So, we have OpenStack up and running, and after ~150 lines of blog, you’re wondering when I’ll get to the Foreman bit - well, that’s now ;)

    I’m going to assume a few things about your setup:

    • Foreman is already configured with an appropriate Architecture
    • Foreman already has an Operating System that matches the distro image you uploaded to OpenStack
    • Foreman and OpenStack are on different machines
    • Both are on the same network (in my case 172.20.10.0/24)
    • Foreman controls DNS for this network (optional)

    The first step is to add your OpenStack API credentials to Foreman. Go to More -> Provisioning -> Compute Resources and add a new one. Give it a Name of course, and select OpenStack as the type (slightly obvious, but if you don’t see OpenStack, ensure you have the additional compute packages installed on your Foreman server). The URL is the API endpoint, which will be something like http://amethyst:5000/v2.0/tokens (set the hostname as appropriate). The Username/Password will be ‘admin’ and the admin password from your localrc. Then you can press Load Tenants and pick the one which you uploaded your distro image(s) to. Hit Test Connection and (assuming it’s fine - remember that Test Connection silently succeeds) save the CR.

    Now we need to map the image on OpenStack to the Operating System on Foreman. Click the new CR and select ‘New Image’. Select the appropriate image from the dropdown, and fill in the rest of the boxes (be careful with ‘Username’ as it varies from distro to distro - Ubuntu use ‘ubuntu’, Debian use ‘debian’, and my Arch image uses ‘root’). Save the image.

    We’re almost there, but we also need a Finish template to run on the machine when Foreman provisions it. You could use any existing Finish template,but I’m going to add an OpenStack specific one (I also created an ‘openstack’ Puppet environment, to completely isolate my test area, but this is optional). Under More > Provisioning > Provisioning Templates I created a new template, containing just

    mkdir -p /root/.ssh
    echo "<my ssh key>" >> /root/.ssh/authorized_keys

    Assign this template to the appropriate OS, and also restrict by environment if you’re using one. You can always extended this further, using exising templates for inspiration (such as installing packages, running Puppet, etc). You can also check out Ohad’s EC2 blogpostfor more ideas - OpenStack works very similarly to EC2, and those scripts should work for OpenStack too.

    Deal the hand

    So, after a lot of UI setup, we’re ready to try it!

    Go to Hosts -> New Host, select your OpenStack CR from Deploy On, give it the usual things like Name, Host Group, Puppet Environment, etc. You’ll notice Network only has a choice of Domain now (which matters for your DNS records). On the Operating System tab, after selecting the correct Architecture and Operating System, you should be able to select the Imageyou created above, and load the templates to make sure the new template is being used. The Virtual Machine tab allows us to select how many resources we should give to the new VM, and importantly we must select the Floating IP network (otherwise Foreman won’t be able to reach the VM). You should also select the Security Group which has SSH access (otherwise Foreman won’t be able to log in).

    Once we’ve done all that, Submit the Host, and you should see Foreman spin up the VM on OpenStack, SSH in, and run the template.

    Magic! :)

    Conclusion

    OpenStack plays very well with Foreman, as they split the required duties - OpenStack deals with all the resource management, quota management, security, and networking, and simply tells Foreman the IP it chose to assign to the host. Foreman can then do it’s usual job of managing DNS, certificates, and providing Puppet on the host with ENC data (obviously DHCP and TFTP are not required in this scenario).

    As a result, I now have the joy of booting my most commonly used distributions in ~1min rather than the ~20min it takes to do a PXE install on Libvirt, with DNS, and (if requested) a full Puppet run already done. Of course, I still have my Libvirt host of course, and can do a PXE install if I need a specific setup on the host, but so far I’ve had little need for that.

    While this blog is quite long, the above took me less than a day to figure out. OpenStack is fairly intuitive (assuming you’ve used some other virtualization before), and of course, once you get to Foreman, you’re getting the same consistent interface you’re used to. It’s all very smooth, really.

    As ever, if you try this yourself, do let me know how you get on!

    Managing SSH Host Keys in a Reliable Way

    I’ve been managing my virtual machines using Foreman for close to 2 years now, and that’s brought me a huge set of benefits in terms of how I test new code (or changes to existing code), and new packages. That’s just awesome :)

    But repeated rebuilds of a machine lead to one small niggling problem. One which bites you on every rebuild. One which doesn’t stop you working, but requires a few extra keypresses after every rebuild, and possibly at every login.

    Not got it yet? Does this look familiar?

    [greg:~]$ ssh test2
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that a host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    c1:63:5e:c2:e4:c7:2a:19:fd:80:11:a2:73:c2:f6:b1.
    Please contact your system administrator.
    Add correct host key in /home/greg/.ssh/known_hosts to get rid of this message.
    Offending RSA key in /home/greg/.ssh/known_hosts:217
    RSA host key for test2 has changed and you have requested strict checking.
    Host key verification failed.

    (If it doesn’t, the answer is changed SSH host keys, but the rest of this blog probably won’t make much sense :P)

    Yeah. Irritating, isn’t it? Every time, every rebuild, a few seconds wasted. It adds up. There’s got to be a way to make this go away, right?

    The ‘Traditional’ Way

    The way most Puppet users might approach this problem would be to use one of Puppet’s greatest features: exported resources. This feature allows you to exchange information between hosts that the puppet master doesn’t know a priori.

    This is a quick solution, from a code perspective, and making sure all the machines in your infrastructure have each others keys is really simple (example courtesy of Puppet Labs’ documentation:

    class ssh {
    # Declare:
    @@sshkey { $hostname:
    type => dsa,
    key => $sshdsakey,
    }
    # Collect:
    Sshkey <<| |>>
    }

    The problem with this is two-fold. Firstly, it’s a quick win in code, but only if you’re already using exported resources. See, to use them, you have to be using a database backend for Puppet itself. Given the database backend is otherwise optional, not everyone does. If I want to write a nice solution for everyone to use, this isn’t going to work.

    Second, it’s also slow. When a machine changes it’s host keys, it’s going to take two puppet runs for the systems to catch up - the first is when the new VM uploads it’s new key to the puppet database, and the second when my laptop retrieves the new key and updates it’s key list.

    So that’s actually slower than just deleting the changed key from my known_hosts. No good.

    The ‘Foreman’ Way

    My next thought was to look at how Foreman can be used to replace exported resources. This has been covered in other blogs (The Foreman Blog covered this a while back).

    So, since the facts of a host are uploaded to Foreman we could replicate the above code by doing a Foreman search for all the $sshkey facts and writing them to a file. Pretty neat.

    However, this doesn’t help either. Not everyone uses Foreman (even if I think they should :P), and we still have the 2 run problem which makes it slower to fix the problem than to suffer it. We’re not getting closer…

    Inspiration Via Services

    The solution came to me when working on a module for backups. I needed a way to allow the backup system to SSH onto the backup targets to initiate rsync. Ideally, I didn’t want to store the private key in puppet, since I was planning to publish the repo. But I also want the module to work without an end user having to manually add an admin ssh key for the service.

    I Googled around and came across this: Github: fup/puppet-ssh

    This is a function to generate keys as required on demand, and store them on the Puppet master, and make it possible to read them back (both the private and public parts) for use in Puppet manifests.

    Bingo! With this function I can create a key the first time it is requested, but thereafter it will be re-read from the keystore dir on the puppetmaster. Since the puppetmaster isn’t the VM being rebuilt, when I recreate my vm, it’ll get the same SSH key as last time it was built.

    The function didn’t quite fit my needs, as I want to keep some types of keys separated by environment (development backup servers should have access to production machines, for example), so I forked the function and extended it a little. You can find the result at Github: GregSutcliffe/puppet-modules, but let’s take a quick look at how the module works.

    The Code

    Most of the ssh module is fairly tedious - make sure it’s installed, manage the config file, start the service, yadda yadda…. The only interesting bit is the key handling. Lets take a snippet straight from the repo:

    $rsa_priv = ssh_keygen({name => "ssh_host_rsa_${::fqdn}", dir => 'ssh/hostkeys'}) 
    $rsa_pub = ssh_keygen({name => "ssh_host_rsa_${::fqdn}", dir => 'ssh/hostkeys', public => 'true'})

    What’s happening here? Well, a couple of things. Firstly the dir parameter is my extension to Fup’s original function - it allows me to specify where to store the keys on the puppet master. Otherwise, we’re asking the function to read (and if required, create) a key named ssh_host_rsa_myvm.fqdn.com and read both the private and public parts into appropriately named variables.

        file { '/etc/ssh/ssh_host_rsa_key':
    owner => 'root',
    group => 'root',
    mode => 0600,
    content => $rsa_priv,
    }
    file { '/etc/ssh/ssh_host_rsa_key.pub':
    owner => 'root',
    group => 'root',
    mode => 0644,
    content => "ssh-rsa $rsa_pub host_rsa_${::hostname}\n",
    }

    Here we take those variables and apply them to the server in question. Thus, when this module runs, it will overwrite the auto-generated ssh key with the one from the puppetmaster. As such, whenever I log into the machine, it will always have the same key (and the same fingerprint) so my known_hosts file is happy.

    Success!

    Security Caveat

    There is a minor security issue - all the keys generated by the function live on the puppet master. Technically, if they got into the wrong hands, that could be bad. However, as the machine which hands out configuration data to your infrastructure, if someone compromises it to the point where they can read those keys, it’s already game over. Plus, if they leak some other way, regenerating all the keys of your infrastructure is only a rm -rf /etc/puppet/ssh away ;)

    So there you have it - consistent SSH host keys for your machines every time! Better, it works for everyone, regardless of database backends or other external stores of data. It’s also fast - since I do a small Puppet run as part of the provisioning of my machines, the host key is already set when it comes up at first boot. All my requirements met. Wonderful :)