WordPress, Ansible, DigitalOcean and something about pizzas

Author: Andy Hartwell

Let’s say we run a pizza joint, and we bake a mean calzone. We can tweak that pizza for different customers’ requirements (vegetarians, those with a dairy allergy, or just sensible people who don’t like anchovies). We follow a recipe to cook our pizza, and customise various steps, ingredients and measurements to suit each client.

We have a killer recipe for hosting WordPress sites, and like any recipe, we can tweak the ingredients or the measurements to suit each site. We use Ansible to put our recipe into action (to bake our pizza), and because of the way it works, we can run that recipe on multiple machines, each one of them installing and configuring just the bits it needs (like baking custom pizzas in a number of ovens).

Our Ansible playbook (the recipe) consists of a number of plays (steps), many of which have customisable variables (ingredients and measurements) that can be tweaked on a per-site basis. The steps go like this:

  1. Increase the available memory on the box (by creating a swap file, so if the box runs out of “physical” memory it can borrow it from the hard disk)
  2. Set the timezone of the server to the UK to overcome any potential date-based issues
  3. Install all the prerequisites we need (the Apache web server, MySQL database server, etc)
  4. Download WordPress and any third-party plugins we need
  5. Download the code and any bespoke plugins for that particular site (via Git)
  6. Configure mail sending (for servers where we don’t use an external email provider)
  7. Install WordPress, activate plugins, and activate the site theme
  8. Configure Apache to serve the site, activating any of the necessary modules
  9. Setup scheduled jobs that need to run in the background (like backing up the site to S3)

The only missing bit is finding a box for our pizza (or a server for the website… OK, so the analogy’s wearing thin at this point). We use DigitalOcean, which is a great source of fast virtual machines with IP addresses we can spin up in around a minute. DigitalOcean comes with one-click recipes for WordPress – and lots of other apps – but that doesn’t quite conform to our setup. So we just spin up a bare Ubuntu server, create and assign an SSH key – we use different keys for every server – and run our script on the server, using the SSH key to login as root.

We then put the pizza in the box (deploy WordPress on that server) by following our recipe. Simple, eh?

Why we think this is ace

Administering a loosely-coupled WordPress network – where each site runs entirely independently but is administered from a single point – means we can keep our clients up-to-date with the latest security and functionality releases of the CMS core and any third-party plugins each site requires. But not all plugins are created equal, so we can opt a server out from updates for a day, while we figure out how to implement the changes the plugin has introduced. We can even install plugins that aren’t on the WordPress plugin directory (that are instead stored in S3) for which we have multi-domain licenses.

As you any server admin knows, running a WordPress server is like having a big target on your back, but here’s where running a loosely-coupled network trumps one that’s hosted on a massive server with one database. If a site falls over because a script kiddy has found out how to run a denial-of-service attack – it still happens! – we can quickly find and diagnose the issue, ban the IP address(es), and then deploy that ban across the entire network, so if that kiddy tries to find their way into another box, if it’s provisioned via our Ansible recipe, they’ll have no luck.

Things to be aware of

DigitalOcean servers are like Ikea furniture. They’re stable until they suddenly break, but they’re cheap and easy to assemble, so you can get a replacement really easily. We use DigitalOcean’s floating IPs, which work a little like a poor man’s load balancer. Instead of pointing a website’s domain name to the actual server that’s hosting the site, we point it to a floating IP. That floating IP points to the server, but can be switched on a sixpence, so if that server fails, we can (almost) instantly spin up a replacement, attach that to the floating IP and carry on as normal. (We can either clone the box itself, or provision a fresh new box and restore the database from our nightly backup… so far we’ve not had to do this, but the plan’s in place should we need it.)

Because we’re using technology where the cracks are beginning to show – Apache and MySQL not being the modern developers’ first choice – we find that it’s useful to keep an eye on the servers, and give them a quick reboot if they’re running slow or losing memory (which happens regularly). The reboot process takes all of 15 seconds, and we do it automatically via a small Python script that talks to the DigitalOcean API. And, because we like fractals, that’s also hosted on DigitalOcean.

Our little watchman script nudges us in Slack if it’s had to restart a server, and if it can’t contact DigitalOcean or an individual server for whatever reason, raises more of an alert. This means we always know when something’s amiss.

Staging

We use Springloops to host a lot of our code. It’s a Git store but comes with added benefits, like managing deployments for us. Our Ansible recipe creates a virtual user account that is only accessible internally – by the server itself – and by Springloops. No-one else can login to the server using those credentials, because only Springloops’ IP addresses are permitted, and Springloops can only use the wordpress username. Our script runs as root, then hands ownership of the files it downloads over to the wordpress user.

Ding! Pizza’s done

This setup is around 9 months in the making, and now runs for the majority of our WordPress clients. We run regular morning sessions – called WordPress Wednesdays – where we run through our staging and production servers to keep them all up-to-date: upgrading the plugins and core code. We’re also experimenting with caching systems like Redis, and once we’re happy with its stability we can deploy that across the board, so everyone gets the benefit.

We’ve built and run WordPress websites for knocking on 10 years, and between this and our Heroku setup for meatier sites, we think we’ve got a pretty good setup going.