Many outages are not caused by code, but by a setting someone once changed by hand and then forgot about. Configuration management is our answer.
Key idea: at any moment we can say how a system is set up, why, and since when.
A concrete example
This may sound abstract, so here is a recent example. Users could no longer access one of our applications: a critical incident (see incident management). The cause was a rate limit that blocked all users.
After debugging, we found that the rate limit used the internal IP address instead of the external IP address. In short, we had to change the configuration to fix the problem.
The short answer could have been: configuration changed, application working again, client happy, us happy.
However: this could affect more than one client.
That is why we record the configuration item centrally, correct the faulty setting and then make sure the other client applications are updated too.
Every setting is a configuration item
A configuration item (CI) is a logical group of settings with one purpose: the web server settings, say, or those for log collection, security tooling or scheduled tasks. The versions of installed packages count too. Each CI has an owner and its own version number, independent of the others.
One source of truth
For every item we define the official source. For our own infrastructure that is a central, version-controlled store, organised per component. Each component has a short readme containing at least the version number. If there is a risk of compatibility problems with the application, it also holds a changelog and the combinations we have tested.
The rule is: what runs on a server comes from that source and from nowhere else. That means changes go through the same review as our code: proposal, review, approval. You can see who changed what, and you can retrieve an earlier version.
For other people's software we only use public, widely used package sources that track vulnerabilities. A private source is allowed only if we or a direct partner control it. Whoever installs something also checks that the security updates are applied.
Knowing what runs where.
Next to the source we keep a configuration database (CMDB), part of our asset overview. For each production server it records:
- who owns it and which project it belongs to
- which environment it runs in
- which application version is on it, and which version of each CI
- who performed the last release and when
That lets us compare a server with what it should be, and spot drift: the slow deviation from the agreed state.
Changing during a release.
We change production configuration only as part of a production release, in the agreed window. The engineer:
- retrieves the approved version of the CI from the source;
- checks the syntax and whether the CI fits the application version going live;
- applies it, by hand following fixed steps or with a script;
- updates the CMDB with the new version, the reference to the source, time, deployer and the related change request.
Skipping a step would cost us later: an accurate CMDB is why we know afterwards what is running.
Rolling back. Where needed, each change request has a rollback plan. If a new version causes problems, we execute that plan, restore the previous version from the source and correct the CMDB. If there was downtime or a security exposure, a review follows.
Exceptions and manual intervention. Sometimes you have to step outside the standard route, for example for an extra backup before a release or a restore after an incident. That means using higher privileges than the settings were meant for. Such manual actions on production, outside a fixed procedure, are recorded in an action log. Where possible we plan them in advance and have a second person approve them (four-eyes principle). We also set up our tools so you can't easily hit the wrong environment by accident, for example with clear labels.
Network, at a high level. The same principles apply to networks. Development, test and production environments are kept logically separate and access is given with as few rights as possible. At home or on the road you work only on managed or approved devices, with encryption, a firewall and automatic updates, and you use only approved secure connections to reach company systems.
Findings from configuration reviews, such as an outdated CI, drift between a server and the CMDB or an unpatched vulnerability, become improvements with an owner, as described in continual improvement, our Quality Day and audits. Thanks to the versions and rollback plans, we can also rebuild a server if we have to.