Almost every outage starts with a change: a new version, an adjusted setting, a modified process. That's why we want to be able to see, for every change, why we made it, who checked it and whether it did what we expected. That way a fault can be traced back to a specific change, and you know what changed before you start looking.
Pull requests and the four-eyes principle
Changes to our code travel through pull requests on GitHub. Nothing goes to production without approval from a second person: that is our four-eyes principle, and it is the heart of how we work. A pull request records why we changed something, what exactly changed and who checked it. Every pull request also gets a review from GitHub Copilot, which often spots mistakes or missing edge cases before a colleague even starts. It supports the human review; it doesn't replace it. If a change touches our security objectives, the Security Officer looks at it too.
The person who wrote the code does not approve their own pull request.
From proposal to closure. Every change has an owner. We record the reason, what and who it affects and the plan. We assess the effect on confidentiality, integrity, availability, quality and continuity, and whether it fits contractual and legal requirements. After implementation we check whether the change did what we wanted and whether it caused side effects or new vulnerabilities. We distinguish changes to our products and services (including client-specific configuration) from organisational changes such as policies and ways of working; the principles are the same.
In practice: an emergency change may go ahead before approval when something critical is at stake, but it is reviewed afterwards.
Separated environments. Development, testing and production run in separate environments on separate servers, as similar as possible apart from the data and the connections to external systems. That way you really test before a change reaches clients, and only authorised staff can reach production. A list of security rules for our technology is part of what we call "done", and we automatically check the components we use for known vulnerabilities.
Configuration management. Code is only part of what changes. Settings, versions and sources are changes too, so we manage them in the same way and can roll them back. How we do that is described in our post on configuration management.
How a release works
We number versions by fixed rules. Larger updates are "minor" versions (1.2 to 1.3), which for us are effectively the big releases. Small improvements and bug fixes are patches, and the procedure for those is lighter. A release runs in five steps:
Preparing.
- Update the version number and pull in security updates for components.
- Remove debug code.
- Test new database migrations on a copy of the test database.
- Try the version in at least two browsers, one of them without extensions.
The code then goes through review to the test environment, where we check everything works, especially the connections.
Acceptance test. For a minor release the client gets a message with the new features and the length of the test, which is at least several weeks. We can't count on the client to find our bugs; the client judges whether the software does what's intended or expected. For a patch we skip this.
Choosing the moment. Outside office hours, ideally so users notice nothing. Whoever does the release keeps plenty of time free afterwards to deal with anything that breaks. For a minor release we agree the moment with the client and allow at most one release moment per week across the company. The client is told, and the application shows a banner.
Releasing. We test migrations once more on a copy of the real database. If something has to be done by hand on the database, we take an extra backup first and log the action. Before the final merge, everyone present holds a go/no-go moment: has everything been checked, and is it still wise to release? Afterwards we check live that everything is reachable and the data looks right. The client receives a release email listing the changes.
Server maintenance. On a minor release we also update the operating system, restart if needed, check that everything came back up, that backups are in order and that disk space and memory are healthy.
Setting up a new environment. For a new environment we follow a fixed checklist: update the server, create a dedicated user with limited rights, give it its own configuration with unique passwords and ports, set up a secured connection, add protection against repeated login attempts and connect it to uptime and log monitoring. Because every environment is built the same way, you know what you'll find when you change something later.
After a release, we check whether the change did what we intended and whether it had side effects. What we learn feeds back into our way of working, as described in our post on continual improvement: an unexpected outcome becomes an improvement item with an owner.