The customer data entrusted to us is the most valuable thing we manage. That is why we make sure we have good backups.
Why one copy is never enough. Keeping more than one copy has a simple reason: things break in ways you didn't anticipate. Hardware fails, software writes something incorrectly, or someone deletes exactly the wrong folder.
More important than the number of copies is making sure they cannot fail in the same way. Two copies that are overwritten by the same bug, contain the same mistake or share the same threat are, in practice, one copy with twice the false sense of security. For every copy, we therefore ask: what could affect this copy?
That is why, alongside periodic backups on the server, we also keep an offline copy (cold storage) on a secured USB stick. Anything that is not connected to the network cannot be erased or encrypted from the network. This is sometimes less convenient, but that is exactly the point.
How many backups should you make? If you make a backup every ten minutes and keep every copy, your storage will be full in no time. That is why we use exponential backoff for how often we save a backup:
- We make a backup every 10 minutes and keep it for one hour.
- We make a backup every hour and keep it for one day.
- We make a backup every day and keep it for one month.
- We make a cold-storage backup with every release.
Checking that it works. A backup is only worth something if the data in it is complete and usable. That is why we regularly restore some of it to check: are all the tables included, and does the process work in practice? An exercise that does not go well is not bad news: it is better to discover a problem on a quiet Tuesday than during an outage.
Restoring a backup.
If something goes wrong, we naturally restore a backup as quickly as possible, taking our release procedure into account. Two terms help us agree on what "recovered in time" means:
| Term | Meaning | Description |
|---|---|---|
| RTO | Recovery time objective | From the moment something fails, how long may it take before we are working again? |
| RPO | Recovery point objective | How much recent work can we afford to lose before it becomes a problem? |
Anything that proves difficult during an exercise or after a real disruption becomes an improvement with an owner, following our approach to continual improvement.