What happens to us if a datacentre goes away?
You find out whether your restore works. Mine did, but only because I did it live. A datacentre under a platform I'd built went dark, and I rebuilt the platform elsewhere in days with no data lost. The backups were fine. The restore had never been rehearsed, and that's where the days went.
What this usually turns out to be
You have backups. You’ve never restored all of them, in order, onto empty machines. Nobody has, until the day they must.
What I’d do
- Restore everything once, on purpose. Onto empty machines, with a clock running. Whatever breaks is your real recovery plan.
- Write the order down. Secrets and identity before applications. Storage before databases.
- Size the second site to run, not only to store. A site that holds your backups can’t necessarily carry your load.
- Make failover boring. For a bank’s GitLab I kept the same SSH host keys on the standby, so git clients didn’t notice the switch.
What it takes
A rehearsal costs a few days. The first real outage without one costs more.
What I won’t do
Sign off a recovery plan that has never been run.