Production server died. Restored from backup in a day
Production server died. Restored from backup in a day
The backup test was not a drill: the server was shut off over an unpaid invoice, taking the control panel, the inbound connections and every client config with it. Restoring from a backup went from an abstract checkbox to the only plan for the day within minutes.
What saved it
The backups were already in place and on another host: a 03:30 cron job pushed the state to external storage with two-week rotation. This is exactly the case they are set up for — not “useful someday” but “today, because yesterday’s server no longer answers”.
The 14-day rotation is not an arbitrary number either: it survives a weekend and a vacation without turning the storage into a landfill. In one day I restored the panel, the inbound connections and the transport, and switched the CDN to the new address.
Two traps along the way
Swapping a live SQLite database. You can copy a database file over a running service and everything will look fine — right until the restart. SQLite keeps a write-ahead log next to the database, and some of the recent data lives there at the moment of copying. Replace only the database file and the log stays behind from the old one, and the two disagree. The correct order is to stop the service, checkpoint the log into the database, and only then touch the files.
Locking myself out over SSH. A socket configuration applied before I had verified access, and network login closed. From there only the provider’s console helps — the one nobody tests until it is needed. The order is different now: first make sure a second way in exists and works, then change the thing you are currently connected through.
The takeaway
A backup without a rehearsed restore is not a backup, it is hope. The incident stayed cheap precisely because the copies and the external storage were prepared before anything broke, not after.