You might have noticed a system outage last night, unfortunately this was beyond our control and we did take steps to warn everyone in advance -- if you didn't get the memo here's what happened.
We received this message from our hosting provider a couple of weeks ago;
At 11pm on 7th August, we will perform a series of updates to the systems on which your server is hosted, which will require rebooting these machines. This is scheduled maintenance work designed to improve system performance and increase reliability.
When we run the updates, we will send your system a shutdown request, however some configurations may have this disabled.
With that in mind, we suggest you perform a shutdown of your server, before the work starts, at 11pm on 7th August, allowing you to ensure that your server has been shut down nicely. Once the migration has completed, we will turn the server back on.
The reboot will cause a downtime for your server that should not exceed 3 hours.
... it was well over 3 hours, but everything seems to have gone smoothly and normal service has now been resumed.
When these things happen we automatically switch into "disaster recovery mode" and take steps to ensure that we have a plan in place just in case the hosting provider manages to balls it up completely! -- not that we don't trust them of course!
Our disaster recovery implementation consists of;
- Ensuring all website data has been backed up to the moment prior to the shutdown
- Taking multiple backup copies and storing them in separate secure locations
- Ensuring that our "mirror server" is up-to-date and ready for a switch-over if required
Thankfully we didn't need to implement the disaster recovery plan this time, but like fire drills, this is a good exercise in being ready for the worst!
Thanks for bearing with us during this exercise!