There was a great conversation at work today, about what a real emergency is. Someone pointed out that a real emergency is something you haven’t planned for. In this case, it wasn’t the total system crash (that’s all in the disaster recovery plan) or the mirror server not being online when it crashed (because that’s not really a problem and it was only a couple of hours of data lost). It was that the crash happened at the end of the month, when pay normally is transferred into bank accounts, and the four hours recovery time would make a significant difference to how long the pay transfer took. And there were a couple of members of staff who had mortgages to service and credit card bills to pay and speed of access to their dosh was a bit vital. So the manager at this GP practice went round the car park and emptied all the parking meters. And with the fine collection of £1 pieces and other items of change, managed to procure enough cash to provide emergency tide-over to anyone who really needed it.
It was wonderful. I only heard about this second-hand, because our IT manager/support queen was out there with tranquillisers, damp towels, caffeine tablets and the off-site back-ups. Apparently the manager told everybody what the situation was, and they all decided who really needed the money. Of course, that might be because it’s a doctors’ practice, and let’s face it, doctors are normally in the job (or at least started in the job) because they wanted to help people one way or another, so it’s not really surprising that that’s how they dealt with the problem.
Could the problem have been averted? I don’t see how. Even if we’d done a full model and considered the effect of different scenarios at different times of day or coinciding with crucial events, we probably wouldn’t have done anything about it. And let’s be honest, we wouldn’t have spent the time or effort modelling, what happens if … the crash happens during a royal wedding. We might have got as far as thinking, what happens if it crashes with someone in the office/no-one in the office, but I don’t think, realistically, we’d have considered what happens if the four hours downtime coincides with payday. And if we had, we would have decided that it probably wasn’t worth the effort of providing cash on the premises to tide people over.
So I might blog about risk analysis another day, but until then, I want to send many many thanks to the guy who saved our bacon by opening the parking meters. Or rather (because our bacon is absolutely fine, thank you very much) the guy who helped the staff members who would have been severely disadvantaged by the side-effects of a risk that had been assumed to be acceptable.