Is there anyone at the state government level who is involved with disaster recovery implementation and testing? I’d be especially interesting in hearing from people in DHSS.
Here’s the deal in a nutshell. Delaware Health & Social Services (and more broadly, Delaware state government in general) contracts with a private company to help us with our disaster recovery efforts. They provide data center space, servers, networking equipment, etc. If we were ever to suffer a New Orleans or World Trade Center type of disaster that took out some or all of our technology infrastructure, we could relocate to any Sunguard location and set up shop there.
Twice a year we test the whole system.
The thing is that every time we test it — and this would be just as true in an actual disaster — we have to rebuild our entire infrastructure from scratch. Much of it comes from backups but we’re still rebuilding.
I’ve been campaigning for years for us to team up with one or two other states. These would have to be in a geographically different region. We offer up some percentage of our network and server capacity and the other state offers up a similar amount of theirs. We each permanently house some of our production systems with our partner state.
users would access and use these systems just as if they were still being hosted right here. All technologists would have remote access to their systems just the same as if they were installed locally. They’d also have an open invitation to call or make site visits at any time.
By actually running part of our daily operations from a remote location every single day, we wouldn’t have to rebuild anything. Say we partnered with two other states and gave up 20% of our total capacity to each of them. If we truly did suffer a catastrophic disaster, I’m sure we could pare down to just our most critical systems and run things indefinitely on 40% capacity. At the same time, all that our partner states would stand to lose is 20% (assuming similar sized operations) of their capacity if we went down entirely.
On the other hand, they stand to gain the same things that we do. Namely, a secure safety net that would not be subject to the same risks as we each face locally. (A hurricane that hits Georgia is not also going to hit West Virginia. A severe ice storm that cripples Wyoming is not likely to hit Oklahoma at the same time. An earthquake in California will not be felt in Illinois.)
An added benefit, aside from essentially instant recovery and knowledge sharing, is cost savings. We wouldn’t have to pay a private company to insure against something we hope never happens. Neither would the other state (or states) that partner with us.
Up to now, everyone who hears my proposal agrees that it makes sense. But no one has the clout or persistence to actually implement it. So my thought is if I can do the preliminary ground work myself — find states with similar technology and similar sized operations which fit the geographic bill — and can present a near turn-key solution, maybe I can get someone higher up the food chain to take it seriously.
So I ask again, is there anyone at the state government level who is involved with disaster recovery implementation and testing who’d like to tilt at windmills with me?
What you speak of is Disaster Avoidance. Spread out your workloads and when it is time to cash in that insurance policy i.e. disaster time, you are running at say 50% utilization. Depending on your SLAs with app owners, you could power down and transfer workloads/compute power to your mission critical applications. Good post!