The Cole family learned a hard lesson this week about "points of failure" and operational redundancy.
I won't bore you with all the details, but suffice it to say the event involved an amusement park, a set of lost car keys (the point of failure), the lack of a spare set on hand (redundancy) and multiple problems with the recovery plan intended to get everyone back home safe and sound.
Throughout the ordeal, and despite that fact that yours truly initiated the "human error" that set the disaster in motion, I couldn't help but notice how closely the situation resembled the challenges faced by CIOs every day. Or perhaps I just work too much.
Nevertheless, concern over single points of failure in the data center have grown in recent months thanks to the rise of the cloud. While it may seem counterintuitive, the cloud actually represents a return to centralized, mainframe architectures following years of decentralization, according to tech blogger Mark Rushing. When you get down to it, the cloud is simply a collection of CPUs and storage that, when it fails as demonstrated by last month's Amazon outage, can bring an entire enterprise to its knees.
That's part of the reason why more and more voices are calling for a "many-cloud" approach. As described by The Armada Group's Brad Vaughan, broad interoperability and open standards like Red Hat's emerging API set are the best way to avoid vendor lock-in, improve data portability and federation, and allow for greater elasticity and resource pooling.
Whether you're talking about the cloud or traditional data center architecture, there simply is no substitute for adequate redundancy. Kevin Hayes of Michigan's Wayne State University describes a recent reboot of the institution's firewall - a task that normally would have cut off all student and faculty data communications. But by wisely provisioning multiple modules within its security architecture, traffic is simply rerouted with a few keystrokes. For a planned outage, the system avoids a mere inconvenience. For an completely unexpected event, however, it could be a lifesaver.
According to Forrester, enterprises across the board plan to increase DR budgets by 5 percent over the next year. Much of that will undoubtedly go toward systems and technology, but there is something to be said for planning, testing and continually updating DR programs as well.
The cold, hard fact of life is that every now and again, the unexpected happens. And while it's best to put as much forethought into preparing for bad things, fate has a way of throwing curve balls. Rather than beating yourself up for failing to keep on top of things, maybe give a few moment's thought to ponder the question: "What if ...?"