How to Avoid Downtime with a Proper Disaster Recovery Plan
When Hurricane Katrina slammed into New Orleans 10 years ago this week, countless businesses were submerged in water and were partially or totally destroyed. Those without reliable disaster recovery operations in place paid an enormous price. In many ways, those with reliable disaster recovery operations in place paid an enormous price, too. But at least for them, their core business operations were unscathed.
A compelling example of the rewards of that foresight and preparedness is the case of Cooperative Processing Resources (CPR), a debt management system software provider in Richardson, Texas. CPR, which oversees a network of agencies that provide credit counseling services to consumers, has a New Orleans office that took a direct hit from Katrina, and was shut down. The good news is that the services provided by that office were back up and running within an hour.
That near-instantaneous recovery was made possible by the data hosting and disaster recovery operation that CPR has had in place since 2002 with Wayne, Pa.-based Sungard Availability Services. I recently had the opportunity to speak with CPR president Kate Campion about her company’s disaster recovery operations, and I began the conversation by asking her about the backup strategy the company had in place since its inception in 1994, and before Sungard AS came along. For many IT veterans, her response will bring back a lot of memories:
We did have some typical nightly backups in a tape rotation. In 1994, all of our agencies were also doing a nightly tape backup—taking that tape remotely offsite every night, and doing weekly and monthly backups themselves. Sungard AS came into the picture around 2002—that’s when we began offering the service of data hosting. That helped many of our customers with their disaster recovery, their nightly backups, and their security monitoring. We provided all of those services to our customers via Sungard AS.
Campion said what enabled CPR’s preparedness when Katrina hit was the simple fact that the data was no longer onsite:
Of course we did have some customers who continued to host themselves, and they have their own disaster recovery programs. But the services that Sungard AS provided not only took care of having built-in disaster recovery for the agencies if something were to happen to their locations, but if something horrible happened to the Sungard AS location, Sungard AS had its own disaster recovery to point all of our customers to another backup site. We were very interested in that—many of our customers could not afford such a complex and complete backup and disaster recovery system, so this a very popular service that we’ve provided.
Campion noted that CPR did a lot of vetting before selecting Sungard AS as a provider. I asked her what that selection was based on, and she said Sungard AS had the complete package:
A lot of the other companies that we looked at could do the data hosting and some of the nightly backup, but they didn’t have anywhere near the reliability, nor the expertise, that Sungard AS had. Because we were offering this as a service to our customers, that was critical to us. We needed a spotless reputation; we needed the security expertise; and of course the cost was obviously a factor.
I wanted to get a sense from Campion of what problems companies need to be aware of that aren’t addressed by the disaster recovery service provider. I asked her what issues arose as a result of CPR’s New Orleans office being submerged in water that fell outside the scope of what Sungard AS does. She said it was simply the destruction of the physical location:
We wouldn’t necessarily become involved with that, since our primary concern is with their data. And of course there was no downtime at all with Sungard AS around Katrina. We simply rerouted the remote offices to be able to access the data themselves directly, instead of going through the main office, which was shut down. Within an hour, they were up and running. The impact to our New Orleans office was the fact that since they couldn’t get to the main office, many of the main office employees, once the water had subsided and the roads were somewhat passable, could drive to the remote offices and pick up their responsibilities from there. But CPR continued to provide service within an hour of the disaster striking, because they were able to access all of their data. The database and our application actually run their offices, so everything that they needed to perform their services was there at Sungard AS.
I told Campion that in my experience, and in the experience of everyone I’ve ever spoken with on this topic, no disaster recovery situation ever occurs without glitches of some sort. So in the case of Katrina, what didn’t go exactly as CPR would have liked? What did CPR learn from that experience that enabled it to improve its disaster recovery operations post-Katrina? I got a kick out of her response:
Like I said, they were up within an hour. Could we have asked them to be up within 30 minutes? I don’t know how you would improve on that. There were no glitches at all. The sad part of New Orleans was the fact that the main office was basically destroyed. Katrina didn’t make the people in the company suffer in terms of service. They suffered in many other ways, with the loss of their equipment and that kind of thing. But certainly not in terms of the services or performance they provided to their customers.
Campion wrapped up the conversation by emphasizing that without the data hosting and disaster recovery services they had in place, CPR would have been crippled for months:
What that proved to the rest of my agencies, and even the ones who host themselves, was the real need for a true disaster recovery plan, and one that works. It’s one thing to have it on paper in a nice notebook on the shelf, but it’s another thing to make it come to fruition, and to know that it works without any glitches. So some of my agencies who host themselves decided to send a secondary backup to us, just as another element of disaster recovery for themselves.
A contributing writer on IT management and career topics with IT Business Edge since 2009, Don Tennant began his technology journalism career in 1990 in Hong Kong, where he served as editor of the Hong Kong edition of Computerworld. After returning to the U.S. in 2000, he became Editor in Chief of the U.S. edition of Computerworld, and later assumed the editorial directorship of Computerworld and InfoWorld. Don was presented with the 2007 Timothy White Award for Editorial Integrity by American Business Media, and he is a recipient of the Jesse H. Neal National Business Journalism Award for editorial excellence in news coverage. Follow him on Twitter @dontennant.
