Mercy Medical Center in Cedar Rapids, Iowa, often takes part in community disaster drills and tabletop disaster exercises — so much so that its executives used to question whether too much time was spent on that.

Not anymore.

Last June, floodwaters from the Cedar River, which runs through the city's downtown, reached 6 feet deep outside the 350-bed hospital's front doors.

"There are three flood plains in town, a 100-year, a 500-year and a no-man's land way out there. We're in the no-man's land," says CIO Jeff Cash, who had to fly back from vacation in Paris to deal with the flood.

About 5,000 families were forced out of their homes, including 140 hospital employees. Most of those families have not been able to return and it's still not clear whether many downtown buildings will be salvageable, he says.

Maximum flood stage for the river is considered 21 feet, but the river was so swollen, it reached 32 feet. The hospital had about 178 patients at the time and about 1,000 community volunteers working to protect the facility. But the water came fast and strong, and soon the hospital was nearly surrounded.

"We had water coming in three ways," says Cash. "We had water breach the sandbags we had out and make it through the front doors. We had groundwater squirting in through the side walls — the groundwater pressure was that strong. And the sewer system began to back up. We had plugged up the outgoing sewer pipes with rubber bladders, and in some cases the porcelain fixtures cracked and came right off the walls because the water pressure was that strong."

Planned Resiliency

He says the hospital has planned for resiliency in its IT infrastructure and that worked well, but a few problems came to light.

The hospital has redundant data centers on separate floors at opposite ends of the building, making them a city block or so apart. But the facility's power-distribution equipment was on the lower level, which is built like a walkout basement. "... as fast as the water was coming in (through the walls), we were concerned that we wouldn't be able to stop it long-term, so the decision was made to evacuate our patients (while we still had power)," says Cash.

The hospital did lose power from the city, but has redundant generators at a higher elevation behind the hospital. Though it never lost emergency power, no one wanted to take that chance.

So multiple ambulance services, the National Guard and everyone else that could be enlisted came to transport patients to other facilities in the state. Cash says of the process, which lasted eight or nine hours: "It went fairly smoothly considering the chaos it could have been."

That was on Friday, June 13, and the CEO insisted the cancer center had to reopen on Monday to keep those patients on their treatment protocols. Other units were brought back up individually over two weeks.

No water reached the lower-level data center, but all the outpatient services on that floor — registration, lab, radiology, etc., — had to be relocated upstairs. Because everything around it had to be evacuated and rebuilt, however, that data center was moved out, too, and the hospital ran on the upstairs data center alone.

"They're independent of each other. The idea was to have one above ground in case we had water and to have one below ground because we're also kind of in tornado alley," Cash explains.

Split Services

"We've tried to split all our services. We're a Meditech hospital, so we've got two Meditech installations, one in each data center," he explains. Meditech is a suite of 20 or so applications integrated into a single delivery platform of hardware and software. The applications for things like clinical, financial and supplies management can be called up from a single interface.

"Fortunately, we just came off a rebuild of all our network infrastructure and we did that as we rolled out a new VoIP system. We wanted our network infrastructure to be as resilient as we could. So we have dual-honed fiber from all of our nursing floors from all the work areas to both data centers, then we have them cross-connected. So fortunately, we had that all in place and it was all working."

Among the things that worked well: With an in-house repeated cellular system from U.S. Cellular put on emergency power, the hospital probably had as good or better coverage than the town, which lost a couple of towers downtown, Cash says.

The VoIP telephone system allowed employees to unplug their phones and take them to a new location, whether that be upstairs or to any of the hospital system's 21 clinics around the city and state, plug them back in, and still receive calls at their original numbers.

Also the Vocera wireless devices, which employees wear on a lanyard around their necks, allow people to push a button and merely say, for example, "Call Jeff Cash" and do so. Anyone on the network could be found anywhere in the facility, and that was a boon since so many departments — 250,000 to 280,000 square feet of services — had to be relocated.

"We also learned that the VMware installation we'd done recently was a big help for us. VMware lets us virtually host applications in either data center. So instead of even having to move servers upstairs, with VMware, all we had to do was fire up a couple of extra servers, then we could bring applications back up about 10 at a time," he says.

Essential Communications

He says two communications channels were considered essential: the hospital's Web site and Patient Keeper, its physician portal to its electronic medical records system.

"We [switched] our Web site very quickly to an off-site location that we knew would continue to be available even if we lost the hospital any further," Cash says.

The Web site was used heavily by people in the community and also by employees who might have missed one of the twice-daily updates held during the crisis in the hospital cafeteria.

Not having access to patient records would be no big deal for the hospital when it wasn't providing care, but because patients were transferred all over the state, doctor access to those records was vital.

"Patient Keeper is located in Boston, so we transferred a snapshot of our database to them and they reloaded it. It took about an hour and a half, and they brought that EMR back up in their location. So if a physician outside the hospital needed access to a patient record, we were able to provide it," says Cash.

A Few Bugs

"One of the oversights on my part was that both of the data centers have dual feeds from two different sets of switch gears for power. The problem was those feeds came from different switches that were both located in the basement," Cash explains. "Those electrical-distribution switches are already up on cement platforms and we've got pumps and all that sort of stuff, but because water was coming in sideways through the walls, we got concerned. We had those switches covered with plastic to keep that water from hitting them, but we didn't know how long we'd be able to stand that."

He says they eventually decided to run a direct line above ground from the generators to the upstairs data center, just in case, though it turned out to be unnecessary.

And there was another problem. The hospital connects to Qwest's Sonnet Ring through two sides of the hospital, making that connection, too, redundant. Qwest provides the hospital's local, long-distance, Internet and networking services. But the D-mart, which Cash calls "the single point of failure for that system," was still in the basement. He says moving it to one of the data centers had been "one of those things we were planning to do" for about a year, but it hadn't been done because Qwest said it would mean being down for 12 to 18 hours while it was moved, put back up and retested. But with water coming in that room, it was obvious that it had to be moved quickly.

Cash says that about 90 minutes from the time he called Qwest, it was moved and reconnected. Of that lesson, he says, "Don't wait. If you need to move something, get it done."

Since the whole lower floor had to be remodeled, the hospital also redesigned the lower data center. Both data centers now have overhead electrical distribution systems. And they learned that when moving large amounts of equipment to a different site — much of the equipment for ancillary services was not redundant and had to be moved upstairs — the power outlets might not be where you need them. A bus-bar system has been run in two strips down the center of the lower data center providing power, with the proper plug, anywhere the outlet needs to be.

And these days, disaster planning comes with a whole new sense of purpose.

"I think that work helped us," Cash says. "When we kicked off our emergency-operations center, everyone knew where to go, all our equipment and all our communications channels were already there."

See also Hurricanes' Wrath Drives Disaster Planning and When All Else Fails, Make Sure You Have Backup.