How good disaster planning helped Houston companies keep IT systems running
Leading Houston-based corporations and data center companies continue to keep their IT systems running as rain and flooding from hurricane Harvey inundate broad sections of the city.
They’ve been expecting this disaster for some time.
Since Houston sits on an historical hurricane highway from the Gulf of Mexico, area CIOs have planned for years to face just this kind of weather-based disruption. The I.T. leaders at Halliburton, Sysco, Waste Management and other companies have moved applications to the cloud, enabled employees to work remotely and put in place further contingency preparations that are now helping them weather the storm.
“Rain and wind just keep on going from Friday afternoon and they’re still hammering us,” says Gautam Roy, VP of information technology for Waste Management. Streets have flooded, cutting off whole neighborhoods.
But Roy is sitting safely in his house in Katy, Texas, about 40 miles away from downtown Houston. Waste Management has shut down its headquarters and Roy and his team are keeping the company’s data systems running remotely.
By plan, Waste Management’s primary data center sits in Austin, 145 miles away from the devastation. “We don’t wait for an event like this,” says Roy. “We go through yearly tests for our critical apps.”
Most of those applications don’t live on any map. Meetings are held on Skype and WM personnel use Office 365 driven by Microsoft cloud services. “This has helped us stay mobile and be at home and be able to conduct business,” Roy explains.
Sysco Systems also moved core IT systems to the cloud in anticipation of the storm, says a company spokeswoman.
“With the exception that most associates are working remotely during this unusual, catastrophic event, business technology at Sysco is executing close to business as usual,” she says.
Oil services giant Halliburton keeps its critical applications at data centers far from Houston as a matter of course. It has facilities in the United Kingdom and the Middle East, as well as two in the U.S. that are not in Texas.
“We have a follow-the-sun operation,” says CIO Ken Braud. If there’s a breakdown in one place, other data centers fill in the computing gap.
“We design systems with disaster recovery in mind,” says Braud, with redundant circuits and power generation. “We think we’ve done a pretty good job of designing a scalable and robust infrastructure.”
Like Roy, Braud conducts a preseason table-top exercise to play out hurricane scenarios. “We’ve essentially been through it already,” he says, and that makes a real emergency like Harvey run that much more smoothly.
Roy follows a testing roadmap to make sure if he shuts down one data center it will “fail over” to the other without a hitch. “We make sure as we add servers that they are replicated [at the other facility],” he says. The key is to make sure there is enough computing capacity at the beginning all around. Otherwise a technical hiccup can become a bad case of indigestion.
Local colocation company Data Foundry has been lucky and both of its facilities sit on higher ground in Houston. But that didn’t preclude being ready for the worst. “In the week leading up to the storm we were monitoring it and we invoked our emergency management plan,” says CTO Edward Henigin.
That required assembling “ride-through” teams that would stay on site throughout the emergency. The groups show up with a week’s worth of clothing and personal items, including video games, to camp out at the data centers. Data Foundry stocks food. “We treat it as if it’s a self- sustaining operation and assume a worst-case scenario,” says Henigin.
Fibertown, another Houston data center company, is similarly up and running. “We’re in excellent shape with power and cooling stable, and the roads [where they are] are still passable,” said Sam White director of marketing for the company.
Fibertown has a colocation center in downtown Houston and a second facility in the town of Bryan, about 90-miles away. The municipality is out of the way of historical storm tracks and sits at an altitude of 362 feet. So far, the town has received “only” 13- to 20-inches of rain during the storm. Still, that’s a dramatic difference from the 30- to 50-inches of precipitation pounding Houston.
The company has several customers that co-locate from Houston. “They have everything they need to operate out of here for extended time,” says White.
Most important: There are 35 restaurants and bars within four blocks.