Tom Raftery’s Social Media

Rackspace are a high-profile data centre in the US. They had a couple of outages in the last few days which have badly damaged their reputation. The main outage, according to Rackspace, happened when:

at approximately 6:30 PM CST Monday, a vehicle struck and brought down the transformer feeding power to the DFW data center. It immediately disrupted power to the entire data center and our emergency generators kicked in and operated as intended. When we transferred power to our secondary utility power system, the data center’s chilling units were cycled back up. At this time, however, the utility provider shut down power in order to allow emergency rescue teams safe access to the accident victim. This repeated cycling of the chillers resulted in increasing temperatures within the data center. As a precautionary measure we decided to take some customers’ servers offline.

When I read this, something about it didn’t seem right. I couldn’t understand why the chillers (the machines which cool the water for the aircon) would need to be power cycled. Then, an explanation showed up on the Texas Startup blog:

It turns out that in most multi-tenant commercial property in the United States, the building owner provides chilled water so that tenants can run their HVAC systems. In general, most buildings do NOT put these chillers on power with generator backup

If this is true, it is frightening!

In other words, if power fails to these buildings, the diesel generator will ensure that their aircon will be circulating air which is rapidly increasing in temperature because the water is no longer being chilled.

I’m director of Cork Internet eXchange, the first professional data centre in Ireland outside of Dublin and I can absolutely guarantee that our chillers are on power with a backup from our diesel generator. Of course they are. Why would you design it any other way?

Om Malik put it well when he said:

our Internet infrastructure, despite all the talk, is as fragile as a fine porcelain cup on the roof of a car zipping across a pot-holed goat track