Rackspace apologizes for cloud outage, prepares to issue service credits

Rackspace found itself apologizing to its users again this week and promising to earn back customers' trust after a power outage in its Dallas-Fort Worth data center.

Several thousand customers were affected by Tuesday's power outage, which caused downtime of up to a few hours for customers using Rackspace's Cloud Sites, a Web site hosting service, and Slicehost, which provides hosted virtual servers.

Rackspace previously suffered outages in June and July. This week's outage occurred during maintenance work that was meant to solve the problems that caused the previous outages.

“This summer our DFW facility had power issues, and as a result, we invested significant resources to improve all aspects of our power systems,” Rackspace cloud general manager Emil Sayegh wrote in a letter to customers. “[On Tuesday], during one of these steps, we encountered issues and had a brief loss in power. The power disruption was approximately 5 minutes in duration. Despite this short power disruption, many customers experienced downtime that was significantly longer. Since the power disruption hit the core of many of our cloud services, recovery of full operations required more effort than simple recovery of power. The experience you had last night is not acceptable to us.”

After the summer outages, Rackspace said in a filing with the Securities and Exchange Commission that it would issue service credits totaling $2.5 million to $3.5 million. This week, a Rackspace spokesman said the company will honor all service-level agreements but did not say how much money will be paid out to affected customers.

In the letter to customers, Rackspace said multiple power distribution units lost power during preventative maintenance on a UPS cluster early Tuesday morning.

“Although the power outage was very brief (5 minutes), it forced a hard re-boot to occur on a portion of our cloud infrastructure,” Rackspace said. “As our engineers worked to bring hardware back online, we experienced several unforeseen hardware failures. Further complicating our recovery effort, the incident also created internal DNS issues, which caused additional delays. With that said, the vast majority of cloud customers affected by this outage had service restored within one hour's time (many in as little as five minutes); however, depending upon the service, a few customers experienced service interruptions for up to few hours.”

Going forward, Rackspace said it is reviewing its policies for maintenance notifications, and reviewing procedures and systems to ensure quick resumption of service in case of events like the one this week.

“We have invested massively in the DFW facility to ensure it delivers at a level you expect from Rackspace,” the company said. “Despite last night, we feel very good about our plan and have high confidence in the DFW facility – clearly we have to prove it.”