A system configuration mistake caused the outage that affected Windows Azure customers in western Europe last week, according to Microsoft.
As a result, the Microsoft public cloud application hosting and development platform was unavailable for about two and a half hours on Thursday. Microsoft didn’t say how many customers were impacted.
At issue was a “safety valve” mechanism in the Azure network infrastructure designed to prevent cascading network failures. It does so by capping the number of connections that network hardware devices accept.
“Prior to this incident, we added new capacity to the West Europe sub-region in response to increased demand. However, the limit in corresponding devices was not adjusted during the validation process to match this new capacity,” wrote Mike Neil, Windows Azure general manager, in a blog post.
A sudden rise in the affected cluster’s usage led to the “safety valve” threshold being exceeded, which generated a storm of network management alerts. “The increased management traffic in turn triggered bugs in some of the cluster’s hardware devices, causing these to reach 100% CPU utilisation impacting data traffic,” Neil wrote.
At the time, Microsoft solved the problem by increasing the affected cluster’s “safety valve” limits. To prevent the situation from recurring, Microsoft is patching the identified bugs in the networking hardware devices, and it is also improving the network monitoring systems, so that they can identify and address connectivity issues before they cause outages.
