A rogue piece of software, which triggered a cascading failure through Google's data centres around the world, led to the shutdown of its Gmail system last week, according to the FT .
The failure happened during routine maintenance work at a European data centre when a piece of code, written in-house, triggered a chain reaction. The code was designed to direct data to the data centre closest to users to accelerate response times.
However, when data were moved to a back-up centre to carry out maintenance, the code unexpectedly started working on new data, overloading the back-up data centre, which automatically pushed data to a third centre, which led to an overload, which then meant a fourth data centre was involved, and so on.
The FT quoted Matt Cain of Gartner saying, "That's just not acceptable. It was poor thinking-through of a code change. In a corporate environment, you can't just tell your CEO it was bad luck."
However, he added that the overall reliability of Gmail was still superior to the in-house e-mail systems run by most companies.