Signaling spike took down Telenor's network

Telenor today revealed its largest service outage in almost 20 years was caused by a surge in traffic following a network reboot.
 
The firm’s Norwegian mobile network collapsed on June 10 because servers were unable to cope with a spike in signaling traffic after it restarted the network following a software upgrade to a central server. In a report to Norwegian telecoms officials, Chief Ragnar Kårhus says the outage is the largest to affect the operator since it launched its mobile network in 1993.
 
Voice and SMS services were affected by the outage, caused when traffic between servers increased “far beyond normal levels,” a company statement reveals. Signaling traffic continued to grow to the point that servers were unable to connect calls and deliver messages. Telenor traced the fault to a single voice server, however it decided to systematically shut down and restart all its servers.
 
“It is not normal that a restart creates such heavy signaling traffic between network servers,” Kårhus notes, adding that the scale of the problem made it “difficult and time consuming to localize the cause of the breakdown and have it corrected.”
 
The firm has now increased network capacity and “established protective mechanisms that come into force if abnormal increases in signaling traffic occur,” Kårhus explains. It has already pledged to refund customers affected by the outage for the period June 10 to 13.