TNZ reveals details of 3G network failures

Telecom New Zealand has finally revealed the network faults that have plagued its Alcatel Lucent-supplied XT Mobile 3G network since December.

On a new section on its web site, TNZ said  the first network outage in December was caused by a “corruption of software during a routine operational process.”
 
In turn, this resulted in a “signalling storm” into its southern radio network controller (RNC), leaving customers in the region without a service for “several hours.”
 
The network next went down for up to three days because of a “hardware failure in a router.”
 
“The normal reset then caused a signalling storm that caused an overload on the Southern RNC and impacted all clients on the southern RNC with [a] degraded service for several hours on the Wednesday [Jan. 27],” said TNZ.
 
In that episode, up to 58 cell sites remained out of service for three days to January 29.
 
A further 16 cell sites were down for around half a day in February due to a “process failure” when migrating from southern cell sites to a new RNC in Taranaki and the Hutt Valley.
 
Less than a week later, “the Southern RNC experienced a severe reduction in voice traffic” when up to 81 of the 450 cell sites served by the RNC lost all services.”
 
“For about an hour and 45 minutes all voice traffic was lost south of Taupo,” said TNZ. To fix the problem, the RNC had to be rebooted.
 
The outages have left TNZ in damage control mode, as rivals – Vodafone and twodegrees – swoop in with offers in an attempt poach customers.
 
To fix the network, TNZ says it is improving coverage and capacity.
 
It is also increasing the number of RNCs and their processing power which “will mean that if there is an event that impacts an RNC, the regional impact will be much lower.”
 
More processing power “will also reduce the effect of unusual traffic spikes on the existing network.”