The boss of DTAC, Thailand’s number two telco, has apologized after the firm experienced its third massive network outage in as many weeks.
Chief executive Jon Eddy Abdullah issued the apology at a press conference called to convince the public that the outages were all down to improbable odds rather than mismanagement or incompetence. The event was organized amid falling public confidence and the prospect of an enquiry by Thailand’s regulator.
The trouble started when DTAC’s entire network collapsed on 21 December because of a technical issue with the migration of the HLR from one database to another. The migration was part of a two-year network modernization plan signed off with Ericsson to replace the 18-year old 2G radio and backhaul network with a modern one ready for 3G or LTE.
Abdullah said his technical people and Ericsson’s are now confident that they understand the root cause of the outage.
On 5 January, a switch in the South of Thailand failed, but what usually takes five minutes to restart took hours because of the HLR migration. The problem spread nationwide when a parameter change aimed at helping users come back online faster inadvertently knocked iPhone 4 users off the network.
Abdullah has ordered a doubling of HLR database capacity before further migration takes place, which is expected to take six weeks.
On 8 January in the latest major network crash, a car accident severed one of two redundant fiber cables to the South and two minutes later, a bush fire cut the other cable. DTAC plans to build triple redundancy into critical network paths from now on.
Abdullah said that the chances of this happening were down to unbelievably bad luck and ruled out sabotage. “There are easier ways to cut a fiber optic cable,” he pointed out.
DTAC has already paid out 300 million baht (€7.4 million) in compensation in the first outage, 50 million baht in the second, and is still calculating the level of compensation for the third.
But the real damage is to the company’s reputation and it remains to be seen how the churn and number portability figures tally up when the dust settles.
DTAC has committed 40 billion baht in network investment over the next three years. Abdullah was asked why DTAC was investing so much given that its concession ends in 2018 and the previous, aborted, draft 3G license specifically prohibited re-use of old 2G network infrastructure. He responded that only 20% of its users were now on 3G (850MHz) and the company and its concession holder, state owned CAT Telecom, has a commitment to bring the best technology to all its subscribers, whether 2G or 3G.
Pressed if the network upgrade was a bid to pre-empt and shape the new regulator’s 3G licensing terms and conditions, Abdullah only said, “It is the right thing to do. Whether we have six years left or two years left, it is the right thing to do.”
Abdullah said he expects 2.1GHz 3G licensing in Thailand in 3Q this year, and that the company has ample free cash, generating around 5 billion to 7 billion bahtevery quarter, to invest in a license and network.
Later, after meeting with DTAC officials, National Broadcasting and Telecommunications Commission vice-chairman Settapong Malisuwan announced a formal investigation and requested access to log files to see if the network crash was due to negligence or a technical issue as claimed. No immediate sanctions will be levied, though further migration is prohibited pending the outcome of the investigation.
Commissioner Prawit Leesathapornwongsa said that while DTAC had effectively offered 48 hours compensation for what was a four to six hour outage, consumers could still sue the operator in civil courts if they believed their consequential damage was higher, and offered help in doing so.
More worrying was his statement that in the future, any database migration would have to be submitted to the NBTC six months in advance for approval.