The FCC is asking for public input about what happened during T-Mobile’s June 15 network outage and impacts to public safety and general consumers.
The agency’s Public Safety and Homeland Security Bureau is investigating the outage, which lasted nearly 13 hours and primarily affected Voice over LTE (VoLTE) calling.
T-Mobile has said the problem was triggered by a leased fiber outage from a third-party provider in the Southeast that exploited a routing platform configuration, causing circuit overload. That resulted in “an IP traffic storm that spread from the Southeast” creating substantial issues across the IMS stack in T-Mobile’s core radio network. This effectively stopped phones registering and call completion, so many customers couldn’t make or receive calls, as well as sometimes text messages. T-Mobile did say data services were still up throughout the day.
According to a public notice (PDF), released June 23, the investigation is happening “given the large area affected and the critical importance of dependable and resilient 911 services throughout the United states.” It also makes good on FCC Chairman Ajit Pai’s Tweet at the time, when he said the commission would do so and called the outage “unacceptable.”
The network problems started around noon ET and were fully resolved around 1 a.m. ET.
T-Mobile President of Technology Neville Ray, speaking June 18 at a Wells Fargo investor forum, said about 20% of T-Mobile’s normal call volume was lost during the outage.
The FCC is asking for comments on all relevant information about the causes, effects, and implications of the outage. Specifically, it has asked for input from public safety entities, as well as state and local governments. The FCC cited social media reports showing the network issues affected also affected 911 calling for T-Mobile users. It wants to know if there are estimates of how many calls, including 911, failed or were impacted, if data services were disrupted for public safety and governments, and what Public Safety Answering Points (PSAPs) experienced. The agency also asked if T-Mobile notified those potentially affected PSAPs in a timely manner with actionable information, and if not, how they found out about it.
In addition to public safety, the investigation is gathering information on how the outage affected individual and business consumers including; how did they learn about the outage and if T-Mobile communicated effectively; the services impacted like voice, text and data; whether alternatives like FaceTime, and Zoom were effective; and what effects the outage had on businesses and critical services like hospitals, as well as consumers’ personal activities.
The agency also wants to know whether customers of other carriers were able to complete calls to those on T-Mobile’s network.
At the time of the outage a Verizon spokesperson told Fierce that it was aware another carrier was having network issues and “calls to and from that carrier may receive an error message.”
Interested parties can file comments on or before July 8. As of Thursday afternoon, there were 123 comments filed.
During the investor forum, Ray said 2G and 3G voice services remained up throughout the outage, though acknowledged some congestion. He said there was a minor period in Atlanta where LTE data was lost for about an hour and reiterated that many customers were able to turn to OTT services like FaceTime and Google Voice to complete calls without VoLTE.
T-Mobile had hundreds of engineers working alongside vendors and partner to fix the issue before it was finally resolved. During that time T-Mobile “added a lot of capacity on the fly and very quickly once we understood where the problems really existed,” Ray said, adding that a lot of work has gone on since then to harden the network.
“It’s not good enough from us,” he said, noting there were many of third parties involved. “But it’s our network and it’s our service and it’s what our customers come to us for, so we have to do better.”