T-Mobile on Tuesday offered some additional details about the cause of its major network outage earlier this week that left many customers across the country unable to text or use VoLTE calling.
Based on an update from T-Mobile President of Technology Neville Ray, it appears a leased fiber-optic circuit failure set off a chain of events that led to major capacity issues across T-Mobile’s IMS core network, ultimately resulting in the roughly 13-hour outage on June 15.
T-Mobile said the circuit failure was leased fiber from a third-party provider in the Southeast, but didn’t say who. That failure alone didn’t cause the outage, as Ray wrote: “this is something that happens on every mobile network” and redundancy is built-in, but the backups put in place also failed.
“This redundancy failed us and resulted in an overload situation that was then compounded by other factors. This overload resulted in an IP traffic storm that spread from the Southeast to create significant capacity issues across the IMS (IP multimedia Subsystem) core network that supports VoLTE calls,” wrote Ray.
RELATED: T-Mobile resolves 13-hour outage, cites IP traffic issue
T-Mobile has added “permanent additional safeguards to prevent this from happening again.” The operator’s also still working to figure out what caused the initial overload failure.
Ray went on to apologize to T-Mobile customers for the inconvenience and thank them for their patience.
“This is a responsibility my team takes very seriously and is our highest priority,” he wrote. “Yesterday, we didn’t meet our own bar for excellence.”
The issued was fully resolved early morning around 1 a.m. ET on June 16, with hundreds of engineers working alongside vendors and partners to fix the problem.
During the outage, T-Mobile users were still able to access apps and services like FaceTime, iMessage, Google Meet, Zoom and Skype. Many customers could also use circuit-switched voice connections.
RELATED: T-Mobile hit with widespread network issue, customers report service outages
Despite reports of issues with Sprint service, Ray said customers on the Sprint network weren’t impacted. AT&T and Verizon’s networks were also functioning normally, though calls to and from T-Mobile might have resulted in error messages for some.
The situation combined with misinformation sparked erroneous claims that went viral on Twitter of a supposed massive DDoS attack impacting multiple carriers and services like Facebook and Instagram.
Notably, in an accurate Tweet that didn’t go viral, FCC Chairman Ajit Pai called the T-Mobile outage “unacceptable” Monday night and said the agency is launching an investigation.