T-Mobile network outage triggered by fiber circuit failure

T-Mobile said it's added permanent additional safeguards to prevent the network issue from happening again. (Getty Images)

T-Mobile on Tuesday offered some additional details about the cause of its major network outage earlier this week that left many customers across the country unable to text or use VoLTE calling.

Based on an update from T-Mobile President of Technology Neville Ray, it appears a leased fiber-optic circuit failure set off a chain of events that led to major capacity issues across T-Mobile’s IMS core network, ultimately resulting in the roughly 13-hour outage on June 15.

T-Mobile said the circuit failure was leased fiber from a third-party provider in the Southeast, but didn’t say who. That failure alone didn’t cause the outage, as Ray wrote: “this is something that happens on every mobile network” and redundancy is built-in, but the backups put in place also failed.

Sponsored by Nokia

Webinar: Capture customer value with agile operations in 5G

This webinar looks at how CSPs can gain the agility to port use cases and services from one customer or segment to another, quickly and seamlessly. In the webinar we will examine what agility means for concrete services and use cases, how we can quickly adapt a service for multiple different industries, how catalog based service orchestration and DevOps priniciples bring agility to operations and more.

“This redundancy failed us and resulted in an overload situation that was then compounded by other factors. This overload resulted in an IP traffic storm that spread from the Southeast to create significant capacity issues across the IMS (IP multimedia Subsystem) core network that supports VoLTE calls,” wrote Ray.  

RELATED: T-Mobile resolves 13-hour outage, cites IP traffic issue

T-Mobile has added “permanent additional safeguards to prevent this from happening again.” The operator’s also still working to figure out what caused the initial overload failure.  

Ray went on to apologize to T-Mobile customers for the inconvenience and thank them for their patience.

“This is a responsibility my team takes very seriously and is our highest priority,” he wrote. “Yesterday, we didn’t meet our own bar for excellence.”

The issued was fully resolved early morning around 1 a.m. ET on June 16, with hundreds of engineers working alongside vendors and partners to fix the problem.

During the outage, T-Mobile users were still able to access apps and services like FaceTime, iMessage, Google Meet, Zoom and Skype. Many customers could also use circuit-switched voice connections.

RELATED: T-Mobile hit with widespread network issue, customers report service outages

Despite reports of issues with Sprint service, Ray said customers on the Sprint network weren’t impacted.  AT&T and Verizon’s networks were also functioning normally, though calls to and from T-Mobile might have resulted in error messages for some.

The situation combined with misinformation sparked erroneous claims that went viral on Twitter of a supposed massive DDoS attack impacting multiple carriers and services like Facebook and Instagram.

Notably, in an accurate Tweet that didn’t go viral, FCC Chairman Ajit Pai called the T-Mobile outage “unacceptable” Monday night and said the agency is launching an investigation.

Suggested Articles

Vodafone U.K.’s head of networks said if the government decides to strictly ban Huawei it would cost Vodafone “low-single-figure billions."

A new NTIA report determined 100 MHz in the 3.1-3.55 GHz band looks most viable for near-term sharing with commercial wireless services.

Instead of running on virtualized machines, Verizon is changing its underlying software architecture to run using cloud-native container-based tech.