Here’s what caused yesterday’s O2 and SoftBank outages

It appears that most mobile carriers, including O2 and SoftBank, have recovered from yesterday’s cell phone network outage that was triggered by a shutdown of Ericsson equipment running on their networks. That shutdown appears to have been triggered by expired software certificates on the equipment itself. While Ericsson acknowledged in their press release yesterday that expired […]

It appears that most mobile carriers, including O2 and SoftBank, have recovered from yesterday’s cell phone network outage that was triggered by a shutdown of Ericsson equipment running on their networks. That shutdown appears to have been triggered by expired software certificates on the equipment itself.

While Ericsson acknowledged in their press release yesterday that expired certificates were at the root of the problem, you may be wondering why this would cause a shutdown. It turns out that it’s likely due to a fail-safe system in place, says Tim Callan, senior fellow at Sectigo (formerly Comodo CA), a U.S. certificate-issuing authority. Callan has 15 years of experience in the industry.

He indicated that while he didn’t have specific information on this outage, it would be consistent with industry best practices to shut down the system when encountering expired certificates “We don’t have specific visibility into the Ericsson systems in question, but a typical application would require valid certificates to be in place in order to keep operating. That is to protect against breach by some kind of agent that is maliciously inserted into the network,” Callan told TechCrunch.

In fact, Callan said that in 2009 a breach at Heartland Payments was directly related to such a problem. “2009’s massive data breach of Heartland Payment Systems occurred because the network in question did NOT have such a requirement. Today it’s common practice to use certificates to avoid that same vulnerability,” he explained.

Ericsson would not get into specifics about what caused the problem.”Ericsson takes full responsibility for this technical failure. The problem has been identified and resolved. After a complete analysis Ericsson will take measures to prevent such a failure from happening again.”

Among those affected yesterday were millions of O2 customers in Great Britain and SoftBank customers in Japan. SoftBank issued an apology in the form of a press release on the company website. “We deeply apologize to our customers for all inconveniences it caused. We will strive to take all measures to prevent the same network outage.”

As for O2, they also apologized this morning after restoring service, tweeting:

Ericsson software problem has been causing widespread cell phone outages

A problem with the software in Ericsson equipment is causing outages across the world, including O2 users in Great Britain and SoftBank users in Japan, according to a report in the Financial Times earlier today. Ericsson took blame for the outage in a press release. It apparently involves faulty software on certain Ericsson equipment used […]

A problem with the software in Ericsson equipment is causing outages across the world, including O2 users in Great Britain and SoftBank users in Japan, according to a report in the Financial Times earlier today.

Ericsson took blame for the outage in a press release. It apparently involves faulty software on certain Ericsson equipment used on the affected company’s mobile networks. While Ericsson indicated it involved multiple countries, it appeared to try to minimize the impact by stating it involved “network disturbances for a limited number of customers.” The FT report indicated that it was actually affecting millions of mobile customers worldwide.

Regardless, the company said that an initial analysis attributed the problem to an expired software certificate on the affected equipment. Börje Ekholm, Ericsson president and CEO, said they were working to restore the service as soon as possible, which probably isn’t soon enough for people who don’t have a working cell phone at the moment.

“The faulty software that has caused these issues is being decommissioned and we apologize not only to our customers but also to their customers. We work hard to ensure that our customers can limit the impact and restore their services as soon as possible,” Ekholm said in a statement.

While the press release went on to say they are working to restore the service throughout the day, as of publishing this article, the O2 outage maps still showed problems in the London area and throughout Great Britain.

The AT&T and Verizon outage pages are also currently showing outages in the U.S. We reached out to Ericsson by phone and email to confirm if this was part of their software problems, but had not heard back by the time we published. If we do, we will update this story.

(Note that Verizon owns this publication.)