Politics
The 2022 Rogers outage that left 12 million people without wireless and hard-wired services was caused by human error and made worse by management and system "deficiencies," according to an independent review conducted for Canada's telecommunications regulator.
The Rogers outage lasted 26 hours, cut 12M people off from 911 and other network services
Peter Zimonjic · CBC News
·
The 2022 Rogers outage that left 12 million people without wireless and hard-wired services was caused by human error and made worse by management and system "deficiencies," says an independent review conducted for Canada's telecommunications regulator.
The review report also sayssteps taken by Rogers since the outage are "satisfactory to improve the Rogers network resiliency and reliability, as well as to address the root cause of the July 2022 outage."
The 26-hour outage started early in the early morningof July 8 and left individualsand businesses without access to their mobile, home phone, internet and 911 services.
The Canadian Radio-television and Telecommunications Commission (CRTC) commissioned Xona Partners in September 2023 to undertake the review and determine what caused the outage.
The engineering consultancy was also tasked with looking at whether the measures taken by Rogers since the outage are sufficient to prevent another incident.
Full report to be released, with redactions
Xona Partners' findings were contained in the executive summary of the review report, released this month. The CRTC says the full report contains sensitive information and will be released in redacted form at a later, unspecified, date.
The report summary says that in the weeks leading up to the outage, Rogers was undergoing a seven-phase process to upgrade its network. The outage occurred during the sixth's phase of the upgrade.
- Remember the Rogers outage? It's added 221,000 new phone customers since then
"The July 2022 outage is attributed to an error in configuring the distribution routers within the Rogers IP network," the report says.
Staff at Rogers caused the shutdown, the report says, by removing a control filter that directed information to its appropriate destination.
Without the filter in place, a flood of information was sent into Rogers' core network, overloading and crashing the system within minutes of the control filter being removed.
Algorithm designated network upgrade as 'low' risk
The report says Rogers' core network manages wireless and hard-wired data both internally, within the company, and externally, for outside customers and service providers.
"With both the wireless and wireline networks sharing a common IP core network, the scope of the outage was extreme in that it resulted in a catastrophic loss of all services," the report says.
- Rogers CEO announces new protocols, investment in wake of major service outage
Having wireless and wireline services share the same network is a practice "common to many service providers," the report says, adding thatcompanies find it an efficient way to "balance cost with performance."
Rogers has since announced that it will developa new, separate, network for its wireless systems while keeping hard-wired services on the old core network. The report says that work is ongoing.
The review says that because the first five stages of the network update took place without incident, "the risk assessment algorithm downgraded the risk level for the sixth phase" of the upgrade.
How Rogers is addressing the issue
Designating risks in phase six as "low" meant Rogers' staff could avoid additional levels of scrutiny and approvals as the upgrade proceeded, even though doing so "contravenes industry norms," the report says.
Rogers says it has since installed a new risk assessment algorithm to address the issue.
WATCH | Report looks into 2022 Rogers outage: 12 days ago Duration 2:08Human error triggered massive 2022 Rogers service outage, report finds
The executive summary of Xona Partners' review also says the "network failure could have been prevented" if Rogers had "overload protection mechanisms" limiting how much information is funnelled into the core network.
The review recommends that all telecom networks in Canada implement overload protection mechanisms for their core networks.
Challenges restoring the network
A central issue frustrating Rogers' efforts to get its systems back up once they went down was the corporation's inability to communicate properly.
The report saysthat when the core network went down, remote employees were unable to access Rogers' systems or use the internet and could not get online by using other service providers.
- CRTC hires private company to investigate 2022 Rogers outage
"Rogers had to dispatch staff to remote sites to physically access the affected routers, which delayed network recovery efforts," the report says.
All incident response and crisis team members at Rogers have since been provided with backup, third-partyaccess to the internet to "maintain communication capabilities during outages."
The review also saysthat Rogers staff could not access critical error logs detailing the root cause of the outage until 14 hours after the outage began, which "adversely impacted outage recovery efforts."
More transparency needed, says critic
John Lawford, executive director of the Public Interest Advocacy Centre in Ottawa, has been pushing Rogers and the CRTC for more transparency onthe outage.
He criticizedthe CRTCfor taking two years to deliver a report on the outage, describing itas a "whitewash in the sense of both the CRTC and Rogers being very much let off the hook."
"The report makes a claim that Rogers has rectified the issue and there is insufficient evidence for me there to see that," Lawford said. "This is just one particular expert's viewpoint."
Rogers declined CBC News' requestfor an interview.
In a statement, a Rogersspokesperson said the company will "remain focused on delivering the most reliable network experience so Canadians can connect when and where they want."
Government claims network investments underway
The spokesperson said, citingan August 2023 report from analyticsfirm umault,Rogershasthe most reliable wireless network in Canada.
To come to that conclusion,umault ran a series of tests over a 47-day period last year and found Rogers was the most reliable wireless provider in Canadaduring the time surveyed.
The spokesperson also pointed to another recent study from Opensignal, anetwork testing company, that found Rogers hasthe most reliable internetservice in Canada.Opensignal collected its data on Rogers and other providersbetween February 1 and April 30 of this year.
"We completed a full review of our networks, strengthened our network resiliency, and implemented all the recommendations of this report. We will continue to invest to ensure Canadians enjoy the best networks in the world," theRogersspokesperson said.
- Former industry minister Navdeep Bains gets Rogers job, sparking criticism
In a letter to Rogers, the CRTC said the company had "confirmed the implementation of all measures" recommended byXona Partners.
A spokesperson for Minister of Innovation, Science and IndustryFrançois-Philippe Champagnetold CBC News that Rogers has addressed all recommendations in the report and is continuing to invest in network resiliency.
Corrections
A previous version of this story said the Rogers outage started in the early evening of July 8 and lasted 15 hours. In fact, the Xona report found that the outage started in the early morning of that day and lasted more than a full day.
Jul 05, 2024 6:06 PM ET
ABOUT THE AUTHOR
Peter Zimonjic
Senior writer
Peter Zimonjic is a senior writer for CBC News. He has worked as a reporter and columnist in London, England, for the Daily Mail, Sunday Times and Daily Telegraph and in Canada for Sun Media and the Ottawa Citizen. He is the author of Into The Darkness: An Account of 7/7, published by Random House.
CBC's Journalistic Standards and Practices|
About CBC NewsCorrections and clarifications|Submit a news tip|
Related Stories
- Rogers CEO announces new protocols, investment in wake of major service outage
- Service restored to most Rogers customers hit by outage Wednesday
- CRTC hires private company to investigate 2022 Rogers outage