CenturyLink testifies on December outages

By Chrissy Suttles

Wyoming Tribune Eagle

Via Wyoming News Exchange

CHEYENNE – CenturyLink representatives testified before the Wyoming Public Service Commission on Tuesday after a multi-state outage disrupted some emergency services in Wyoming late last year.

The commission continues to investigate what caused the outage and how to prevent it from happening again.

Local Verizon customers, Cheyenne Regional Medical Center staff and Laramie County Combined Communications Center dispatchers all experienced lapses in services during the Dec. 27-28, 2018, event.

Some residents couldn’t make calls or access the internet for hours.

Cheyenne Regional Medical Center was unable to make or receive calls from outside numbers, and, while the Laramie County Combined Communications Center was able to receive 911 calls, the system used to identify the phone number and location of incoming callers was out of service. Dispatchers in Laramie and Sweetwater counties both experienced these problems.

The multi-state service interruption prompted a Federal Communications Commission investigation, but Wyoming regulators want more immediate answers.

Ultimately, the outage was caused by a faulty third-party network card in Denver. Technicians were able to replace the hardware once it was identified, but, due to limited network visibility, it took longer than expected to troubleshoot. Leadership wasn’t immediately sure how many customers were affected, which stifled communication.

Cheyenne Mayor Marian Orr was one of the first to criticize CenturyLink’s limited communication with local governments during the outage. She said corrective measures are necessary to better prepare small communities.

“I appreciate the fact that it sounds like measures have been taken as far as how communication will go out to elected officials and hospital personnel,” she said. “Picking up the telephones that were working would have gone a long way as far as confidence and public safety with our consumers.”

Jorge Magana, CenturyLink’s vice president of global operations, said the company has a Network Operation Center to communicate with public safety answering points, but noted customer and municipalities could be better alerted.

“When we have the ability to determine when customers are impacted, we proactively send push notifications,” he said.

Other expected changes include more social media alerts and personalized messaging.

“Rolling” outages made it difficult to communicate, too. Various customers experienced interruptions spontaneously while technicians were working to repair the network.

“One of the things we took out of this is, in our quest to provide information to our customers, we estimated it would take four hours to resolve. About 3 1/2 hours in, we knew it would take much longer.”

To address the hardware failure, Magana said staff made a number of technical changes to how the network operates. During this process, they found another card likely associated with the December outage.

If the company does experience another card failure, those responsible should be able to isolate it quickly by preventing similar network visibility loss.

Magana said these incidents are bad for business, noting there’s a lot of incentive to be reliable.

“Operationally, we feel like our customers’ experience is probably the thing that’s going to make a difference for us as a business in a competitive environment,” he said. “We narrowed down our objectives to three things, and the first one is ‘don’t let it break.’ We were able to remove that network component and take steps to prevent this type of event from ever occurring again.”

The commission will bring the matter back before the public after more investigation. It’s unlikely CenturyLink will incur any penalties, according to Wyoming Public Service Commission Senior Counsel Chris Petrie.

“We’re sorry that this happened,” said Tim Goodwin, associate general counsel at CenturyLink. “It’s our responsibility and our fault. It’s a problem for us when we let down our customers, and that’s what we did.”