Triple Zero Senate Inquiry, opening statement from Vicki Brady

My opening statement is detailed because we have not yet had the opportunity to set out the full sequence of events surrounding last week’s mobile network outage. 

Our investigation is continuing, and there will be more to learn, but I want to be clear with the Committee and with Australians about three things, as we understand them today: what happened, how we responded, and what we are doing to reduce the risk of this happening again.

We have commissioned an external expert investigation to test our findings, and I commit to providing the final findings of this investigation to the Committee when it is complete.

Last week, Telstra let Australians down. We let our customers down, we let the community down, and we fell short of what people rightly expect from us. For this I am deeply sorry. 

Australians rely on Telstra every day – to run businesses, stay connected to family, access essential services and, most importantly, to get help in an emergency. When our network fails, the impact is real. 

I particularly want to acknowledge the concern caused by impacts to some Triple Zero calls. Triple Zero sits at the heart of public trust in Australia’s communications system. Telstra has a critical role in that system, and we take that responsibility extremely seriously. No one should be left wondering whether a call for emergency help will get through.

What happened to our mobile network on 8 July?

Let me start with what happened.  

At 3:38am last Wednesday, Telstra staff completed in-person maintenance on a device in one of our network sites called a Network Time Protocol server, as part of work to address a backup power fault. Put simply, this server helps different parts of the mobile network stay in time with each other.

The team followed the relevant procedure, including restarting the server. But because of an underlying software configuration, the server restarted with the wrong date.

That mattered because parts of the mobile network rely on accurate timing to work properly. As the incorrect date spread through the network, some mobile voice and data services began to fail.

We first detected an issue at 4:20am and posted an update to our website at 4:38am to confirm we were looking into it. At that stage, the impact was limited. We saw a small increase in call failures and our team began investigating immediately.  

We communicated more broadly while the issue was still developing and before its full scale was known. We responded to our first media enquiry at 6:33am and published a statement on social media at 6:38am to let our customers know that we were investigating an issue.

At that point, the impact had not met the threshold of a major outage. As more people woke up and network activity increased through the morning, the timing issue reached more parts of the network, and the customer impact grew. Our early updates reflected what we knew at the time, not the greater impact that emerged later.

Our teams completed test calls to Triple Zero and these were successful. 

Once the cause was understood, our teams isolated the affected device at 7:11am and progressively restored the network.  

We activated the Triple Zero Disruption Protocol Partner Bridge at 7:20am to advise the Emergency Services organisations and key stakeholders that we were investigating the issue, with 17 further bridges held during the outage.

By 10:00am, most calls and data services were working correctly, and by 4:00pm the initial issue we identified had been addressed. 

At its peak this initial issue impacted approximately 45% of all calls and data sessions on our mobile network.

What was the impact to Triple Zero calls?  

Late Wednesday afternoon, further investigation identified a subsequent issue affecting some calls to Triple Zero. In these cases, callers received an error message, and the phone may have attempted to connect to an alternative mobile network. This issue was linked to the same software configuration that caused the original outage but required a separate fix.

Over the course of the outage, 58,835 calls to Triple Zero successfully connected and 604 experienced the error. 

As part of our operational procedures, a welfare check was initiated automatically when a Triple Zero call was identified as unsuccessful. We completed 604 welfare checks.

We worked closely with Emergency Services nationally and are not aware of any life-threatening outcomes as a result of this outage. Minister Wells confirmed at her press conference at 1:00pm on Friday 10 July, that Police had completed all 144 follow-ups and no adverse outcomes were reported. We have also confirmed this with Emergency Services.  

At 10:38am on Thursday 9 July, we implemented a solution that addressed the issue affecting some calls to Triple Zero. Since this solution was put in place, no further welfare checks have been required for this issue.

Camp-on operated as expected during the outage, with about 3,200 more Triple Zero calls from Telstra customers carried over the TPG and Optus networks than average.

How we kept customers, the public and key stakeholders informed

From the moment this outage began early Wednesday morning, our team prioritised fixing the issue and keeping Australians and key stakeholders informed. This included communicating with Minister Wells’ office at 6:44am and the Triple Zero Custodian andthe Communications Department at 7:14am.   

We conducted four media conferences and published numerous updates to our website, app and social media.

What caused the outage?

We now understand the likely root cause of the outage, though our investigation and external expert investigation will test this fully.

The team carrying out the maintenance work followed the correct procedures. However, when the device was re-powered, a GPS card within it did not operate as expected. 

We now believe this occurred because of an intentional design change that had previously been made to the equipment to fix an earlier fault. That design change had not been properly documented, which meant the maintenance team was not aware of the way the device would behave when restarted.

A software update had also not been applied to the GPS card. Had that software update been completed or had the design change been properly documented and reflected in the maintenance procedure, the outage may not have occurred.

That is clearly unacceptable. If maintenance work can trigger this kind of outage, it suggests our controls were not good enough. We are accountable for that, and our external expert investigation will address why that design change was not documented, why the software update was not completed, and what needs to change in our controls so known risks are captured, prioritised and closed before they can affect customers. 

Modern mobile networks are complex, but complexity does not reduce our commitment to prevent failures, detect them quickly and ensure the right safeguards are in place. 

We invest significantly in the resilience of our network and in modernising it as technology evolves, but this incident shows we have more work to do. 

I want to be clear with the Committee and with Australians: we will be open about what we find through our investigation, cooperate fully with this inquiry and with regulators, and take the actions required to strengthen the resilience of our network and the services Australians rely on.