ATO details causes of outages

A mixture of hardware and software failures was to blame for the high-profile outage of the ATO’s online systems in December and February, according to Tax Commissioner Chris Jordan.

In a statement to Budget Estimates today, Jordan said the ATO will next week publish a comprehensive report into the outage, which pins the blame for the outage on a failure of HPE’s turnkey data storage services for its 3PAR storage area network (SAN).

Analysis of the failure found that fibre cables feeding the SAN were “not optimally fitted”, disk drives on the network had software bugs that made the stored data inaccessible or unable to be read, and monitoring features including a tool to report operating errors had not been activated.

“The SAN design and configuration meant we had an overemphasis on performance features rather than stability or resilience — a relatively small disk drive failure had a large impact — only 12 of some 800 disk drives failed, but they impacted most ATO systems,” Jordan said.

“The recovery was slower because some of the recovery tools required were stored on the same SAN that failed.”

Jordan said the ATO has reached a confidential commercial settlement with HPE that recoups key costs associated with the outage and provides the agency with extra and higher-grade IT equipment.

The report will include an interim root cause analysis from HPE as well as expert advice from consultancy PwC, as well as the results of the ATO’s internal review and feedback from stakeholders.

Jordan said the ATO’s analysis found that the agency’s business continuity mechanisms, communication and engagement worked well, “but into the future need to be more inclusive of our partners such as software providers, tax professionals, and the superannuation industry”.

Follow us on Twitter and Facebook