Automated decision-making systems: ensuring transparency

Ensuring transparency is essential in government decision-making when using AI and automated systems.

The NSW public sector is currently experiencing a transformative shift in how government decision-making is done. A report released by the NSW Ombudsman in early March, ‘A Map of Automated Decision-Making in the NSW Public Sector’, highlighted this shift.

The report marked the first comprehensive effort in NSW to identify and publish the extent of the public sector’s current and intended use of automated decision-making (ADM) systems. The report’s findings revealed that the use of these systems in state government and local councils was widespread and continuously increasing, with the majority of projects being in the development stages.

The integration of ADM systems is a significant advancement in the operations of the NSW public sector. These technologies rely on data, decision trees, algorithms and sometimes artificial intelligence to automate either all or part of decision-making processes. The adoption of these technologies has the potential to streamline decision-making processes, enhance service quality for NSW citizens and fundamentally transform public sector functions.

However, while these systems have great potential, they also bring unique challenges and risks. The Robodebt scheme is a prime example of what can go wrong with using automated decision-making systems. This system relied on automated data matching between Centrelink records and ATO data without human intervention. It resulted in errors and discrepancies based off an incorrect algorithm. More than half a million Australians were affected by the policy and received incorrect and inaccurate letters saying they owed thousands of dollars of debt.

While this is a lesson public sector organisations need to learn from, it shouldn’t prohibit them from continuing their adoption of these technologies. Instead, there needs to be careful consideration of best practice development and implementation processes.

Below are some key learnings to consider when using AI and ADM systems for decision-making.

Continuous discovery and testing

Success in digital projects is grounded in rigorous and ongoing discovery, testing and legal sign-off at the project’s inception, build and implementation phases. Throughout every stage of an ADM project, assumptions should be identified by the project team and key stakeholders; for example, assumptions that the technology will lead to efficiencies, that the ADM complies with the law and NSW government policies, and that it solves the end users’ pain points. Often these assumptions are not tested at all for validation, or if they are, then this happens only once during the life of the project. As a result, many projects or new technologies end up creating new problems rather than solving existing ones. Challenging and continually testing these assumptions throughout the design and build phase is therefore fundamental.

Striking a balance between automation and human involvement

The adoption of ADM systems in the public sector requires a delicate balance between automation and human involvement. Where an agency is required to consider something to make a decision, the process to the decision can never be fully automated. Agencies must ensure that when humans are involved, they’re not merely approving whatever the automated decision-making system generates. Human involvement in ADM systems requires careful consideration, discovery, planning and testing.

Transparency should be a cornerstone in ADM system deployment

The increasing use of ADM systems brings with it a significant challenge: transparency. The NSW Ombudsman report highlighted that although these systems are growing in use, public visibility into how these systems operate and make decisions is limited. This is an issue for the public sector as a lack of transparency could adversely affect public understanding and accountability. When developing ADM systems, a key feature must always be that the end user should be given a meaningful and understandable explanation about how the decision was made. The goal is to remove the ‘black box’ and give users the ability to review and question the decisions made by the ADM system.

Seeking expert guidance

Developing and implementing new systems in the public sector is complex. This is why seeking expert advice and feedback is crucial for achieving a successful project. The report by the NSW Ombudsman highlighted this, noting that agencies seek limited advice from experts especially in regard to the legality of projects. This is why cultivating cross-functional teams of lawyers, engineers and designers is crucial. For a successful ADM project, there needs to be greater collaboration between all parties to ensure that the project achieves its goals and is legally compliant as well as technically sound.

The road ahead

Moving forward, when developing ADM systems government agencies must conduct ongoing discovery and testing, ensure transparency, include human judgement and seek expert advice. By doing so, agencies will be able to successfully navigate and deliver effective, efficient and fair public services in the digital age. *David Fischl is the Legal Digital Transformation Partner at Hicksons Lawyers.

Top image credit: iStock.com/bagotaj