DTA releases report into whole-of-government trial of GenAI

For the first six months of 2024, the Digital Transformation Agency coordinated testing of a generative AI service. It made Microsoft 365 Copilot (formerly Copilot for Microsoft 365) available to over 7600 staff across more than 60 government agencies. The results are now available in a report published last week.

The release of the Evaluation of the whole-of-government trial of Microsoft 365 Copilot outlines the positive reception to the efficiencies and outputs of the AI productivity tools across the Australian Public Service (APS). It also identified areas that still need improvement.

“The DTA has always been clear in its goal to not shy away from innovation,” said Lucy Poole, the DTA’s General Manager of Strategy, Planning and Performance. “To swiftly provide the APS with tools to experiment with generative AI in a safe and responsible way, we identified a tool that would quickly integrate into most workplaces with minimal disruption.”

In the report, the overall use of Copilot over the six-month period is rated “moderate”, with “only a third of trial participants using it daily” and “with its use concentrated in summarising meetings and information and re-writing content”. By contrast, some 46% of users found value a “few times a week” and 21% “a few times a month”.

According to the report, trial participants estimated time savings of up to an hour when summarising information, preparing a first draft of a document and searching for information. The highest efficiency gains were perceived by APS levels 3–6, Executive Level (EL) 1 staff and ICT roles.

The report also highlighted some concerns around integration, data security and information management considerations that agencies must consider prior to Copilot adoption. Training in prompt engineering and use cases tailored to agency needs is also required to build capability and confidence in Copilot.

There are also broader concerns mentioned, on the potential impact of generative AI on APS jobs and skills, particularly on entry-level jobs and women. LLM outputs may also be biased towards western norms and may not appropriately use cultural data and information, the report said.

There are also concerns regarding vendor lock-in and competition, as well as the use of generative AI on the APS’s environmental footprint.

Overall, users initially were mostly “underwhelmed” by the effectiveness of Copilot in their day-to-day work, although positive sentiment increased once additional training was given.

“As we are testing these tools at such an early stage, there are clear opportunities for tailored solutions to be developed that can handle highly technical material,” Poole said. “The evaluation points to the importance of agencies carefully considering detailed and adaptable implementation of these solutions.

“They should consider which generative AI solutions are most appropriate for their overall operating environment and their specific use cases. We’re pleased that a lot of the recent work released by the DTA helps government agencies identify and address these very considerations.”

Image credit: iStock.com/boggy22