Qld Department of Environment adopts Azure Data Lake

Queensland’s Department of Environment and Science has rolled out a cloud-based data and analytics platform to manage and monitor the ecological recovery of the East Trinity Reserve.

The department is restoring the East Trinity Reserve wetlands from the environmental damage caused by the area being used to grow sugar cane between the 1970s and the late 1990s.

Since the remediation project began in 2001 the department has been collecting data every 10 minutes from sensors located in 15 stations, but the data has been siloed and difficult to analyse.

The department’s scientists have traditionally relied on manual processes to manage sensor data cleansing, error analysis, storage and version management, resulting in a large backlog of unvalidated data.

To tackle this challenge, the department selected Microsoft partner Versor to help transfer the data into an Azure Data Lake, where it is cleansed using Azure Databricks, turned into an SQL database and made available for analysis.

The platform also allows for opportunities to augment data collections with external information such as weather forecasts, and for easier data-sharing with external parties with an interest in the information, such as Indigenous-owned ecotourism businesses.

According to Daniel Brough, Science Leader for Science Information Services at the department, the project will serve as a proof of concept for future efforts to improve the organisation’s data governance.

“I think moving to some of these new systems, having things like Data Lake, Azure Databricks, and some of those other things where we can actually do more as code is really improving the way we manage the data so that it’s repeatable,” he said.

Department Business Analyst Jennifer Richards said the strategy will be extended to support the entire science division.

“The science division is a division built on data. That’s what we do. So at the moment we have all these roadblocks in making that data reusable. The ultimate aim is to get data out onto open data and in use outside of the department as well as inside of the department.”

Image credit: stock.adobe.com/au/flashmovie