A new approach to accessing citizen data

Government departments and agencies hold vast quantities of citizen data spread across hundreds of systems. Gaining a view of a single citizen across those systems is a challenge that many government projects have strived to achieve but always come up short. This is a significant challenge the Australian Government faces as it works to implement its whole-of-economy vision to become a modern data-driven society by 2030. But the strategy will be held back unless the government embraces new and emerging technologies that overcome the problems of legacy approaches to data access and management.

Multiple challenges in data sharing

The challenges come from several angles. There are legacy systems, built with the needs of monolithic departments, that were never made to facilitate information sharing. There are also governance and privacy concerns. Much of the data is highly sensitive and giving access can, potentially, increase the risk of an accidental or intentional data breach. All of this means it is difficult to gain insights, particularly when the data is spread across multiple systems.

Data sharing across government departments is complex because data owners are reluctant to share data due to privacy concerns. There is significant community fear that data will be mishandled or misused and that data management platforms can introduce errors or security problems. A networked data platform allows data owners to retain control without the need to duplicate or give away extracts of data to other parties. Ultimately, a networked data platform enables the data owner to oversee queries on data without it ever leaving their governance.

Once data is shared, under traditional data management and sharing processes, there is no way to ever rescind access to data. The original owner cannot be guaranteed of that data ever being deleted and returned. A networked data platform overcomes this stumbling block and ensures the data owner retains control of where and how data is used at all times.

Another barrier to sharing is the old approach of extracting data from systems, transforming it to fit a specific database design and then loading it into a new database where it could be analysed and used. This approach, called ETL (extract, transform and load), was complex and required significant resources. ETL requires very specific expertise and significant investment in storage solutions. ETL never delivered on its promise — because of the cost and complexity, many datasets were omitted and adding new data resulted in expensive projects that took so long to deliver the desired outcome that the data was often no longer needed and vastly out of date.

New rules and a new approach

A new approach to accessing data for rapid decision-making at a fraction of the cost of traditional data warehousing projects has emerged as a game changer. The first part of that approach starts by asking business experts how the data is to be shared. They can set the rules around access, security and how frequently the data can be accessed.

The second part of the new approach to data access is a networked data platform. This uses metadata to identify data sources and how disparate data from different sources can be joined. Data owners determine how to share data and make it available to authorised parties. Rather than needing to copy data to a centralised data warehouse or data lake, it becomes possible to query data without extracting it.

Rather than extracting, transforming and loading data into a single system, the analytics tool goes to the data, at its source, in real time and creates a coherent view at a speed never achievable before, with far less complexity and cost. With this approach, the time, effort and resources needed to add new data sources is vastly reduced. New data sources can be added in hours or days rather than months or longer. As well as being considerably faster, this approach is far more cost-effective and doesn’t rely on major capital expenditure. For government departments that are increasingly focussed on ensuring costs are managed, this is a major benefit.

For many years, successive governments have been trying to find ways to improve services for citizens across digital platforms. One of the biggest obstacles they have faced is that citizen data is distributed across multiple systems, often duplicated with each department holding its own copy, and difficult to access as legacy systems were not made to facilitate information sharing.

A networked data platform allows those legacy applications to be retained but makes the information they hold accessible. Instead of information being locked in departmental silos, it can be virtually joined in real time so a single view of a citizen becomes possible. It is faster, more cost-effective and easier to manage and maintain. It is a modern secure solution to an old problem that governments have struggled to solve.

Image credit: iStock.com/gremlin