1901: Inspiration to deliver the future data strategy

As the Australian Government seeks to deliver its commitment to deliver connected public services through world-class data and digital capabilities, it would do well to look back in time to the birth of our current system of government for inspiration.

As every 12-year-old child will tell you, it was in 1901 that six British colonies — New South Wales, Victoria, Queensland, South Australia, Western Australia and Tasmania — united to form the Commonwealth of Australia, in the process known as Federation. Federation allowed states autonomy over their own unique interests, while facilitating coordination in the national interest, and allowed the country to make huge advances in vital strategic policies such as trade, the economy and defence.

In order to achieve the very 21st century goal of creating a national ecosystem of data that is secure, accessible and reliable, it is crucial to create a data architecture that simultaneously enables both systemic coordination and local autonomy. This principle applies equally to the foundational requirements of data security and privacy, enabling the government to achieve its aim to “deliver simple, secure and connected public services through world-class data and digital capabilities”.

Without a holistic approach, successes in making data available can’t necessarily be replicated, and disparate and retroactive data and privacy protection initiatives are just bandaids on the problem. To meet its ambitious target for our government-held data to be both accessible and private, one has to think differently about data.

This is a very different type of federation to the 1901 version, of course, but a similar principle applies — let’s get into some details.

What is data mesh and why is federation relevant?

A new approach, called data mesh, is a useful start to seeing data in a fresh new way. Data mesh combines a number of technology architecture and organisational principles to enable federated governance of data. Managing data as a product in distinct domains allows local autonomy and local responsibility. Providing a self-service technology platform reduces duplication of effort between domains and allows many federated governance policies to be automated. The data mesh model thus contrasts with historical models of highly centralised management and governance of data.

We see some evidence of the data mesh model being used already in government where departments make their data open to citizens, and public and private consumers as usable products. But if we look at other organisational boundaries, such as within departments, the data mesh model presents many more opportunities to better align the management of data to the interests of government and the constituents it serves.

By moving away from a central model, towards data products aligned to source or consumption scenarios, we facilitate not just better accessibility and reliability of data, but also enable privacy by design. This is because these data product owners can make better decisions about balancing a range of privacy controls against the utility of data in their specific context. By designing data architecture and technology platforms to align to this organisational model, we make the approach scalable.

To recap, as we’ll see them below, the key concepts of data mesh are:

• Domain ownership of data
• Managing data as a product
• Removing friction with self-service platforms
• Federated data governance with substantial automation

Principles of data mesh.

How might data mesh improve accessibility and reliability of data?

Accessibility starts by understanding that consumer requirements and reliability depend on good-quality feedback cycles.

Thoughtworks partnered with Geoscape Australia, the national provider of authoritative location data to businesses and governments including Australia’s emergency services. Geoscape Australia identified a range of new data products to bring to market, but were hampered by the highly manual nature and long feedback cycles of their existing data processing approach.

With data products at the core of Geoscape’s business, applying principles of data mesh, we built a custom technology platform to support rapid and high-quality delivery of data products.

In just 10 months, the combined Thoughtworks and Geoscape team went from discovery and inception to launching the streaming data platform for real-time customer consumption, alongside a suite of quality assurance tools. The team focused equally on building the platform and lifting the capability of Geoscape to continue to extend the platform.

The new platform shortened the feedback cycle from consumers to producers of data through a streaming data architecture, while improving the governance capabilities, including full lineage and sanitising sensitive information. The combination of data product thinking, a self-service platform and automated governance dramatically improved the accessibility and reliability of data to government and business.

How might data mesh enhance privacy and security?

By putting domain experts in charge of data collection, storage and use, we allow those who know the data the best to make informed privacy and security choices. We enable them to enact those choices via a self-serve technology platform, to appropriately turn the knobs and enforce privacy standards from the source and across the lifecycle.

This approach is similar to the tradition of librarians. BJ Ard observed that “librarians have a rich history as privacy advocates who have mobilised lobbying, litigation and education campaigns to combat state surveillance of their patrons’ reading habits.”

By giving them oversight and enabling more collective power over data, we also empower our citizens and build trust. Without treatment as a first-class concern, data use may become disconnected from its provenance. The architecture of data products in data mesh, however, ensures this governance integrity of data.

For instance, in order to facilitate the sharing of trusted data assets across agencies, the Singapore Government Data Strategy (GDS) aims to build the right infrastructure for data sharing by building Trusted Centres (TC) to be data intermediaries for individual, business, geospatial and sensor data.

Thoughtworks published a white paper in conjunction with Amazon Web Services (AWS) describing how data mesh enables this vision, as a structured way of integrating people and processes with technology.

Considering the concept of federated computational governance, we demonstrate how each data product implements standard governance concerns for interoperability, including discoverability, access control, security and auditing. We demonstrate that automated auditing can surface inconsistencies in the handling of Personally Identifiable Information (PII) in an ecosystem.

Elaborating on domain ownership, we show how individual data product owners can implement additional controls beyond the minimum requirements, tailored to different consumption scenarios. This might include de-anonymising or even applying differential privacy to data before sharing.

This demonstrates how data mesh principles support private and trusted data sharing in a federated environment.

Takeaways for government

Without a trustable and resilient data sharing framework, it’s impossible to bootstrap new initiatives and policies quickly and responsibly. Lack of transparency and controls during sharing causes cracks in data privacy and security. And delays in data access requests lead to loss of valuable government resources.

Data mesh culture embraces connecting citizens, creating empathy, and a structure of shared responsibilities. This federated approach brings us closer to goals of accessible, reliable data, built on the trust engendered by a sharing infrastructure that protects citizens’ privacy and security.

If we do it well, the next few years may just rank as some of our country’s historic years, right alongside 1901.

Image credit: iStock.com/piranka