The three biggest data challenges for Australian organisations

Data is hitting the headlines almost daily due to ongoing cyber attacks on Australian public and private sector organisations. While data breaches are dominating news, there are much greater challenges facing Australian organisations when it comes to managing data.

The types and volume of data that organisations manage will continue to explode due to factors such as the growth of on-premise and multi-cloud solutions, social media and the Internet of Things (IoT). This will lead to more regulations and requirements beyond the current privacy and cybersecurity reforms, which will place even greater pressure on organisations to adopt consistency in the way they manage data. It has created a perfect storm of three key data challenges for Australian organisations: the complexity of data management, cost overruns and compliance.

1. Complexity of data management

There is a growing concern in Australian organisations about the number of different solutions in place to manage data.

An unfortunate reality across all sectors is that too many companies have taken a siloed approach to storing and accessing data instead of adopting an enterprise-wide and strategic approach to data management. The result is that over half of recently surveyed organisations need to rely on more than five tools to support their data management priorities. This introduces more complexity, risk and cost into their technology landscape because all these tools need to be integrated so they can work together effectively.

Due to the vast volume and complexity of modern data, it’s no longer sufficient to simply store data in a legacy technology landscape and manage it across siloes. To truly drive digital transformation, organisations must become data-driven by holistically managing the data they hold. This is because in the cloud, we no longer own applications, databases or even infrastructure. All we own is our data, so intelligently managing it enables organisations to rein in chaos and unify the enterprise.

However, many organisations still rely on multiple point solutions for data integration. They also have more solutions for data curation, profiling and cataloguing, and often disconnected solutions for data governance and data privacy. This complexity results in several issues such as:

• data becoming difficult to find and understand
• poor data quality meaning information can’t be trusted
• systems unable to scale to meet changing enterprise needs
• data and applications becoming even more siloed and fragmented
• data being difficult to share and not sufficiently governed or protected
• cost overruns
• resource constraints.
   

Data leaders who invest in data management capabilities that promote better control, trust, utilisation and a unified understanding of their data can overcome these issues while at the same time addressing business imperatives such as reducing costs, improving efficiencies and growing revenue. This is because they can gain access to reliable, trusted and governed data that delivers better business insights.

The way to achieve this is through rationalising and simplifying data management by focusing on data intelligence that helps business users find, understand, trust and access the information they need, and simultaneously provides the right level of governance, quality and privacy. The key outcome is to provide users with the information they need through data storefronts, such as a data marketplace solution that lets data consumers easily search for the data they need to fuel data-driven decision-making that is balanced with the right level of governance and data privacy.

2. Cost overruns

In the face of a global recession, organisations must now do more with less. This is not a new concept for businesses, but by reducing the complexity of data management they can lower their technology TCO through increasing efficiencies across the entire software development lifecycle — from the build to deployment and maintenance stages.

When rationalising software, it’s important to use adaptive data management technologies to avoid the cost overruns that are often associated with hand-coded and limited point solutions. Manual or code-based processes can render them unproductive and slow, which adds to costs while exposing the organisation to operational risks. They also place more time pressures on data engineers when code changes are required.

Conversely, intuitive data management technologies supported by AI and ML provide faster time to value by accelerating digital transformation, which reduces repetitive manual tasks and the requirement for hand-coding. These solutions future-proof data analytics initiatives through the ability to scale and adapt in the evolving public cloud ecosystem and help companies avoid lock-in vendor contracts in a multi-cloud or hybrid environment.

Best of breed, low code/no code and agnostic solutions can process data on premise, in the cloud or in any cloud, which results in improved efficiency, scalability and flexibility, simultaneously delivering trusted insights and business value, while avoiding cost overruns.

3. Compliance

Every time we board a plane, we assume that the flight crew has completed its essential safety checks. You must do the same thing to ensure that your data is protected. There is nothing more critical. You want to be assured that trained professionals have checked every aspect of the plane you are on — mechanical, physical, digital. You need to pay the same amount of attention to your data.

Questions you should ask yourself include:

• Should this dataset be protected from misuse or loss?
• Is it appropriate for my business users to access and consume it?
• Is it necessary to expose this data to meet a business value need?
• Does this dataset contain Personally Identifiable Information (PII), and thus require special handling?
• Is my organisation at risk if I allow this dataset to be used outside of our policies?
• Have customers consented to having their data used except for specific purposes?
   

Data privacy and compliance, much like aviation regulations, is not to be ignored. It not only puts you at risk of making negligent decisions but could also expose the organisation to severe fines and remediation costs if you aren’t careful. More importantly, the organisation’s reputation will suffer if there is a data breach that makes headlines, which undoubtedly directly impacts revenues and customer loyalty. This is the reality for many leading Australian organisations right now and is likely to continue with the rising number of cyber attacks that are targeting high-profile brands across finance, telecommunications and healthcare — as well as government.

However, rather than thinking of completely restricting business users from accessing particular datasets, ask yourself if you could permit usage under certain conditions — perhaps by anonymising sensitive data, while monitoring proper sharing and use. Develop opportunities for permissive use within an appropriate context, instead of only restrictive policies, to accelerate giving users safe access and the ability to create value from the data.

As data volumes grow and industry regulations become even more complex, the ability to access, understand and use data securely is key to organisational success. Without a unified and comprehensive data platform, organisations are forced to cobble together disparate point solutions that were never designed to work together in the first place. Integrating these systems is time consuming, costly, risky and inflexible to change. If one point solution changes, then you need to redo and retest all data management processes and your ability to drive data literacy programs reduces considerably.

As well as scalable, secure data management technologies, data literacy programs are often the most effective way of increasing an organisation’s data-driven culture. The better informed employees are about what insights are available and how those insights can benefit them, the more likely they are to embrace it and access data securely. This is why building a robust foundation for data intelligence is key to empowering data consumers and addressing business imperatives, without compromising data security.

With predictive data intelligence, you can transform organisations, bringing people and data together by simplifying and automating the ability to quickly find, understand, trust, secure and access data. And with a data consumption focus that improves data delivery, business outcomes can be accelerated.

Image credit: iStock.com/NicoElNino