Filling shot glasses with a fire hose — the reality of data management

Seagate’s 2020 Rethink Data report revealed that enterprise data is expected to grow, in 2021 alone, at an average annual rate of 42%. IDC also predicts an astonishing increase in the total data generated from 64ZB in 2020 to 180ZB by 2025.

While this data growth is fuelled by growth in Internet of Things (IoT) devices (especially cameras), and automated M2M interactions including utility smart meters and healthcare device management systems, Government organisations are significant contributors. This is the case particularly around smart city initiatives with substantial needs for ramping up its data analytics infrastructures and intelligent transport systems.

The value and power of this data is considerable, but so too are the challenges standing in the way of managing the data effectively to drive innovation. Indeed, the process of trying to access data can sometimes feel a lot like trying to fill shot glasses with a fire hose. As a result, just 32% of enterprise data gets used — because capturing, storing, and managing the data deluge can be tricky, as can accessing and transporting mass data.

Organisations that overcome the cost and complexity challenges of data lakes are rewarded with an asset whose value grows rather than a data sink hole that is complex and costly.

The question is — how to reach that point? That requires answering two key questions — how can we get more value from our data in the distributed world, and what kinds of storage strategy can reduce impediments to the movement of massive data sets?

Ten years ago, organisations debated between storing data in the public or private cloud. Now, the situation is far more nuanced. Data flows through endpoint devices, edge, and cloud systems, but the pace and rapidly rising volumes of that movement must be accounted for. According to Seagate’s Mass Data on the Go report, today the leaning is toward multi-cloud and hybrid cloud models, which can help optimise where data gets stored and how to best distribute, access, and use it.

While the Australian federal government has a clear digital transformation strategy and mature service capabilities, it still faces the same data pains that enterprises around the world encounter — that is network capacities do not keep up with the data growth. According to the Mass Data on the Go report, in addition to network and capacity constraints, limited access to fibre-optic networking, hidden costs, data security and compliance concerns, and storage capacity limitations also constrict the effective movement of data.

So, what kind of storage strategy can assist? Today, data creation is particularly vibrant at the edge, which means the decade-old model of choosing whether to keep data local or in the cloud has proven too simplistic. While 47% of enterprises today use a centralised cloud storage architecture, in two years that number will fall to 22%. Conversely, the number of those that currently have a hybrid storage architecture spanning centralized and edge locations will jump from 25% to 47% in the same period.

The next storage technology leap forward for Government organisations will likely involve a shift toward using storage-as-a-service offering that complements but does not replace existing storage services. Seagate Lyve™ Cloud is a great example of this — a world-class object storage service where data can find permanent, cost-effective residence, be activated for a host of applications, and be instantly available for flowing to edge locations via high-speed backbone links.

At the data centre level, the self-healing high-density high-performance storage system, Exos CORVAULT, enables streamlined mass storage management and reduces human intervention for edge and data centre environments. The next-gen storage intelligence automates maintenance and reduces e-waste while delivering maximum data density and security without controller-level overhead.

The deluge of enterprise data is not going away any time soon, and this affects Government organisations. Thankfully, data storage and management tools have come a long way — from storage-as-a-service to physical mass data shuttles and self-healing storage systems — so we can fit the right ways to get the most out of data to the right data sets, no matter how big.

Image credit: ©stock.adobe.com/au/Kalawin