Paper rules, digital world

Digital natives used to argue that formal recordkeeping was a dying art – but with the world now awash in unstructured information, records managers are having the last laugh.

NSW government agencies will soon be fielding enquiries from State Records Authority (SRA) of NSW, a Department of Finance and Services agency that has spent the last year figuring out how to manage digital records with the same sort of meticulous approach it has used on paper records for decades.

Like similar archival authorities in other jurisdictions, SRA NSW found itself facing the need to record, index and archive all kinds of natively digital information – as distinct from digital images created in long-running programs that include the scanning of physical documents.

“We were missing the ability to take those records that are generated in government, for archival retention of records of significance to the state that were never on paper,” explains Cassie Findlay, Digital Archives project manager with SRA NSW. “These are records that have been kept in digital format, and that we would want to take in and keep in digital format, and eventually to make available through a variety of means.”

Working with service provider Logicalis, SRA NSW last year embarked on a major review of similar efforts in other jurisdictions. Advice from organisations such as the National Archives of Australia and Public Record Office of Victoria was consistently cautious: digital archives require both discipline and storage space – lots of it.

“They told me to estimate what we think we might use in the first few years, then multiply it by ten,” laughs Findlay, noting that SRA NSW’s initial system offers 24TB of online EMC Isilon storage. “We know we’ll be needing to refresh our technology in five years anyway, so it will be a matter of seeing how we go as we take in some of our early transfers.”

More than a dozen pilot projects are already underway as the Digital Archives project moves from its first, research and design, phase into its current second stage. Yet the project is more than a novelty for state agencies, which face a statutory requirement for records retention and will no doubt welcome the ability to have SRA NSW take over the burden for them.

Records-retention laws mandate the same sorts of protections for digital records as they do for paper: access control rules and minimum retention times, for example.

Yet the digital archive has proved even more complex because of the sheer range of data types and sources used by government bodies on a regular basis: typical word processing, spreadsheets and presentations are complemented by geographical information systems; case-management databases; customer relationship management environments; student registration systems; health records; Commissions of Enquiry reflecting tens of thousands of documents; and much more.

Even records-management systems used within individual agencies fall under SRA NSW’s scope.

“The challenge for us, and the complexity in this project, is mainly around how we work on a case by case basis,” Findlay explains. “We analyse the records, map the metadata, assess format issues, and work out the best possible way to store those records so they are accessible.”

By working with individual agencies, Findlay anticipates that many will be involved in transforming their documents to one of numerous open, standardised formats before the records are transferred to SRA NSW; this will attenuate the need for file conversion and also encourage agencies to be active participants in selecting and preparing relevant documents.

“We have no intention of being a dumping ground for every last piece of digital information the state generates,” Findlay says. “We have a particular statutory role that we need to fulfil – and, like all government agencies these days with limited resources, we need to be careful.”

The new recordkeeping

Across Australia and the world, authorities are tackling similar issues as they seek to keep a grip on the exploding volume of information whose gathering momentum is pulling it ever harder towards the open-slather Internet.

This time around they have more standards to help out. Victoria’s VERS (Victorian Electronic Records Strategy) has retained its primacy in the recordkeeping world, while the National Archives is among the agencies that has endorsed the international ISO16175 standard, entitled Principles and Functional Requirements for Records in Electronic Office Environments, which provides considerable guidance for agencies concerned about the new digital recordkeeping.

Just as effective recordkeeping systems ensured that paper records can be located, identified and carefully managed according to document policies, today’s records-management systems ensure equal treatment for the digital records that comprise the majority of new primary-source information in today’s government.

Unlike in the past, however, those documents are likely to be tied up in much more-active lifecycles as they are made available online, archived, referenced, and recombined into new efforts. This has forced vendors to build modern systems as large, flexible and highly scalable environments capable of keeping up with long-term demand.

John Palin, general manager of product management with content-management vendor Objective, believes two major trends are driving the development of the market.

One is the ability to easily share information with people outside of the organisation – and the attendant need to maintain control over that information even when it’s been extended there. This is something of an issue for internal information usage, but it’s particularly pointed when Web portals – and a growing push towards open government and information sharing – now make it simple to publish information for public consumption.

“We’ve really focusing on providing ease of use for end users while hiding the complexities of records management and its associated controls,” Palin explains.

This ties in with his second major trend: the interaction with mobile devices, which demand a new level of design utility as they lend both immediacy and relevance to the use of stored records. Users expect to be able to do everything they want from a mobile device, so records managers can’t design their procedures around the assumption that users will be spending all their time at desktops.

“You’ve got to fit into how they’re working, as opposed to forcing them to work the way records systems work,” says Palin. “That’s been a common theme throughout the adoption of digital records systems, but it’s becoming even more important as people look at these sorts of mobility and sharing problems.”

Such concerns have informed SRA NSW’s work and were part of the reason it chose EMC’s Isilon storage platform, says Findlay. “That gives us a lot of flexibility and makes records easy to manage and migrate way into the future,” she says. “The records that sit at the heart of this will ideally be here forever, while the technologies around them all change.”

Old challenges are new again 

Although the tools and procedures for structured record keeping may be evolving, many of the old issues still plague organisations working to get their records management practices up to speed. Employee buy-in remains a particularly challenging issue: individual business-unit managers may be able to set the tone across their small corner of the organisation, but history has shown that implementing this kind of change across many departments at once is bound to leave some people disappointed.

Throw in the historical lack of clear penalties for non-compliance – when was the last time you read about an employee being fined for failing to add appropriate metadata to a records- management system? – and it’s clear why doing this sort of project properly is about a lot more than just technology.

“It’s a balancing act for a lot of the records managers out there,” says Alan Gregg, ANZ senior consultant for global services with Open Text. “The National Archives say a record is a record regardless of format, while the IT approach is very much a catalyst-allopathic approach where they throw technology at the problem to make it go away. We end up with IT departments running email archiving solutions to keep everything, forever.”

That kind of thinking has become impractical given the sheer volume of informal communications through social media channels. To deal with this and similarly massive data sets, some records-management bodies are also considering how to index content that is relevant to their operation, but hosted externally.

Doing so creates additional issues: whether to change the presentation of the information so it’s compliant with the metadata and presentation standards of the records- management portal, for example, or whether to leave it as is.

“It’s not necessarily the easiest solution to put everything in the same place,” warns Gregg. “That’s necessarily a paper-based way of thinking, but as new systems are added it will reach its natural threshold fairly soon. The new frontier is taking something amorphous like Twitter, and taking some measure of control over it.”

Could cloud computing, with its effectively infinite capacity, provide some assistance? Certainly not offshore providers, who are unlikely to gain much traction trying to convince state records authorities to store crucial government documents overseas.

Whether or not local providers can elbow their way into the market remains to be seen. In the short term, however, records managers are continuing to deal with the larger issues of data structures, archiving policies and methods of presentation.

In other words, the more things change, the more they stay the same. “At the core, the problems remain the same as they always have,” says Palin.

“Organisations need to meet their compliance needs, have strong governance over their information and strong security. So, core needs around governance and security have had to apply in this new world – and that’s been challenging for a lot of organisations.” – David Braue

This feature originally ran in the September 2012 issue of Government Technology Review.