Tapping people power for big data

Forget high-specialised job applications and secretive development projects burdened by red tape; today's governments are going straight to the people to tap into new creativity in security testing, big data, application development, and more.

Security testing and application development used to be kept quietly inside government departments, conducted by internal staff or trusted consultants with a remit to find and improve deficient elements of their organisation’s IT infrastructure. Innovation, as it were, was often eschewed for rigorous governance, status-quo inertia, and territorial disputes over data ownership.

Times change, though, and so do methods of tapping into the rampant problem-solving creativity that’s now accessible within the global Internet community. And so it was that none other than the federal government – in an unprecedented partnership with Telstra and a number of Australian universities – found itself sponsoring the 2012 Cyber Defence University Challenge (CDUC), a nationwide competition in which dozens of hackers tried to break into a putative secure network throughout the course of a number of purpose-built scenarios (2012 winners are listed here).

A team from the University of NSW eventually beat out a dozen others to take out top honours in the 24-hour competition, which ran in early April and will see the winning team sponsored to attend the Black Hat 2012 hacking conference in Las Vegas, in July (the 2013 competition runs on May 7th 2013, with similar prizes up for grabs – ed.).

Supporting the program were a range of government bodies including the Attorney-General’s Department, Department of Defence, and Department of Prime Minister and the Cabinet – all organisations that would previously have shied away from explicitly or implicitly encouraging hackers or their activities.

Given this history, it was significant that Communications Minister Stephen Conroy stood up to declare the government’s support for a friendly hacking competition.

“The government is committed to increasing the skills and expertise necessary to protect online environments from cyber crime such as data theft,” Conroy said while announcing the CDUC in March. “By engaging government departments, universities and the private sector, the Challenge aims to highlight the variety of career opportunities available in ICT fields.”

While the government may have seen the competition as an exercise in skills building, however, its participants felt it at different levels – as a validation of their interest in testing the boundaries of network security.

“Just the existence of the competition is of incredible value,” says Richard Buckland, a senior lecturer within the UNSW Faculty of Computer Science and Engineering and mentor for the winning team.

“Applied security has been a poor cousin in the security world; groups that had this expertise have tended to exist outside and independently of the establishment. Initially, [hacking] was a subversive and countercultural thing to do.”

"Applied security has been a poor cousin in the security world; groups that had this expertise have tended to exist outside and independently of the establishment. Initially, [hacking] was a subversive and countercultural thing to do."

With a government-sponsored hacking competition now having run to completion, however, Buckland is optimistic about the field’s future.

Government support for the competition was “fantastic,” he explains. “It legitimised what has been a Wild West as a scholarly field, because things are moving very fast. It’s one of these great new fields, where what everyone knew five years ago is replaced by what we know now. And the CDUC has raised awareness within first and second-year students who are now keen to get involved.”

Call for apps. While competitions like the CDUC reflect a better-the-devil-you-know mentality by security-conscious government organisations, others are going to the public through similar efforts designed to tap into rampant creativity that will – they hope – bring out new ways of thinking about government challenges and using massive stores of government data.

NASA, for one, recently took a big step forward with the execution of its International Space Apps Challenge, a multinational effort modelled under the US Open Government initiative that brought together interested volunteers for a weekend of coding.

Working with big-data sets provided by government bodies in participating countries – in Australia, participants included Geoscience Australia, the Bureau of Meteorology, and CSIRO as well as a number of private research institutions – developers collaborated with each other and with 25 other teams around the world to tackle big problems in areas including software, open hardware, citizen science, and data visualisation.

In Australia, the Space Apps Challenge played out across venues in Sydney, Melbourne, Adelaide, and Canberra, where participants were linked using live uStream video feeds and online chatting. Additional challenges for the Australian teams addressed areas including disaster prevention, the environmental impact of personal travel, activity sensing, and climate adaptation.

“It’s very well to say all this data is there in government bodies, but the challenge is connecting the people to know it’s there and what you can do with it,” says Dr Naomi Mathers, program developer with the Victorian Space Science Education Centre, which hosted the Melbourne leg of the Space Apps Challenge. “This is a chance to bring together a diverse group with very different ideas and different skill sets.”

The very existence of the competition reflects a growing comfort amongst government organisations in sharing their public data sets, Mathers says. “In general, everybody’s becoming more comfortable with opening up and getting the community more involved,” she explains.

“It’s very well to want to look at the data, but we need something that a user can interface. The apps bring data together to empower users to make an informed decision. And it’s always better to have the best minds on your side to show you where the gaps are, so you can fix them.”

Although its short-term deliverables may be of interest, the Space Apps Challenge’s longer-term goals include the creation of new partnerships that may foster further collaboration and use of massive government data sets. In effect, the competition has been the government’s way of stimulating new innovation that may well pay even bigger dividends down the track.

“Often, developers and researchers stay in their own industry and environment, and there aren’t too many opportunities to collaborate broadly,” says Mathers. “But once you start having these conversations, it generates the next need. Rather than just throwing a whole lot of money at a project, we can throw together a lot of ideas, bring together these groups, and look at how we can spin that off to make an impact in other industries.”

Big data, big possibilities. NASA isn’t the only government body tapping into the worldwide base of skilled developers. Following on from early experiments such as Apps for Democracy and Victoria’s App My State competition, state bodies and even local governments are now running competitions to engage with developers in new ways – and, potentially, to find new solutions for problems that really matter to constituents.

Brisbane City Council (BCC) is among the latest organisations to jump on the bandwagon: its hack::Brisbane competition, run in May 2012 as a successor to 2011’s HackFest Brisbane, offered $10,000 for the best Web site or application built using BCC data. A separate category also offered $10,000 for an app that “helps make Brisbane a more accessible and inclusive city” – reflecting the potential for such competitions to directly reflect the needs of citizens (winners and entrants can be viewed here).

Whether or not the competitions produce commercially viable applications with long-term possibilities is yet to be seen: many similar past efforts have fallen by the wayside through neglect or a long-term commitment to updating the government data sets provided.

“I’ve been maintaining my app since the AppMyState competition, but my complaint has been a lack of follow through from the government departments who provided the data sets,” developer Richard Nichols, who authored the onmydoorstep.com.au application in the Victorian competition, recently warned in an online posting.

“I’d be happy to incorporate new data sets and process updates to the current ones, but almost none have been touched since the comp. Without fresh data, the usefulness of these apps and the data is limited.”

Whether the specific results of the competition survive, however, they still serve a purpose by building greater awareness in areas like open data and security – and that, says Telstra chief information security officer Glenn Chisholm, makes competitions like CUDC more than worth it.

“The government was fabulous – not good, not great, but fabulous,” Chisholm explains. “Government involvement in CUDC was critical in that they increased the legitimacy of the competition and helped us with the interactions with universities.”

“This sort of effort is all about embedding good practice within the entire IT ecosystem,” he adds, “and the entire game was designed around these people having an opportunity to learn in a way that you can’t always learn in a theoretical environment."

"If people don’t trust information systems and security, it undermines us as a society – and whether or not the CUDC participants go into computer security in the end, if they design systems with security then we’ve gotten that outcome.” – David Braue

This feature originally ran in the April/May 2012 issue of Government Technology Review.