Inside the Census: capturing the nation's data

Trust has never been more critical, or easier to lose, in our increasingly digital world. Nowhere is this more evident than in the relationship between a government and its citizens. For the 2021 Census, the ability to deliver an experience with security, accessibility, simplicity and ease of use at its core was essential to its success. The insights gained from the Census help inform decisions for the essential services of today for Australians, while helping to shape the Australia of tomorrow.

The ABS, telling the nation’s story

As Australia’s national statistical agency, the Australian Bureau of Statistics (ABS) plays a vital role in collecting and making available data to the benefit of society by informing some of the most important decisions impacting the lives of all Australians. The statistics provided by the ABS tell the nation’s story — and the most important input is the Census.

A powerful combination of human-led, tech-powered

For the 2021 Australian Census, the ABS worked with PwC Australia (PwC) and Amazon Web Services (AWS) to build a world-class digital solution to capture the data completed online. It was essential that this solution had robust cybersecurity to keep our nation’s data confidential and secure, whilst maintaining high levels of availability and performance for all participants. The ABS brought its strong focus on designing and delivering a contemporary level of usability and accessibility across a range of devices and browsers.

The Census — more than a single night

A national exercise to record data relevant to Australia’s population, the Census is conducted every five years, counting every household and person in Australia. The Census captures a point-in-time snapshot of Australia.

The ABS delivered three key success factors for the CDS. The first was ensuring the smooth running of the operation, so that everyone had an experience with the CDS that was easy, secure and simple. The second was ensuring it had the confidence of government, business and the community given the sheer scale of participation required. The third was that the data delivered would have to be of the highest quality.

Underpinning these success factors was the evolution of the relationship “from one of a more traditional, supplier or vendor type relationship to where all parties were aligned and collaborated towards a common goal”, said Scott Evans, Census Lead Partner, Digital Innovation & Cloud Engineering at PwC.

Innovative by default

Given people’s expectations and previous experience, the ABS expected the public would largely complete the 2021 Census online, and sought a cloud-based solution to support an online completion target of above 75%, which represented over 18 million people. To support this, PwC proposed to build a solution leveraging the latest innovative cloud capabilities of AWS. PwC’s proposed solution was to maximise the use of cloud services “to design, implement, test and operate a solution to fulfil a demanding set of security and performance requirements, using the scale and resilience of AWS’s cloud native services”, according to Evans.

Scalability, performance, and resilience

When the ABS came to market in August 2018, it was looking for a knowledgeable and experienced organisation, and a solution that would be easy for Australians to use and allow it to mitigate the risk of cyber attack. This was especially important given the significance of the Census, the sensitivity of the data captured and the security incidents that disrupted the previous Census in 2016.

“A key focus of the 2021 Census was to ensure the digital service had redundancy, performance and protection against cyber threats,” said Gwil Davies, Partner, Digital Innovation and Cloud Engineering at PwC Australia.

As the solution Lead, Davies focused on the architecture, engineering and operationalisation in readiness for the Census. The end-to-end solution architected and developed by the team would be custom built using a variety of technologies to address these specific needs and would leverage 55 services from AWS.

A community of over 140 solvers

The ABS, PwC and a number of others worked together as a community of solvers, with the PwC team comprising over 140 people from different teams and diverse skill sets during the project focused specifically on the delivery of the CDS.

The application required a wide range of specialists working together, including front-end developers, accessibility specialists, content management specialists and Java developers for the back end. The platform required cloud engineers specialising in the latest DevSecOps delivery methods, and automation of highly scalable and secure solutions, with a cyber team focusing on cyber architecture, application security and security operations.

These teams were supported by a test team focused on functional, performance and security testing leveraging extensive automation. For operations, the team included online operations and service management specialists to oversee the running and support of the service. To help manage and coordinate the full program delivery, the team included agile delivery managers, scrum masters, agile specialists and business analysts.

Building trust at the intersection of human and digital

Robert Di Pietro, Partner, PwC Australia Cybersecurity & Digital Trust, was the Cyber Lead for the Census project.

“From tender to delivery, we knew that cybersecurity would have to be embedded in the solution and throughout all phases of project delivery,” Di Pietro said.

Given the profile of the project, and the accompanying risk of cyber attack, the ABS embedded experts from the relevant government cyber agencies.

“It was incredibly important to see a strong working relationship with the ABS develop.

“Trust doesn’t happen by accident — it is earned and developed through collaboration, respect for diverse views and inputs, and brought to life through commitments to a joint and shared outcome. All parties committed to a strong spirit of collaboration to support the ABS,” Di Pietro said.

The PwC Cyber team also ran regular security awareness sessions and cyber threat briefings for the broader project team. Cyber became everyone’s responsibility on the project, and not the role of one team to get right.

“Success from a security perspective is based on having people who are not only technically smart, but understand people and human behaviour.

“That meant knowing how to communicate potential security risks and issues, as well as recommending mitigations, while still balancing other priorities such as user experience and performance,” Di Pietro said.

A sustained focus on security

In solving this important challenge, the PwC cyber team was responsible for managing security operations to defend the Census from cyber attack.

“This wasn’t just design and architecture, but the front line of operations and cyber defence,” Di Pietro said.

Given the nature of the solution it was a matter of when, not if it would be targeted. The PwC cyber team were responsible for leading 24/7 security incident response efforts and working with multiple stakeholders — including the ABS, AWS and government cyber agencies.

Di Pietro explained that the uniqueness of the Census is that not only is it a significant exercise in collecting data, but also a time-sensitive activity that cyber attackers could be drawn to. Attackers knew not only what their target was, but exactly when they should strike to maximise disruption, particularly for distributed denial-of-service (DDoS) attacks which were a key concern given the challenges encountered by the 2016 Census.

“This posed a significant cyber challenge that the team had to defend against from day one,” he said.

This combination of the availability of the system, the confidentiality of the data and the importance of its accuracy meant the platform had to be highly secure and resilient. Often in cybersecurity, one of these three factors would be prioritised over the other, but in the case of the Census, all were of equal importance.

“The Census is the largest statistical collection undertaken by the government.

“This information must not be lost or stolen, or be interrupted, and it has to be accurate, given the decisions the Census informs,” he said.

A rapid pivot through the pandemic

The team had a year working together before the pandemic hit but flexibility had been built in from the start.

“Working hard through the tender process, we were clear on our preferred way of working, co-located with agile hubs and delivery sessions.

“When the pandemic hit, and lockdowns were introduced, the team had to pivot to a working-from-home model. The upfront investment in security meant we handled the transition smoothly,” Evans said.

The team maintained its velocity, even with the pivot, according to Davies.

“With a large multidisciplinary delivery team, we’re proud of the resilience that they all showed to pivot and operate remotely, pretty much overnight.

“With our investments in security, tools, capabilities and risk management, the project barely missed a beat during the pandemic. We were able to adapt quickly to keep the show on the road, and the foot on the pedal,” he said.

The Census is held every five years, which helps reinforce the importance of keeping to schedule, and the pressure that comes with it. As an example of the scale of the endeavour, the ABS was recruiting 33,000 people trained in COVID-safe protocols to knock on doors, while the digital solution was being built.

An important milestone

In building, earning and sharing trust the project reached an important milestone when it went live with a Census Test in October 2020.

“We had a number of simulation events, and rehearsed cyber incidents to test processes and ensure we had the muscle memory developed for the main event. This was in addition to multiple rounds of security code reviews as well as penetration tests designed to emulate the tactics of a real attacker,” Evans said.

Running for two months, 100,000 households were asked to participate in the Census Test.

“This test provided an important proving ground for the project, embedding trust in the relationship,” he said.

At the same time, there was still a significant way to go before the launch day. However, the success of these tests meant that by the time Census night came along, all parties — PwC, AWS, ABS, government cyber agencies and other important third parties — knew how they’d respond to an incident.

Succeeding on Census night

This year the ABS made it clearer to the public that they could respond over a period of time rather than focusing on Census night as was the case in 2016. This allowed for households and individuals to submit their responses ahead of Census night.

By Census night, confidence and trust in the solution was strong. With over three million Census submissions before 10 August 2021, the team had been proactive in operations and mitigated risks as they emerged, and on the night everything went smoothly and as planned.

“We were confident going into Census night given the extensive preparations and hard work by everyone involved, and were delighted with the outcome for the ABS.

“We had a number of operational dashboards, to monitor the service and watch the level of submissions. On the night, volumes peaked at over 270 logins per second, and 142 form submissions per second, and it was great to see a total of 2.8 million forms submitted on Census day,” Davies said.

The system remained live after Census night until the end of September, and continued to receive thousands of forms per day throughout the period. The team stood side by side throughout the event with the ABS and other providers involved.

A resilient outcome

“When it came to the solution, we did everything we said we would. As a cloud native solution, available to anyone online, we built it for maximum security from the start,” Evans said.

The proof of this is substantial, the solution was successful in blocking around 130,000 malicious IP addresses on the system across the lifetime of the CDS.

“We built a resilient solution, capable of withstanding attacks and with no interruption to service,” Di Pietro said.

“The investments made into security mitigated and effectively stopped in their tracks anyone attempting to find a way in. It did not mean they didn’t try, they did, but the solution was able to withstand these attempts,” Evans added.

A new benchmark for digital delivery in government

While the ABS does much more than the Census, it’s the most prominent of its activities.

“A successful Census not only meant a smooth, resilient and available digital solution for the ABS, but one that people could trust,” Evans said.

Given the target user base comprises the whole of Australia, the ABS brought a keen focus on ensuring usability and accessibility of the Census Digital Service (CDS). The CDS was able to operate on 95% market share of physical devices, browsers and operating systems. The CDS has been heralded as a “world-class” service in support of accessibility for deaf and vision-impaired people, through its expansive application of Web Content Accessibility Guidelines (WCAG 2.0 AA).

In terms of building trust and delivering sustained outcomes, the project successfully used AWS for critically sensitive government data, at a PROTECTED level of government classification. The successful delivery of this project provides a valuable example of what can be done with the use of innovative digital technology.

“This sets a blueprint for public-facing digital services going forward.

“The government can embrace cloud for critical and sensitive workloads, and realise the benefits it can bring. “It’s not only scalable and resilient, but also more secure and as a result builds trust,” Evans said.

Recognising that Australia now has a leading example of how the government has worked across agencies with PwC to deliver a successful digital outcome, the team view this as a ‘stand-out achievement’. The project sets the standard, as a global reference that the government can be proud of on a global scale.

According to Steve Hamilton, Chief Information Officer (CIO) ABS, “The 2021 Census has set a new benchmark for digital delivery by government.”

Solving through The New Equation

Instilling trust in the government’s digital abilities, as well as taking care of the data of people, the scale of the project represents significant social impact, instilling confidence in the digital services of government, as well as data and the security behind it. In addition, significant energy was invested by the ABS and PwC working closely together to develop a highly accessible and engaging user experience, with end-user feedback being overwhelmingly positive.

The approach of multidisciplinary end-to-end delivery is one that has been embraced within PwC’s Digital Innovation and Cloud Engineering team more broadly.

“We brought hands-on solution delivery, software engineering, end-to-end cyber, platform engineering and operations,” Davies said.

“The golden thread of PwC’s The New Equation is about bringing communities of problem solvers together, with a new level of collaboration and deep specialist disciplines, that brings the best of PwC and other technology partners, like AWS, together,” he said.

“Where human meets digital, we solved it as a community, for the community, and that's something PwC is very proud of delivering.”

Census 2021 by the numbers

• The Census counts every household and person in Australia — that’s around 10.5 million households and 25 million people.
• The target rate for online completion of the Census online was more than 75% of the total population.
• PwC’s project team designed 29 Epics, ~550 Technical User Stories, ~8500 builds and nearly 2000 releases as part of the build phase.
• Testing included 12 test streams and regular execution of around 15,000 test automation scripts and ~420 test cycles across a range of areas including functional, accessibility, integration, APIs, devices, DDoS, penetration testing, disaster recovery and service resilience.
• On Census night, volumes peaked at over 270 end-user logins per second and 142 completed form submissions per second.
• The Census Digital Service leveraged 55 AWS services and 70 Hardened Cloud automated compliance rulesets with ~37 million compliance checks monthly.

Census technology deep dive

Innovation with cloud native technologies

Given the hyperscale and highly variable nature of the workload, the architectural choices were key to success. PwC chose to use AWS Lambda as the core computing platform for the cloud native application. Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers. In the design phase, PwC’s solution architecture and engineering teams carefully modelled the volumetrics and business inputs to identify potential ‘hot spots’, and took a rigorous approach with validation through automated performance tests, to refine key design principles. These included approaches to optimise execution time including optimised memory allocations to reflect the characteristics of various application components and techniques to maintain front-end application performance for end users, and system throughput under a highly variable workload, with a combination of Provisioned Concurrency and Application Autoscaling.

The importance of observability

In modern cloud operations, it is vital that insights on business and technical metrics can be accessed at near real time. Due to the stringent requirements within the solution for the separation of duties and data confidentiality, the operations engineering team built extensive automation and application logging frameworks. These involved a streamlined approach to log shipping and secure replication, and a variety of techniques to drive insight via analytics into solution performance and availability. In addition, as this solution made extensive use of cloud native services, a tailored approach was developed with a complementary combination of custom solutions and native AWS tools.

Continuous compliance monitoring for additional guardrails

Security and compliance of the solution was paramount throughout, not only during the live operations but also right from the outset of the development phase. Whilst various vendor tools can cover aspects of these requirements, the specific demands of this project also benefited from an enhanced and extended set of capabilities.

Throughout the delivery phase of the project, PwC deployed its Hardened Cloud asset, a continuous compliance checking framework built on serverless technologies. Hardened Cloud implements customised compliance checks for additional guardrails, in addition to those configurable in a native cloud platform. Hardened Cloud helped to make sure developers had the appropriate level of permissions they needed to perform their roles, and helped make sure that the settings of deployed services were compliant with our best practices and regulatory requirements on an ongoing basis, and not just at deployment time. With extensive automation, and integration into the team’s ITSM (IT service management) ticketing system, Hardened Cloud also supported the rapid notification and resolution of any issues if they occurred.

Operational rehearsals and readiness

Operating a large-scale publicly accessible online application can be challenging, and as such, ‘game days’ and simulations are an important part of readiness preparations. Prior to the solution going online to the public, PwC worked with AWS architects and engineers to define scenarios of the most critical potential incidents covering security, reliability, operational excellence, performance, and cost optimisation, following the AWS Well-Architected Framework.

The team was then divided into two groups: the Red Team to design the injection of errors to simulate the occurrence of an incident, and the Blue team, who had to use the monitoring and alarming mechanisms developed to detect and analyse what had happened and then resolve the incident. These simulations, done both within the PwC operations team, and done in conjunction with the downstream ABS operational teams, were an important step in readying the team for potentially known scenarios and to build the troubleshooting muscle memory if required.

Defending at the edge and mitigating DDoS attacks

The highly publicised challenges of the 2016 Census meant that defending at the edge and mitigating DDoS attacks was a primary area of focus in designing a highly resilient and secure Digital Census for 2021. The solution leveraged the DDoS protection mechanisms offered by AWS to protect against large-scale volumetric DDoS attacks at both application and infrastructure layers.

The PwC team undertook extensive configuration and tuning of Web Application Firewall (WAF) rules, which were finely balanced to allow large spikes of legitimate Census traffic whilst also blocking potentially unwanted DDoS traffic. This process included ABS-led large-scale DDoS tests which simulated massive amounts of traffic targeting the CDS and its supporting AWS infrastructure, equivalent to being in the top 1% of DDoS attacks observed by AWS globally. These successful tests were a key step in providing the confidence that the solution would remain available and perform on Census night.

Image credit: ©stock.adobe.com/au/fizkes