COVID-19 vaccination scheduling: Scaling REDCap with AWS

Academic medical centers (AMCs) are playing a vital role in the research and treatment of COVID-19, as well as the delivery of the vaccines that prevent it. With the vaccine rollout within the United States rapidly expanding, AMCs have become a core resource for many state and county health agencies, especially around managing registration and scheduling systems. Texas A&M Health was a key player for the Brazos County, TX vaccination rollout, leveraging Amazon Web Services (AWS) for success.

Providing vaccination rollout and support services to county and state health agencies often means relying on implementing third-party software to meet tight deadlines. Texas A&M Health was on the forefront of COVID-19, handling the COVID-19 vaccine registration system behind the Brazos County Community COVID-19 Vaccination Hub, as Texas rapidly expanded vaccine eligibility from critical populations to all residents 18 and older.

Brazos County decided the vaccines would be made available for scheduling each Friday morning, at which time the Brazos County Community COVID-19 Vaccination Hub website would allow connections to the registration and scheduling site managed by TAMU HSC and hosted in AWS.

Vaccine scheduling challenges

Texas A&M Health chose to build this vaccine registration site on an application they had recently migrated to AWS: REDCap, which served as both the sign-up page for the general public, and the backend scheduling and verification tool for clinical staff administering the vaccine shots in the field. They’d vetted the application through previous research projects.

Texas A&M Health faced different challenges, with outages of their registration application impacting both the general public, who would be unable to schedule vaccination appointments, and the clinical staff who would be unable to verify appointments and confirm vaccine administration.

REDCap is a secure web application for building and managing online surveys and databases, built by a team from Vanderbilt Institute for Clinical and Translational Research (VICTR) based at Vanderbilt University Medical Center. Texas A&M Health deployed REDCap in AWS Elastic Beanstalk and Amazon Relational Database Services (Amazon RDS) based on the REDCap quickstart guide developed by AWS in conjunction with Vanderbilt, which is suitable for capturing HIPAA data as needed for this use case.

Figure 1. Illustration of the recommended architecture of REDCap on AWS with the addition of the Visitor Prioritization solution, showing the data flows from users to the application under normal load, and how users are redirected to the waiting room in high traffic situations.

When Brazos County announced the first round of vaccination doses were available for scheduling, the initial demand on the system was much higher than expected, seeing approximately 48,000 user connections within five minutes. Rob Taylor, manager and lead developer of the REDCap team, noted, “The load and utilization seen here were really unprecedented, but this challenge gave our team the extra motivation needed to really dig in to find a solution rapidly to help make the project a success.”

Despite all best efforts and planning, a 30-minute outage highlighted some architectural shortcomings, and impacted both the general public, who were unable to register and schedule vaccination appointments, and also the clinical staff in the field who were administering the shots. This outage led to Texas A&M Health engaging AWS to identify solutions and provide guidance and resources to resolve over the next few weeks.

How AWS helped

The AWS account team worked with the Texas A&M Health team to review the architecture of the application. A load testing specialist from AWS trained the staff to identify bottlenecks and application weaknesses by load testing their application through imitating real-world customer interaction at scale using the Locust.io load testing framework, initially deployed by the AWS team on AWS resources at no cost to the customer. This load testing environment was later taken in-house by the Texas A&M Health team using the AWS Cloud Development Kit (AWS CDK), leveraging the AWS CDK Locust Deployment and serving as an enrichment exercise to create a Standard Operating Procedure (SOP) on why and when to load test for future projects.

Following this, the AWS support team helped Texas A&M Health identify constraints that exist when rapid scaling occurs before they became an issue. AWS helped mitigate the resource limitations sometimes seen with such sudden bursts of traffic through pre-warming the AWS Application Load Balancers used by AWS Elastic Beanstalk and limit increases for concurrent operations in Lambda@Edge.

AWS teams working jointly with engineers at Texas A&M Health were able to identify points in their solution that might require resource pre-warming to be able to absorb sudden spikes in load, and were able to escalate and route these requests to the correct teams quickly, producing results for our customer rapidly. Through these engagements, we jointly iterated and optimized the REDCap architecture to improve availability, resiliency, and responsiveness.

The AWS team also engaged REDCap to optimize some database queries, as well as some of the REDCap database connection pooling code, and provided the Texas A&M Health team with a pre-release version of these updates to load-test and ultimately deploy into their production environment. The REDCap team was able to identify and resolve performance bottlenecks to make the application up to ten times faster under large loads, and these improvements went on to benefit the many thousands of organizations around the world that also utilize REDCap.

Key to this was the open access calls during scheduled vaccination scheduling events, as well a discovered need to rapidly scale the cloud expertise of the Texas A&M Health team. We leveraged the ACM team, AWS support, and an AWS technical account manager (TAM) to scale the resources available to the AWS account team in order to make sure Texas A&M Health were successful with this project.

Following these optimizations and architecture changes, further load testing concluded that these optimizations were not yet ready to handle the load the Texas A&M Health team expected in their production environment. This led to the Texas A&M Health team modifying and implementing a Visitor Prioritization solution initially proposed by AWS. Also known as a waiting room, this solution utilizes Lambda@Edge and the Amazon CloudFront content distribution network to manage and absorb the initial spike of traffic and limit the traffic to the application to make sure the known application capacity is not exhausted.

Better together

The REDCap COVID-19 Vaccine Administration solution is now able to handle 148,000 user requests over the initial five minute period, and the application stays available and highly responsive at all times. Tens of thousands of people scheduled COVID-19 vaccinations over the first month of widespread eligibility for the Brazos County Health Services Hub, and they reduced the reported error rates from users down to near zero.

Throughout this engagement, Texas A&M Health’s TAM worked to bring internal awareness to the criticality of the issues that the teams at Texas A&M (TAMU) were facing. TAMU’s solutions architect was able to do a number of architecture reviews and make recommendations for scaling instance size and best practices around auto scaling specific for events with known timeframes such as manual pre-scaling, as opposed to taking the time that would be implicit for their standard auto-scaling functions to operate.

Texas A&M Health director of IT and assistant chief information officer (CIO), Dr. Joshua Kissee, said “The collaboration between Texas A&M Health and our partners at AWS has been exceptional. When we faced a difficult situation that required immediate attention, it was then that our relationship was tested and I am very pleased that our AWS Partners not only rose to the occasion with us, but they even joined our actual go-live launch. It was a great experience for me as an IT leader to watch this symbiotic relationship take root and grow for the purpose of doing our part in the race to administer the COVID-19 vaccine.”

The REDCap COVID-19 Vaccine Administration Software, hosted in AWS, facilitated the registration and delivery of 98,552 doses of the Moderna vaccine as of May 17, 2021 through the Brazos County COVID-19 Vaccination Hub.

Advice and where to learn more

If you are planning a high-profile or high-impact launch, or expect a spike in load, reach out to AWS to discuss how we can help through engagements with our AWS Infrastructure Event Management (IEM) or AWS Well-Architected Reviews.

An AWS Well-Architected Review led by a solutions architect can further improve the overall stability and availability of the workload by making sure it’s architected following the AWS Well Architected Framework.

The AWS Well-Architected Framework helps cloud architects build secure, high-performing, resilient, and efficient infrastructure for their applications and workloads. The AWS Well-Architected Framework is based on five pillars: operational excellence, security, reliability, performance efficiency, and cost optimization. Learn more about AWS Well Architected Framework and conduct your own Well Architected Review using the AWS Well-Architected Tool in the AWS Management Console.

AWS IEM will help you assess operational readiness, identify and mitigate risks, and execute your event confidently with AWS experts by your side. The program is included in the Enterprise Support plan and is available to Business Support customers for an additional fee. Learn more about AWS Infrastructure Event Management (IEM). Plus, learn how AWS Premium Support could help your launch.

Subscribe to the AWS Public Sector Blog newsletter to get the latest in AWS tools, solutions, and innovations from the public sector delivered to your inbox, or contact us.