Caltech Uses Amazon EFS to Automate Academic Computing File Management

The California Institute of Technology (Caltech) is a private research university located in Pasadena, California. Often ranked as one of the world’s top-10 universities, Caltech is recognized for its natural science and engineering research.

The central services team of the university’s Academic Computing group, which consists of just three IT personnel, supports the computing needs of more than 300 faculty and administrators. These internal customers rely on the Academic Computing team to host more than 200 public-facing applications and provide high-performance computing (HPC) environments.

As more and more customers requested new applications and other workloads, Caltech’s team struggled to quickly fulfill requests, due to the limitations of its on-premises systems. “We had started to virtualize our environment, and we were using Docker containers to simplify things,” says Aaron McKinnon, associate director of academic computing at Caltech. “However, virtual servers were still limited in capacity. Also, config files and website data were stored in the containers, and we tried to separate the config files and data with different solutions to make application upgrades faster. But this still wasn’t fast enough. We wanted the ability to destroy and create containers on demand to accelerate application upgrades and deployments for faculty and researchers.”

To solve its problems, the department chose to broaden its use of the Amazon Web Services (AWS) Cloud. “We were already using AWS extensively throughout campus, supporting everything from data storage to HPC,” says McKinnon. The Caltech Academic Computing team decided to implement Amazon Elastic File System (Amazon EFS) for its Network File System (NFS) requirement.

Using Amazon EFS, a fully managed elastic file storage service, Caltech moved its Docker containers and application environments to the AWS Cloud without having to alter code. The team can now build generic Docker containers and point all config files to Amazon EFS without altering the Docker containers. “We wanted a robust and affordable solution that enabled automation, which is critical for us because we’re a very small team. Amazon EFS fits our needs perfectly,” McKinnon says. “We didn’t have the resources to build an NFS solution to host our config and data systems. Amazon EFS works as a central repository for everything we deploy and requires very little effort to manage.”

The team also uses Amazon Elastic Container Service (Amazon ECS), an orchestration service for Docker containers, which gives Caltech the ability to run and scale containerized applications on AWS. “Relying on Amazon EFS and Amazon ECS, we can decouple our storage from our application containers so we can scale dynamically and make containers disposable,” says Dan Caballero, senior systems administrator for Caltech. “This solution just works, and it’s scalable. I can integrate as many Docker containers into it as I need to without any reconfiguration of the storage layer.”

Amazon EFS gives the lean Caltech team a centralized file system that is robust and easy to manage, enabling it to respond to the needs of its customers more quickly. The team can now set up new application environments in two hours instead of two days. As an example, the team responded quickly when researchers needed to create a new website announcing a large donation to the Resnick Sustainability Institute at Caltech. “We set up a new website in a few hours to announce an important donation. That task would have taken at least several days using our previous environment. That wasn’t acceptable, because the announcement was very time-sensitive,” says McKinnon. “With our small staff, we wouldn’t have been able to create a site like this without the agility and scalability of Amazon EFS and Amazon ECS.”

For the Caltech Academic Computing team, Amazon EFS has become an essential part of its agile application deployment processes. “We recently deployed two additional faculty websites, and the process was seamless,” says Caballero. “Instead of manually checking files into a config management system and waiting until deployment was finished like we used to do, we simply relied on Amazon EFS to help us automate rolling system updates, without interrupting the service at all.“ As a result, researchers relying on Caltech’s application environments no longer need to worry about system downtime that could potentially limit their ability to run research workloads in a timely fashion.

The Caltech Academic Computing team also has more agility in its testing processes. “If we need to deploy and test a new version of a development tool or a new Docker feature, we can stand up an entirely new Amazon EFS solution with new Docker containers without impacting the data,” says McKinnon. “We’re able to test the new feature with live data and configs and not actually change anything. We did this when we created the new Resnick announcement website, which allowed us to test everything without deploying new servers.”

In addition, Caltech reduced its total number of on-premises applications and supporting systems from more than 500 to 150 by moving to the AWS Cloud. “Caltech is very space-constrained, and power is expensive in California,” says McKinnon. “We can help solve those problems by turning server rooms into space the Institute can use.”

Caltech is no longer held back by the constraints of its on-premises environment. “Using AWS helps us fulfill the promise of virtualization,” says Caballero. “We were still confined by CPU and memory limitations after we started moving from physical to virtual servers. With Amazon EFS and Amazon ECS, we’re aggregating CPUs across a handful of CPU instances. That removes the capacity and scalability problems we had before, and we no longer have any limits on what we can do.”

To learn more, visit thinkwithwp.com/efs.