Rock & Roll Hall of Fame Preserves Rock Music History and Modernizes on AWS

The Rock & Roll Hall of Fame (Rock Hall) is a nonprofit organization on a mission to engage, teach, and inspire through the power of rock and roll. The Rock Hall is dedicated to making rock and roll artifacts spanning every era and genre of rock and roll accessible to audiences at its museum in Cleveland, Ohio, and across the world. The Rock Hall’s archival collections include millions of documents, handwritten lyric and song manuscripts, correspondence, contracts, photographs, posters, audio and video concert recordings, recorded interviews, and video files chronicling the evolution of rock music. 

In 2010 the Rock Hall established a Library & Archives, which includes onsite digitization and conservation labs, processing spaces, and a large public reading room for onsite visitors, classes, researchers, documentarians, and historians. The Library & Archives collects, preserves, and provides access to the Rock Hall’s archival materials. The archival collections focus on such luminaries as record executives Ahmet Ertegun and Clive Davis; disc jockey Alan Freed; performers Del Shannon, Curtis Mayfield, Otis Redding, Art Garfunkel, and Scotty Moore; recording studios FAME, Gold Star, Hyde Street, and Ocean Way; and record companies Atlantic, Sire, Specialty, Bloodshot, and Kill Rock Stars. In addition to archival materials documenting other aspects of the music business, the institutional records of the Rock & Roll Hall of Fame also reside at the Library & Archives, alongside editorial files from Rolling Stone magazine. Most collections are physical, but every week more digital content is donated and created, and portions of the Rock Hall’s physical collections are digitized for preservation and access.

Until recently, the Rock Hall used a variety of technologies to store its files. Some of its storage technologies were years or even decades old, including Linear Tape-Open (LTO) tape storage and offsite backup to preserve its digital files of Hall of Fame Induction ceremonies, B-roll, oral histories, artist interviews from program series like Songwriters to Soundmen, research symposiums from Rock Hall Honors, and live concerts featuring unique performances by the most significant musicians of our time: Aretha Franklin, Paul McCartney, Led Zeppelin, Elton John, Bruce Springsteen, Prince, and many more. These storage mediums were outdated, expensive to maintain, and prone to failure. “We had unique and priceless files on media that had become unmanageable and unsustainable for our core mission of preserving rock history,” says Tim Parnin, vice president of digital technology at the Rock Hall.

The Rock Hall launched a new strategic plan in 2015 focused on encouraging innovation, improving internal efficiencies, and enhancing the museum’s learning experiences for visitors through exhibits, programs, outreach, and an online presence. The Rock Hall wanted to set up a centralized digital media repository, with highly configurable user access, capable of handling multiple types of assets with varying taxonomies and metadata structures that was not reliant on dated technologies and would prepare the Rock Hall for the digital age. “We needed a storage platform that could handle the sheer volume of our assets, as well as a rugged, expandable solution we weren’t going to outgrow—one we could race into the future with,” says Parnin.

In 2018, Rock Hall staff evaluated the organization’s underlying technical infrastructure and decided to take advantage of Amazon Web Services (AWS). To preserve the museum’s digital media, it first deployed Amazon Simple Storage Service (Amazon S3) and then Amazon S3 Glacier Deep Archive storage for its scalable, secure, and low-cost cloud storage.

Once the Rock Hall had a foundation with Amazon S3 in place for storage of its archives, the next step was to create a modern digital asset management (DAM) platform. “Traditional DAM systems don't play well with large, high-resolution video files. Historically, different groups within our organization were creating and storing their own silos of content. We needed to get our digital files situated in one easily accessible location, and then we needed the strength and scalability of AWS to help us store and manage those files,” says Parnin. 

The Rock Hall built a comprehensive DAM on AWS that would support the organization’s video assets, some of which are 2–3 terabytes in size. This enterprise DAM solution increases the cross-departmental utility of media workflows while integrating multiple Rock Hall metadata repositories. The DAM handles the creation, storage, and streaming of high quality proxy video files; allowing Rock Hall users to watch, annotate, create shot lists, sub-clips and partially restore without touching the master asset. This also helps with preservation workflows through automatic inspection and transcode to the prescribed preservation master file format via the AWS Elemental MediaConvert tools for publication and distribution. With Amazon S3 storage classes, the Rock Hall has been able to optimize costs based on the access needs of specific content pieces.

As the DAM system was being implemented, the Rock Hall was undertaking a project to modernize its aging LTO storage and offsite backup for the preservation of large digital files, by moving the files to the cloud. Many of the LTO tapes were not easily accessible due to hardware and software failures and onsite storage limitations. Parnin says, “As the tech landscape progressed from the non-digital to digital, our archival storage system had become unmanageable and unsustainable.”

Using Amazon S3 and S3 Glacier Deep Archive provided the Rock Hall with the confidence that its digital media would be preserved and easily accessible at an affordable price. However, the Rock Hall still needed to recover the data on the LTO tapes. Working with AWS, the project team ingested the files into S3 Glacier Deep Archive via six AWS Snowball Edge Storage Optimized devices. Using AWS Snowball Edge helped to address common challenges with large-scale data transfers, including high network costs, long transfer times, and security concerns. 

The Rock Hall worked with Tape Ark and its strategic partner Seagate Technology (Seagate Powered by Tape Ark) to extract all the data from the LTO tapes and load it onto the AWS Snowball Edge devices. Tape Ark then sent the Snowball devices loaded with data back to AWS, and the data went right into the Rock Hall’s Amazon S3 bucket. Once the Rock Hall’s digital media was in Amazon S3 on the AWS Cloud, Amazon S3 lifecycle policies were set up to automatically move the media files into S3 Glacier Deep Archive to help optimize storage costs for files that were rarely accessed. For the Rock Hall, the process was effortless, with Tape Ark managing the end-to-end migration.

Using AWS services with the help of its partners, the Rock Hall initially recovered and preserved 2,000 files and 300 TB of its most valued digital video assets. With the expertise of Chromata Solutions, a strategic consulting and program management company that specializes in DAM and digital supply chain, and the creation of the Rock Hall DAM, the Rock Hall is now working to preserve an additional 650,000 files. 

After migrating to the modern DAM platform, the Rock Hall expanded search functionality, built-in flexibility, and tiered storage management. As a result, the organization has virtually no limitations on file or library size. The new platform also improved preservation workflows, business process management workflow automation, and integrations to leading third-party tools such as Adobe Creative Cloud—all in a cost-effective, cloud-based infrastructure.

At the same time, Amazon S3 and S3 Glacier Deep Archive have helped improve staff efficiency. “The speed of accessing the files is great,” says Jennie Thomas, director of archives at the Rock Hall. “Before, we would have to download a preservation file overnight and hope it worked by the time we came back in the morning. Now we can download that same file in about 15 minutes.”

Staff can also rest assured that the Rock Hall’s unique assets are safe. “In addition to having all our assets in S3 Glacier Deep Archive, knowing that AWS is making duplicate copies and has disaster planning and recovery taken care of is a huge load off of my mind,” says Thomas. “The fact that we were able to realize all these benefits and still save significant costs with S3 Glacier Deep Archive, compared to the cost and overhead of managing our own tape library, helped us shift resources to our strategic goals.”

With the Rock Hall’s assets preserved and safe, the next step was to enable staff to access and use assets in their day-to-day business quickly and efficiently. With a rich archive of artists and figures, facial training and recognition was critical for the Rock Hall’s use cases. “The files we created over the years didn't have embedded descriptive metadata,” says Thomas. “Archives staff were able to research and identify content from file naming conventions, but with us acting as default gatekeepers to the content, this limited how quickly and easily staff across the institution could obtain access to what they needed.”

To address this issue, the Rock Hall partnered with GrayMeta, an AWS Partner Network (APN) Advanced Technology Partner. The Rock Hall DAM now automatically interacts with GrayMeta and AWS when new assets are added. Using the GrayMeta platform, the Rock Hall team uses the technical and machine learning metadata to understand their assets and instantly search through images, audio, and video. As a result, the archives staff spends less time searching for relevant content. Thomas says, “Because the DAM system is working with our collection management system (CMS) on AWS, it pulls the descriptive metadata from the CMS that we've already created for these assets so we don’t have to do it again in another system. A lot of systems don't have the capability to talk to each other the way ours now do.”

The system also put Rock Hall employees in a better position to continue their work despite shutdown-related, work-from-home requirements. “You can see the assets pouring into the system, which means people working from home are having an easier time accessing content when they need it,” says Thomas.

The Rock Hall can now digitize, preserve, and instantly search a wide variety of content. “The public, as well as our staff, will benefit from this,” says Parnin. “We’re improving internal efficiencies and making hundreds of thousands of priceless, unique assets searchable and discoverable. In addition, we now have reassurance that our archives are protected and future generations from around the world can enjoy the history of rock and roll.”

To learn more, visit thinkwithwp.com/s3.