Interesting way to use EC2, SQS, S3 together : Webmail.us

Backing up data to Amazon’s Secure Storage S3 has been the hot-favorite-use-case for many companies. Webmail.us, the email hosting company, who is currently serving more than 27,000 businesses and hosting more than 350,000 paid mailboxes and growing by close to 10% per month, is now using amazon S3 for backing up encrypted email data.

Smart Guys at Webmail took AWS to next level by using all these services together. Its very interesting to know how creatively they used EC2, SQS and S3 together to scale out, save time and money. They used SQS to queue up their “cleanup” jobs and EC2 to process the jobs concurrently.

Webmail has a backup policy to keep user data on Amazon S3 for 14 days. After 14 days, the magic happens :

Every night, 35,000 “cleanup” jobs to-be-processed gets queued up in SQS. Jobs are fed to hungry EC2 instances one by one. Using the power of parallelism, 3,500 jobs gets processed over 5 EC2 instances in less than 40 minutes. They save 160 minutes by using 5 instances (imagine if they launch 10 EC2 instances!). Plus, there are two more advantages :

• Spawning more EC2 instances will allow them to scale out and process a higher number of jobs (in future) in the same amount of time.
• Cleanup happens at a remote location without sucking up any of their server’s bandwidth

— Jin

Modified 2/1/2021 – In an effort to ensure a great experience, expired links in this post have been updated or removed from the original post.