Avoid the Crunch, go Web-Scale

Over at TechCrunch, Mike Arrington recently shared some of his concerns about linking to up and coming new sites. Any site featured on TechCrunch will see a huge inflow of traffic when seemingly everyone the tech world visits it within the course of a couple of hours to check it out:

Theres a spike, and then most of the people never come back. Hopefully a few stick around, register and tell their friends, but building an application to scale to handle a TechCrunch post is a long term solution to a short term problem.

Before I could even respond, the Crunchback blog proposed theTechcrunch Reference Architecture. In his words:

Build using Amazon EC2 and S3.
Use a load-balanced architecture
Add EC2 nodes when you go live – as many as you can
Alert TechCrunch
Wait for mention (pray for mention)
2 days later start reducing nodes

It almost goes without saying that this is in alignment with our own thinking in this area. Instead of scaling in advance for traffic that may or may not materialize, we believe that developers should create a scalable architecture, host it on Amazon EC2, and then simply “turn the knob” (so to speak) when traffic surges. They pay for actual usage while those servers are active, and then simply turn that knob back down when the surge subsides. No fuss, no muss, and no rack full of servers that are sometimes running at capacity and at other times sitting idle.

— Jeff;

Updates:

1. TechCrunch reference architecture came from CrunchBack, not from Phil Wolff.
2. Lots of good conversation can be found in the comments.