February 03, 2010 by Julian Pscheid
You might have already heard of the "digg effect". Digg.com is a popular social link sharing site, which allows users to contribute hot websites, images, and articles to the website, upon which other users can vote on that entry until the most popular ones show up on the website for the world to see. A fairly well known phenomenon in blogger circles is called the "digg effect", which can drive so much traffic to your web server it has the potential to make your website a huge success or it will simply just crumble under the load.
Today we launched a micro-site for Electronic Arts with our partners at Wieden+Kennedy. The site features downloadable materials relating to the upcoming Dante's Inferno game. In order to access the materials, visitors needed to find 6 passwords posted in the source code of unknown websites (one of those sites being digg itself). Through smart marketing the site was already in the digg upcoming topics before it was officially launched (only a placeholder page was visible at that point). At 2pm we switched the site live and it only took minutes before visitors started collaborating to find the 6 passwords and post their findings in the digg comments section. Traffic started increasing on the site immediately.
Being accustomed to high-volume spikes in our website's traffic we optimize all of our sites to handle large simultaneous amounts of visitors. This site too was optimized for a large amount or concurrent visitors, and throughout the day the server's CPU and memory load remained normal. However as the campaign went viral one factor became a concern: bandwidth.
Did I mention that the downloadable prize package consisted of a 190mb downloadable zip file?
As more and more users started filling in the passwords and downloading the large zip file, the article on digg got bumped into the top 10 featured articles on the homepage--and that was where we quickly learned the true value of having assets in a cloud environment. By 3pm the data bandwidth had far surpassed what any one single server could offer. The server was serving several hundred simultaneous downloads of the same 190mb file.
To keep the servers online, we immediately decided to move the large zip file into a content distribution network (CDN). A CDN hosts files on distributed servers across the world for you, and only charges you for each gigabyte that your visitors download. While larger CDN services such as Akamai can require a lengthy and expensive setup process, there are several more nimble players out there. Unfortunately our first choice, Amazon Simple Storage Solution, wasn't accepting new account registrations due to a bug in their sign-up process. Our second choice, Rackspace Cloud, required new sign-ups to go through a manual verification process. We had to move on to find another service to keep up with the success of the campaign.
Around 4pm we found our 3rd and unexpected solution on Google: SimpleCDN. I was able to set up an account with them in 2 minutes and we easily uploaded the zip file via FTP to their network. After a quick PayPal money transfer, we were ready to redirect all zip file download traffic from our server to the SimpleCDN network, immediately freeing up the bandwidth on our server network. Within minutes of SimpleCDN being online we were serving over a 1,000 simultaneous downloads, all at blazing speeds.
The unlikely hero or the hour turned out to be SimpleCDN. Within a few hours of going online they already had served over 20 gigabytes of zip file to Dante's Inferno website visitors, effectively freeing our servers up to handle website requests instead of large file downloads.
Moving forward I recommend anyone serving large files to keep in mind CDN services such as SimpleCDN, which help avoid not only bandwidth costs, but also unwanted surprise outages due to bandwidth overload. Especially when you consider the "digg effect".
Comments:
Hell is nigh! How strangely apropos.
Benjamin - February 4, 2010 09:02 AM