I got on the front page of digg! Ryan Hadley

And all I got was a day full of stress. OK, that’s a lie. I learned a lot too.

We weren’t ready here, from an IT point of view, for getting that much traffic all at once. I personally wasn’t ready either.

These times are in Mountain time. Around 10:00 AM (Lunch time for EST Digg users) the traffic really ramped up as the post was #1 in upcoming hot news for linux. The server was fine with this rush. Around 11:00 AM the post magically popped to the front page of digg. The server was not OK at this point. We were completely black to all requests from 11:05 until 11:15.

Quite a panic hit. Our first thought was to rsync the data over to another server and load balance. This is when we realized how much we were not prepared for this:

1. blog.indigio.com shares an IP with some of our customers sites
2. The load balancer we use does not make splitting traffic based off hostname (like a VirtHost in apache) very easy at all
3. We didn’t have any servers adequate for the task. They were either heavily used for other things or didn’t have proper apache/php installs to get running very quickly.

It was crazy watching this server hit load averages of 50. Everything was waiting on disk and kswapd0 was the top on the process list. Since we had no quick way to scale, I tried redirection. First I wrapped the wordpress index.php like so:

/* Short and sweet */
if ( $_SERVER['REQUEST_URI'] == '/index.php/2008/02/24/re-windows-vista-vs-linux/' )
{
header('Location:
http://blog.indigio.com.nyud.net:8090/index.php/2008/02/24/re-windows-vista-vs-linux/');
} else {
define('WP_USE_THEMES', true);
require('./wp-blog-header.php');
}

This helped a ton. At least other sites and other pages on the blog could load, however it was all very very slow. So my next plan of attack was skipping calling the file at all. I put in to apache’s conf:

Redirect /index.php/2008/02/24/re-windows-vista-vs-linux/
http://blog.indigio.com.nyud.net:8090/index.php/2008/02/24/re-windows-vista-vs-linux/

This helped a ton more. You can see that analog didn’t record all these 302’s in the report above. At 4:00 PM I removed the redirect and analog picked up the tail end of the traffic. The biggest benefit here had to have been that it did not have to go to disk, compile/run the php, send back the header. All it did was immediately respond back with a 302.

We’re working on separating each site served off of that box, and making sure we have identical environments ready for all the sites we host. Our goal is to be able to respond quicker and more appropriately to huge bursts in traffic.

Personally, I wasn’t ready either. Distinct hosts served yesterday: 32,615. So that’s probably around 32,615 people who read a personal opinion post that I wrote. Talk about overwhelming.

I’ve decided NOT to ever read any of the posts in the digg thread, I’m tired of being called names. Some of the posts to this blog itself were bad enough (all deleted). Why do people have to resort to personal attacks because they don’t agree with your opinion? And I hate it when people just say things like “you didn’t try hard enough” or “what your trying to make windows do isn’t what windows does” and the like. No substance, nothing I can learn from.

I love the posts that disagree with me and provide something new. A new tool to check out, a new way to try something.

But overall I kinda feel exposed. It’s a strange feeling and I’m not sure if I ever want a personal opinion to get 30,000 readers ever again. I enjoy writing, and I used to think that I’d love to someday write a book. Now I’m not so sure… maybe the trick is to just not have such immediate feedback/measurement of your writing. Especially from an audience that has some very vocal users who fail at being a nice human being.

• 6 Comments
• Posted In Engineering, QA, IT