In the last two years we have been linked from some heavy-weights in the on-line world, including having two articles showing on the Slashdot frontpage at the same time. But nothing has prepared us for what happened on 1 March. Suddenly one small article about a new mobile phone was picked up by Yahoo! Japan and posted on their news page. This was during the business hours in Japan, around 4pm in New Zealand.
This shows the power of branding. The article in question is about a new Sony Ericsson mobile phone with the "Walkman" brand in it. Put "Sony" and "Walkman" in one article about a new product, and you have a mix that can potentially drive masses of Japanese readers eager to know more - and bring down a database server.
During the next two hours, the Microsoft IIS and Microsoft SQL services hosting Geekzone had to provide resources for a traffic load that was more than 15 times our average number of pages served per hour. This wasn't spread over the day, but concentrated in a shorter period of time. That single article was viewed by more than 120,000 users in a couple of hours - in addition to our normal traffic.
During this time I had to contact my hosting provider (Intellihost, a service from Iconz) to arrange for the SQL server to be restarted, since it had stopped responding to the ASP engine and to the SQL Query Analyzer tool - such was the volume of transactions going on! I have to say I always had a good service from the support folks working at Intellihost. They also seem to be always in good mood.
Things were under control after a few minutes, and showed the importance to prepare a plan to such events. I am not sure what the next steps are. Most likely someone will suggest a collocation service, database fine tunning - and these options are not out of the cards. The problem is cost going up with hardware expense, environment hosting and of course the server software. And there's the problem of "average". This event is not the average traffic, just a peak.
How we manage this? Just let this happen, or prepare for a huge traffic - that may never come again?