Major Vodafone web outage? [2010 08 24]

Forums › One New Zealand (including Vodafone, WxC, Farmside) › Major Vodafone web outage? [2010 08 24]

1 | 2 | 3 | 4 | 5

2570 posts

Uber Geek

Trusted

#372229 24-Aug-2010 15:59

Zeon: We lost:
- Internet with ICONZ through Araneo
- Backups vodafone data sticks
- International connecitity at home via Xnet

So did the APE stay online during the outage? I couldn't get out on the APE (but probably related to Araneo/Vodafone more than anything) nor was there any international.

Yea. None of our peers went down. We saw a bit of a drop off in APE traffic, but nothing I'd call serious.

raab

262 posts

Ultimate Geek
Inactive user

#372231 24-Aug-2010 16:01

I think what Maurico is saying is that Vodafone were on their high horse about single points of failure on the XT network, yet they're no better with a single point of failure at APE.

muppet

2570 posts

Uber Geek

Trusted

#372232 24-Aug-2010 16:01

aperfectcircle: Isn't Vodafone the "reliable" network hmmm? Gotta love the excuses that it's "an hour outage".

Of their data network only. Caused by circumstances outside their control. That affected more than one provider.

Strange they didn't have some sort of backup/DR failover, but hardly comparable to XT's problems.

Beccara

1469 posts

Uber Geek

ID Verified

#372244 24-Aug-2010 16:20

A little apple and oranges here since XT was major voice outages and Vodafone was a minor Data outage. Had Vodafone's voice services gone down I would agree that it's a bit hypocritical

Most problems are the result of previous solutions...

All comment's I make are my own personal opinion and do not in any way, shape or form reflect the views of current or former employers unless specifically stated

Zippity

683 posts

Ultimate Geek

#372247 24-Aug-2010 16:25

What a great little cut throat world we live in.

When Telcom goes belly-up, the VF fans toss the $4it

When VF goes belly-up, the Telecom fans toss it :)

Who is keeping the score?

4 - 1??

graemeh

2078 posts

Uber Geek

#372249 24-Aug-2010 16:28

muppet:
aperfectcircle: Isn't Vodafone the "reliable" network hmmm? Gotta love the excuses that it's "an hour outage".

Of their data network only. Caused by circumstances outside their control. That affected more than one provider.

Strange they didn't have some sort of backup/DR failover, but hardly comparable to XT's problems.

Using that same logic then Telecom could say the issues with the XT network were caused by issues outside their control.

In both cases the backup/DR systems either didn't work or didn't kick in quick enough. The only differences here are the duration of the outage and the type of traffic affected.

michaelmurfy

meow
13257 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#372250 24-Aug-2010 16:31

Zippity: What a great little cut throat world we live in.

When Telcom goes belly-up, the VF fans toss the $4it

When VF goes belly-up, the Telecom fans toss it :)

Who is keeping the score?

4 - 1??

Something like that :P - Except people tend to complain about Vodafone quite a bit around here, more people seem to complain about Telecom. YOU CAN NOT WIN! >.<

The APE didn't go down during this outage, it was up the whole time, but as Muppit said, there was a drop of traffic. We were just running on UPS power for a hour or so :)

Michael Murphy | https://murfy.nz
Referral Links: Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.

Quic Broadband

Move to New Zealand's best fibre broadband service (affiliate link). Free setup code: R587125ERQ6VE. Note that to use Quic Broadband you must be comfortable with configuring your own router.

vexxxboy

4244 posts

Uber Geek

#372258 24-Aug-2010 16:42

full story from the herald,

"SkyCity was evacuated this afternoon as a precaution after smoke was seen coming from a plant room. Photo / NZPA

Vodafone internet users were cut off today after a power outage at SkyCity.

Smoke was seen coming from a plant room on the Federal Street side of the SkyCity complex this afternoon, so around 200 staff and patrons on the gaming floor and at the bars and restaurants were evacuated and the fire service called.

The SkyTower itself was not evacuated, but due to the power outage Vodafone and other web providers which transmit from the tower were cut off.

Vodafone spokesman Paul Brislen said as the complex had been evacuated, no one was allowed in to switch on a back-up generator to restore the connection.

Vodafone broadband and data users had no access for about an hour.

Services were now running as usual, he said.

SkyCity staff and patrons have now been allowed back into the building and technicians are working on finding out where the smoke came from"

Common sense is not as common as you think.

muppet

2570 posts

Uber Geek

Trusted

#372259 24-Aug-2010 16:43

graemeh: Using that same logic then Telecom could say the issues with the XT network were caused by issues outside their control.

Fair call. Personally I think a fire is much more serious and uncontrollable (force majeure) than any software glitch coded by your supplier, which could be found by testing and review and those sort of things. But you could argue the toss on that I guess.

graemeh: In both cases the backup/DR systems either didn't work or didn't kick in quick enough. The only differences here are the duration of the outage and the type of traffic affected.

I disagree. The difference as I see it is a full network outage (On XT, nothing worked) vs a partial outage (only Data). A large majority of customers wouldn't have noticed the outage at all.
But again, you could argue that if you're a data customer, XT vs VFNZ outages are equally impacting to you.
If it happens another 3 times though, I think they'll be mocked/pitied as much as XT was, because of the backup situation.

Audiophiles are such twits! They buy such pointless stuff: Gold plated cables, $2000 power cords. Idiots.

OOOHHHH HYPERFIBRE!

graemeh

2078 posts

Uber Geek

#372261 24-Aug-2010 16:48

muppet:
graemeh: Using that same logic then Telecom could say the issues with the XT network were caused by issues outside their control.

Fair call. Personally I think a fire is much more serious and uncontrollable (force majeure) than any software glitch coded by your supplier, which could be found by testing and review and those sort of things. But you could argue the toss on that I guess.

The issue is not the fire, the issue is that there was no effective backup in place.

muppet:
graemeh: In both cases the backup/DR systems either didn't work or didn't kick in quick enough. The only differences here are the duration of the outage and the type of traffic affected.

I disagree. The difference as I see it is a full network outage (On XT, nothing worked) vs a partial outage (only Data). A large majority of customers wouldn't have noticed the outage at all.
But again, you could argue that if you're a data customer, XT vs VFNZ outages are equally impacting to you.
If it happens another 3 times though, I think they'll be mocked/pitied as much as XT was, because of the backup situation.

XT was NOT a full network outage. My phone started working as soon as I got off the plane in Auckland, I just couldn't use it at home in Wellington to chase the taxi company

You are right though, the Vodafone outage affected less customers.

VFNZPaulBrislen

970 posts

Ultimate Geek

Trusted

#372291 24-Aug-2010 17:33

Hi all,

we're still investigating just what happened and why - the facility isn't directly operated by Vodafone (or any of the local telcos as I understand it) so we're trying to find out why our back-up generator didn't kick in directly that the first power feed was cut off. I've been told that the back-up supply did indeed kick in but was switched off to allow the fire service guys in. However, that's yet to be confirmed.

Please do not misconstrue this as me blaming anyone for anything - I'm simply sharing the information I have to hand.

Our internet access was cut for about 45-50 minutes and would have affected customers nationwide. Voice and TXT were not affected.

Our plan is to build additional redundancy into the system by opening a second centre at the Telecom Mayoral Drive exchange - that's still being put together.

Paul

Paul Brislen
Head of Corporate Communications
Vodafone

http://forum.vodafone.co.nz

lduncan

189 posts

Master Geek

#372340 24-Aug-2010 19:34

Spawneh:
an hour outage which was caused by a power surge/spike Vs multiple outages for multiple hours. I dont think it benefits this post trying to compare the two personally, as I think they are far different.

Yes, very different. I'd argue this Vodafone outage is far more embarrassing than XT's outages from a technical perspective. Honestly, this is the internet we're talking about. It was designed from the ground up to explicitly avoid localised failures taking out massive parts of the network.

Frankly it's pretty negligent that a large carrier such as Vodafone has a single point of failure like this in their IP network.

Voice networks are a very different beast, never really designed for failure in the same way IP networks have been. I don't know all the technical details behind the XT failures, but this Vodafone outage seems a total rookie mistake, trivial to avoid.

But who knows, I may be totally wrong, and it may have just been bad timing for Vodafone, maybe their alternate routes were down for some sort of maintenance or something ;-)

PenultimateHop

637 posts

Ultimate Geek

Trusted

#372501 25-Aug-2010 10:19

That is pretty poor - there is no excuse for being 100% dependent on the Sky Tower for Internet connectivity. It is a facility which is known and proven to be unreliable - particularly for power.

Also, the Sky Tower and APE are not 1:1 related: as muppet pointed out, APE is a L2 virtual instance on the Citylink PublicLAN infrastructure, and it appears anywhere that Citylink have service in Auckland, so it can easily survive an outage at Sky Tower. It also doesn't tend to have much to do with transit connectivity... Only peering.

dcsharp

15 posts

Geek

ID Verified

#372524 25-Aug-2010 10:51

I guess it looks even worse as it comes during the middle of several other issues Vodafone is having with Latency (over a month), Connection Speed issues, red network issues (apparently unrelated), email unreliability (this has been ongoing for months but far worse recently), dodgy usage meters and the billing Fiasco.

graemeh

2078 posts

Uber Geek

#372527 25-Aug-2010 10:56

PaulBrislen: Our plan is to build additional redundancy into the system by opening a second centre at the Telecom Mayoral Drive exchange - that's still being put together.

Thanks for letting us all know. I assume by now Telecom have gotten over their power shortages in Mayoral Drive :)

1 | 2 | 3 | 4 | 5