Telecom broadband and websites seem to be down

Forums › Spark New Zealand (including Skinny and BigPipe) › Telecom broadband and websites seem to be down

1 | 2 | 3 | 4 | 5 | 6 | 7

BDFL - Memuneh
79133 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#181402 1-Dec-2008 17:46

Simonm: I too would like to know why it went down.

Sharesies

Trade NZ and US shares and funds with Sharesies (affiliate link).

w2krules

488 posts

Ultimate Geek

#181403 1-Dec-2008 17:47

Zippity: Geez - talk about touchy.

Telecom has done SFA to endow itself to many users lately

IMO, that's a bit uncalled for. I think everyone would agree that for many years they've invested minimally in technology, but there are real signs that this is changing, e.g. their new mobile network and the cabinetisation (is that a word?) programme.

I was a geek before the word was invented!

FredDag

206 posts

Master Geek
Inactive user

#181423 1-Dec-2008 18:44

I would bet that it was a combination of faults that took out the redundency.

Like when the southern cross cable was being repaired on one link and then a ships anchor took out another link.

Or when the fibre link in the south Island was taken out by a truck colliding with a rail underpass then caught fire burning the fibre and.. (not sure this was the truth) a rat chewed through the alternative 'redundent' loop path the other side of the south island.

perhaps it was all 4 things in the Auckland exchange... a rat driving a burning truck hitting a ship dragging its anchor in the data centre?

Fred

ajobbins

5052 posts

Uber Geek

Trusted

#181429 1-Dec-2008 18:58

Outages happen. A similar thing happened to Xnet a while back. Don't worry - you will get over it.

This will of course be a learning experience for Telecom, and perhaps other ISP's.

It happened late on Friday night and was resolved before most people were awake on Saturday morning, so it was really only out for a few 'waking' hours.

While it would be nice to know the root cause for interests sake, it's not going to change anything.

insane

3223 posts

Uber Geek

ID Verified

Trusted

#181432 1-Dec-2008 19:05

NealR: Just curious to know whether knowing the route cause changes anything. Or would you rather be more interested to know that everyone on the responsible organisation is doing everything possible (incl spending $$) to ensure it does not happen again...just curious.

ISP etc are always spending $$ on infrastruture ,some even using tax payer money (orcon).

Personally I'm more interested in what happened but I'm also aware that You (telecom) would have to be very carefull about what you say failed as it may negatively affect that partner/supplier of equipment and therefore damage their reputaion if the gear was not used within spec or something along those lines.

I'm not expecting a press release saying that a 11KV power feed to the MDR exchange was overloaded which caused power to the primary UPS to fail which inpacted the DNS cluster. Extra power demands on the B power feed lead to a short whch caused a fire which caused the diesel fuel tanks to ignite which caused an emergency power cut sequence to initiate. I'm not after that kind of detail...

Perhaps something like "a power issue at one of aucklands main exchanges caused telecom system management platforms to fail which caused a major outage across nearly the entire network as systems fell over in a chain like manner" - would be more than enough

Bung

6394 posts

Uber Geek

Subscriber

#181434 1-Dec-2008 19:21

insane:
Perhaps something like "a power issue at one of aucklands main exchanges caused telecom system management platforms to fail which caused a major outage across nearly the entire network as systems fell over in a chain like manner" - would be more than enough

No, it wouldn't. Last year there was another major outage caused by a fault during the weekly testing of the power system. A statement like you suggest would mean they learned nothing.

FredDag

206 posts

Master Geek
Inactive user

#181464 1-Dec-2008 21:14

Insane "a 11KV power feed to the MDR exchange was overloaded which caused power to the primary UPS to fail which impacted the DNS cluster. Extra power demands on the B power feed lead to a short which caused a fire which caused the diesel fuel tanks to ignite which caused an emergency power cut sequence to initiate."

It was a 6 engine call out with a response time of 9 minutes. Fire crews closed off the supply from the diesel tanks to the burning generator and hosed the tanks. Meanwhile crew with breathing apparatus entered the data centre to close off the FM200 system....

(someone going to run with this?)

Fred

insane

3223 posts

Uber Geek

ID Verified

Trusted

#181477 1-Dec-2008 22:29

Bung:
insane:
Perhaps something like "a power issue at one of aucklands main exchanges caused telecom system management platforms to fail which caused a major outage across nearly the entire network as systems fell over in a chain like manner" - would be more than enough

No, it wouldn't. Last year there was another major outage caused by a fault during the weekly testing of the power system. A statement like you suggest would mean they learned nothing.

I wasnt asking a question there... thats what I heard really hppened which i guess in turn was caused by the main outage.

BarTender

3598 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#181794 3-Dec-2008 14:49

Sounds like what happened last year: http://www.Telecom.co.nz/?q=node/89

And a few years ago as well: http://www.geekzone.co.nz/forums.asp?ForumId=49&TopicId=7879

Simonm

181 posts

Master Geek

#181797 3-Dec-2008 15:05

I dont really care what failed. Things fail all the time, but the infrastructure has to be designed to cope with that!
I want to know why it took so long to fix!

n00dy

482 posts

Ultimate Geek

Lifetime subscriber

#181822 3-Dec-2008 16:40

An email came my way indicating "Mayoral Drive" had a power outage, exact wording " the night on Friday when we lost power to Mayoral Drive. Yahoo Xtra and all of our call centres went down as a result, together with numerous other services"

However I am courious to know why backups didnt kick in, and exactly why mayoral drive can have such an impact of just about everything?

BlakJak

1249 posts

Uber Geek

Trusted

#181830 3-Dec-2008 17:11

MDR is their main datacentre for a variety of services, Telephony and Data related.
It has quite a substantial backup power supply system (during the Auckland Power Crisis i'm pretty sure they were feeding power back into the Grid at a fairly decent level) so one has to presume that if power was the issue, the backup system failed to work as advertised. And, no doubt, heads are in the process of rolling as a result.

No signature to see here, move along...

freitasm

BDFL - Memuneh
79133 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#182205 5-Dec-2008 08:45

An interesting article on NBR about the people involved and Telecom's offer to pay a compensation on a case-by-case basis.

Bung

6394 posts

Uber Geek

Subscriber

#182212 5-Dec-2008 08:57

BlakJak: MDR is their main datacentre for a variety of services, Telephony and Data related.

It has quite a substantial backup power supply system (during the Auckland Power Crisis i'm pretty sure they were feeding power back into the Grid at a fairly decent level) so one has to presume that if power was the issue, the backup system failed to work as advertised. And, no doubt, heads are in the process of rolling as a result.

Substantial backup power system or a collection of systems of varying quality?

The telephony side rarely falls over and it was probably those engine sets that were feeding the grid.

BlakJak

1249 posts

Uber Geek

Trusted

#182382 5-Dec-2008 21:47

You'd have to ask someone who knows more than I - back in my Telecom days I only had cause to visit MDR briefly, and though i've seen inside some of their Datacentre levels I never got to see the power gears. Only know it by reputation and by what i've been told...

No signature to see here, move along...

1 | 2 | 3 | 4 | 5 | 6 | 7