XT Network down (outage 14 DEC 2009)?

Forums › Spark New Zealand (including Skinny and BigPipe) › XT Network down (outage 14 DEC 2009)?

1 | ... | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14

675 posts

Ultimate Geek

Trusted

#282397 14-Dec-2009 17:24

freitasm:
NealR: Just know that the restoration of todays service loss has been the focus of a very large number of people from all around the world.

Nobody wanted this to happen, nobody planned for this to happen, nobody "messed up" to cause this to happen, we don't have a "fundemental flaw" in our architecture and nobody was wasting time to resolve this issue. This was a serious event that could not have been planned for or tested around to develop quick and simple restoration plans.

Right now all I can do is appologise for the service interuption and praise the engineers who worked out the problem and resolved it.

Until you have been an engineer of a very complex technology that is responsible for the running of a large proportion of NZ businesses, as well as millions of dollars of revenue which when lost affects the pay packets of thousands of people, you cannot understand the preasure today's engineers had to deal with. Whether you like Telecom or it's suppliers, or it's technology choices, or the services it provides does not remove the fact that the engineers who resolved today's problems deserve respect.

Word!

QFT!

peteinakl

22 posts

Geek

#282398 14-Dec-2009 17:34

Ship happens, it's how we recover and take responsability.

I was floored today to be getting emails from [in a round about kind of way] Chris Quin, showing a commitment to not only resolving the issue, but taking responsability.

Great recovery! (But hopefully the learnings mean it won't happen again)

freitasm

BDFL - Memuneh
80661 posts

Uber Geek
+1 received by user: 41081

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#282399 14-Dec-2009 17:40

peteinakl: Ship happens, it's how we recover and take responsability.

I used this in my blog post about today:

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.

gingereejit

89 posts

Master Geek
+1 received by user: 5

#282414 14-Dec-2009 18:16

NealR: Just know that the restoration of todays service loss has been the focus of a very large number of people from all around the world.

Nobody wanted this to happen, nobody planned for this to happen, nobody "messed up" to cause this to happen, we don't have a "fundemental flaw" in our architecture and nobody was wasting time to resolve this issue. This was a serious event that could not have been planned for or tested around to develop quick and simple restoration plans.

Right now all I can do is appologise for the service interuption and praise the engineers who worked out the problem and resolved it.

Until you have been an engineer of a very complex technology that is responsible for the running of a large proportion of NZ businesses, as well as millions of dollars of revenue which when lost affects the pay packets of thousands of people, you cannot understand the preasure today's engineers had to deal with. Whether you like Telecom or it's suppliers, or it's technology choices, or the services it provides does not remove the fact that the engineers who resolved today's problems deserve respect.

What Neal said.

I think TNZ are actually doing a very good job on the comms side, by having teams and individuals on forums such as this and twitter - it's appropriate and modern mode of communication.

ResponseMediaNZ

518 posts

Ultimate Geek
+1 received by user: 196

ID Verified

Trusted

#282419 14-Dec-2009 18:22

NealR: Just know that the restoration of todays service loss has been the focus of a very large number of people from all around the world.

Nobody wanted this to happen, nobody planned for this to happen, nobody "messed up" to cause this to happen, we don't have a "fundemental flaw" in our architecture and nobody was wasting time to resolve this issue. This was a serious event that could not have been planned for or tested around to develop quick and simple restoration plans.

Right now all I can do is appologise for the service interuption and praise the engineers who worked out the problem and resolved it.

Until you have been an engineer of a very complex technology that is responsible for the running of a large proportion of NZ businesses, as well as millions of dollars of revenue which when lost affects the pay packets of thousands of people, you cannot understand the preasure today's engineers had to deal with. Whether you like Telecom or it's suppliers, or it's technology choices, or the services it provides does not remove the fact that the engineers who resolved today's problems deserve respect.

Nice,

its easy for people to sit here on their computers and flame the big corp for having an outage, I know there was a huge amount of work behind the scenes and I take my hat off for the team that did restore service, they worked there butts off to get it back up and running, also the front line teams thats delt with all the customers that had issues.

Like Neal, All we can do is appolgise for the service interuption and hope people can see that we did do our best to restore service.

My 2cents

naggyman

697 posts

Ultimate Geek
+1 received by user: 86

#282421 14-Dec-2009 18:28

underwatervrg:
its easy for people to sit here on their computers and flame the big corp for having an outage, I know there was a huge amount of work behind the scenes and I take my hat off for the team that did restore service, they worked there butts off to get it back up and running, also the front line teams thats delt with all the customers that had issues.

Like Neal, All we can do is appolgise for the service interuption and hope people can see that we did do our best to restore service.

My 2cents

+1 amen.

Morgan French-Stagg

morgan.french.net.nz

Apple TV

Stream your favourite shows now on Apple TV (affiliate link).

Tel69

Tel69
261 posts

Ultimate Geek
+1 received by user: 4

Trusted

Lifetime subscriber

#282431 14-Dec-2009 18:41

NealR: Just know that the restoration of todays service loss has been the focus of a very large number of people from all around the world.

Nobody wanted this to happen, nobody planned for this to happen, nobody "messed up" to cause this to happen, we don't have a "fundemental flaw" in our architecture and nobody was wasting time to resolve this issue. This was a serious event that could not have been planned for or tested around to develop quick and simple restoration plans.

Right now all I can do is appologise for the service interuption and praise the engineers who worked out the problem and resolved it.

Until you have been an engineer of a very complex technology that is responsible for the running of a large proportion of NZ businesses, as well as millions of dollars of revenue which when lost affects the pay packets of thousands of people, you cannot understand the preasure today's engineers had to deal with. Whether you like Telecom or it's suppliers, or it's technology choices, or the services it provides does not remove the fact that the engineers who resolved today's problems deserve respect.

Totally agreed

exportgoldman

1202 posts

Uber Geek
+1 received by user: 3

Trusted

#282433 14-Dec-2009 18:43

NealR: Just know that the restoration of todays service loss has been the focus of a very large number of people from all around the world.

Nobody wanted this to happen, nobody planned for this to happen, nobody "messed up" to cause this to happen, we don't have a "fundemental flaw" in our architecture and nobody was wasting time to resolve this issue. This was a serious event that could not have been planned for or tested around to develop quick and simple restoration plans.

Right now all I can do is appologise for the service interuption and praise the engineers who worked out the problem and resolved it.

Until you have been an engineer of a very complex technology that is responsible for the running of a large proportion of NZ businesses, as well as millions of dollars of revenue which when lost affects the pay packets of thousands of people, you cannot understand the preasure today's engineers had to deal with. Whether you like Telecom or it's suppliers, or it's technology choices, or the services it provides does not remove the fact that the engineers who resolved today's problems deserve respect.

First off I would like to say a word of thanks to the faceless Telecom engineers which were sweating raw emotion today... Every engineer worth his salt knows that blind fear when the big important thing is broken in ways unknown.

My understanding (and correct me if I'm wrong) is that although Radio Network Controllers (RNC's) have hotswappable and redundant parts you cannot have hot spare or load balanced RNC's, so if a RNC goes down all the cell sites (node B's) off that site are down until the RNC comes back online or is rebuilt/replaced.

So how come Vodafone with it's 'inferior' network has 6 Radio Network Controllers (RNC's) in use for their 3G network so when a RNC outage occurs (and they do - both Vodafone and Telecom have had RNC failures in the last 12 months) Vodafone's RNC failures affect less customers than Telecom's, and Vodafone can move Cell Site's onto different RNC's if one RNC is 'running hot' so doesn't this leave Telecom with far less flexability and far greater risk or larger network outages.

Does Telecom still believe it has a better network design than Vodafone's? I get the impression that Vodafone's network is a trusty Toyota, not that fast, things break down but it keeps plodding along, Telecom's is the V8 but when it hits the wall it's in the pitt stop for a day or two.

Also for bonus points, since RNC's are a new concept in 3G networks (not around in 2G networks, their functions were spread out on different devices) why did the 3G network designers not engineer the ability to have load balanced RNC's? This seems like a shortfall in a industry which invented the term 'dial tone' or the uninterrupted availability of service.

Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

peteinakl

22 posts

Geek

#282434 14-Dec-2009 18:44

peteinakl: Ship happens, it's how we recover and take responsability.

that truely counts.

Yes, I am quoting myself - small addendum.

stevenz

2802 posts

Uber Geek
+1 received by user: 125

#282436 14-Dec-2009 18:47

Wah, my cellphone didn't work for a few hours so my friends couldn't text me their mindless waffling and I deserve compensation.

Unless you're running a business that relies on it, then heed the words of Chopper Reid, and "Harden the f*** up". It was only half a day.

Cheers Neal et al.

@KnightNZ

exportgoldman

1202 posts

Uber Geek
+1 received by user: 3

Trusted

#282438 14-Dec-2009 18:49

zonky:
Felix:

CDMA was a better technology than anything else available at the time. Granted the CDMA camp didn't have the marketing / cool devices etc but no one could have predicted which technology would win (though many claim they did now).

Matter of opinion i guess, but CDMA was always going to be hampered by the relative slow growth in North America of the mobile phones market at the time- some would say mainly due to the american pricing model (paying to recieve calls). GSM , logically being dominated by European manafactueres, were already experencing much quicker growth/penetration rates- you didn't need to be much of a soothsayer to see that GSM phones were where the rapid growth was and were going to compete to define the way people used mobile networks

I remember but can't be bothered googling it that the reason Telecom went CDMA is that it's biggest American shareholder forced them to go CDMA (CDMA is big in the states at the time)

Also here's a link for those that have not seen it for TV3's news video about the outage

http://www.3news.co.nz/XT-network-crash-leaves-90000-people-disconnected/tabid/369/articleID/134053/Default.aspx

Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

Geekzone support

Support Geekzone with one-off or recurring donations Donate via PressPatron.

rpopovici

9 posts

Wannabe Geek

#282440 14-Dec-2009 19:01

I do not believe Telecom will offer 10$, or 5$ or anything similar to their subscribers. And I think they are right to do so. In my opinion, to people who really had their lives/bussinesses affected by this outage a 10$ reward would seem like a bad joke, while for the rest of the others who could not send SMSs/use facebook/watch MobileTV etc, any compensation like that would not be appropriate (and in my view necessary). If I were one of the users in the first category, I think I would be far more happy if Telecom kept their money and used them so that an event like this would be less likely in the future.

NealR

428 posts

Ultimate Geek
+1 received by user: 20

ID Verified

Trusted

Lifetime subscriber

#282451 14-Dec-2009 19:37

For those who care we have simplified the issue by saying it was the Christchurch RNC. It was way more complex that this but the RNC platform was the focus. RNC is not a single box. It is a fully redundant, load sharing platform. The fact that we had a failure of this scale should indicate the magnitude of this situation.

The comments I write on this forum do not necessarily reflect the views of my employer and as such cannot be taken as official statements of my employer.

funnyfela

352 posts

Ultimate Geek
+1 received by user: 48

#282455 14-Dec-2009 19:55

Hey Telecom (/NealR), thanks for the basic info you have just said,

If you can, more information on what happened would be an interesting read,
For instance, I am very interested in the inner workings of a cell networks, so the info on what and why went wrong would be very interesting.

Only if you're allowed to (don't wanna get anyone in trouble)

If you have to run heating in winter, you don’t own enough computers.

bbman

646 posts

Ultimate Geek
+1 received by user: 4

#282459 14-Dec-2009 20:17

Big ups to Nealr and others for their communication and of course the people who as quickly possible brought the outage to a conclusion.

Coming from a ground floor, coal face position that I have spent the last 12 years in I see it from there up and my concerns are leveled with top level management who are the first to duck for cover when this thing happens. I experienced the first 3-4 months as a dealer and was not happy with the way things where going, in saying that I am generally a happy XT punter as as a cellular geek I expect down time but today my business was effected by the inability to communicate and the extra cost of using a landline to call mobiles.

I have to say though, the networks guys do a flippen fantatsic job under some less than ideal circumstances.

Yay for having my phone working and I really do hope some investigating is done into how this could have been avoided.

www.ultimatebroadband.co.nz
Delivering better broadband services

UFB fibre, Rural fibre on EA networks, RBI wireless, Ruralnet & Ultra wireless, wireless networks

1 | ... | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14