Why are all of these XT outages happening south of Taupo?

Forums › Spark New Zealand (including Skinny and BigPipe) › Why are all of these XT outages happening south of Taupo?

1 | 2 | 3

697 posts

Ultimate Geek

Trusted

#301663 23-Feb-2010 21:34

PhoenixNZ: To the original question:

Why is it you can buy two identical problems and one is faulty while the other is fine? The ChCh RNC is not working the way it should and the Auckland one is. Its not a matter of not enough capasity or anything like that. Its simply faulty equipment in one place and not in another.

Perhaps there are more connections on the christchurch RNC than the auckland one.
From auckland to taupo there could be more people on Vodafone rather than XT..

exportgoldman

1202 posts

Uber Geek

Trusted

#301668 23-Feb-2010 21:44

In the full Paul Reynolds interview, Chris Quinn, head of Gen-i explained that RNC's were as big as a roomful of computers, i.e. a small data centre, filled with many hot swap and redundant bits, and also that the RNC in Auckland and Wellington are identical.

Two of XT's four failures have been operational errors, I quote the interview on Close Up (The best I found of all the interviews.)

"There have been two significant operational errors, our partners Alcatel-Lucent loaded software onto the network in the wrong way and it caused the network to fail that was back in December.

we also had another incident, when we put up some new cell sites, we migrated, we changed some new cell sites to a different computer and they put the wrong links in so there have been two of those issues, and they are all in that southern network zone, so what we are both searching for is why is this network not working in the southern part of the zone not working as well as 1500 other places in the world."

Paul is referring to the 1500 RNC's Alcatel-Lucent have deployed world wide.

I also heard on talk back that some people were purposefully sabotaging it, in anger for the Vision stream layoffs. A bit of a stretch but that was the guys story, I wonder how the RNC maintenance teams are organized/secured in Alcatel-Lucent, are there two separate teams within the 700 people Alcatel-Lucent employ in New Zealand to maintain XT's RNC's?

Alcatel-Lucent are flying in their top RNC experts, I wonder what they will find in the logfiles and the engineering teams in Alcatel-Lucent and if it was malicious, or just incompetency which brought down half of XT for 12 hours in two separate incidents, will we be blessed with a juicy high court case with revenge seeking geeks as the villains over the coming months?

Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

exportgoldman

1202 posts

Uber Geek

Trusted

#301670 23-Feb-2010 21:48

cafeg:
PhoenixNZ: To the original question:

Why is it you can buy two identical problems and one is faulty while the other is fine? The ChCh RNC is not working the way it should and the Auckland one is. Its not a matter of not enough capasity or anything like that. Its simply faulty equipment in one place and not in another.

Perhaps there are more connections on the christchurch RNC than the auckland one.
From auckland to taupo there could be more people on Vodafone rather than XT..

Chris Quinn head of Gen-i said that the RNC's are identical, and Paul Renyolds said that two outages were operational, not equipment related.

XT only has 220,000 subscribers per RNC, on the low side for RNC load from what the experts say.

Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

n00dy

482 posts

Ultimate Geek

Lifetime subscriber

#301675 23-Feb-2010 21:59

they also said that the traffic loads that occur North of taupo were different to South, something about traffic management software. When the CHCH RNC came back online it was overwelmed with every cell trying to reconnect / call / txt all at the same time

cafeg

697 posts

Ultimate Geek

Trusted

#301679 23-Feb-2010 22:09

Which would explain why it takes so long to bring it back up fully once it goes down as they would have to bring back up the cell sites connected to it one at a time so as not to overwelm it again..

Is that the right way of looking at it ?

Bung

6480 posts

Uber Geek

Subscriber

#301680 23-Feb-2010 22:10

exportgoldman:

I also heard on talk back that some people were purposefully sabotaging it, in anger for the Vision stream layoffs. A bit of a stretch but that was the guys story, I wonder how the RNC maintenance teams are organized/secured in Alcatel-Lucent, are there two separate teams within the 700 people Alcatel-Lucent employ in New Zealand to maintain XT's RNC's?

I doubt that the 700-800 people working for Alcatel -Lucent are all employed on XT. Your rumour would only make sense if contractors were involved.

insane

3239 posts

Uber Geek

ID Verified

Trusted

#301695 23-Feb-2010 23:47

cafeg: Which would explain why it takes so long to bring it back up fully once it goes down as they would have to bring back up the cell sites connected to it one at a time so as not to overwelm it again..

Is that the right way of looking at it ?

I believe its more of an authentication thing than a bandwidth issue when all the phones try to re-connect all at once. If your broadband ISP has ever had a major radius outage you'll know what I mean, can take a while for you to be authenticated and assigned an IP address, and I'm sure the mobile authentication is even more complicated and taxing.

GoodSync

GoodSync. Easily back up and sync your files with GoodSync. Simple and secure file backup and synchronisation software will ensure that your files are never lost (affiliate link).

matisyahu

1623 posts

Uber Geek

Trusted

#301704 24-Feb-2010 00:52

nix: Just out of interest, why are all of these outages confined to south of Taupo. Has Telecom centralised all of its capacity in Auckland?

Forgive me if this is a duplicate question (and point me to the answer please!). I'm genuinely interested in the answer.

I think the obvious reason is this; 2 RNC for the whole of New Zealand.

When anything is designed one should have two things going in ones mind; KISS (Keep It Simple Stupid) and Murphy's Law. Just an ounce of common-sense will tell you that such a model was just asking for a disaster to occur.

"When the people are being beaten with a stick, they are not much happier if it is called 'the People's Stick'"

insane

3239 posts

Uber Geek

ID Verified

Trusted

#301953 24-Feb-2010 23:37

kawaii:
nix: Just out of interest, why are all of these outages confined to south of Taupo. Has Telecom centralised all of its capacity in Auckland?

Forgive me if this is a duplicate question (and point me to the answer please!). I'm genuinely interested in the answer.

I think the obvious reason is this; 2 RNC for the whole of New Zealand.

When anything is designed one should have two things going in ones mind; KISS (Keep It Simple Stupid) and Murphy's Law. Just an ounce of common-sense will tell you that such a model was just asking for a disaster to occur.

From my reading each RNC does have redundancy built in, so in theory the whole RNC should never fail at once. Paul Raynolds was saying that the active failover did not work on all ocasions and that it was not the same device / componet failing each time.

robjg63

4098 posts

Uber Geek

Subscriber

#301960 25-Feb-2010 05:52

Dont forget that these things are 50% software and 50% hardware.
I gather that at least one of the outages was caused when they tried to upgrade the software to a later level and it all went splat. I'd guess that they just have some really freaky quirk somewhere in the Chch setup and havent been able to find it yet.
Didnt it all run pretty much ok from launch for around 6 months before the current saga started? and the Auckland end of it seems to have been ok.

Nothing is impossible for the man who doesn't have to do it himself - A. H. Weiler

Lambchop

245 posts

Master Geek

#302359 26-Feb-2010 03:50

saw this on egadget today ...wonder if it has any relavance to whats going on with XT ... i mean that network is built primarly around a range of new genration handsets that prob handel data in the same way ...

http://www.engadget.com/2010/02/24/modern-smartphone-radio-design-partly-to-blame-for-atandt-o2-netw/

lestag

100 posts

Master Geek

#302668 27-Feb-2010 11:06

timmyh: Yes this is a great question. Why should the least populous and hence least stressed part of the network keep failing? What is fundamentally different about the southern network from the northern? Differences in equipment and architecture design.

The euipment is bored as it does not have a lot to do... A-L technician called Dave has nicknamed it HAL...

sbiddle

30853 posts

Uber Geek

Retired Mod

Trusted

Biddle Corp

Lifetime subscriber

#302691 27-Feb-2010 13:24

Lambchop: saw this on egadget today ...wonder if it has any relavance to whats going on with XT ... i mean that network is built primarly around a range of new genration handsets that prob handel data in the same way ...

http://www.engadget.com/2010/02/24/modern-smartphone-radio-design-partly-to-blame-for-atandt-o2-netw/

It's very significant. The iPhone has exceptionally poor radio management software which puts significant signalling load on networks. iphone data setup signalling places significantly more load on network and RNC than any other phone on the market.

The 3.0 upgrade caused significant battery issues because of the move to fast dormancy, something that they then had to disable.