XT Network down (outage 14 DEC 2009)?

Forums › Spark New Zealand (including Skinny and BigPipe) › XT Network down (outage 14 DEC 2009)?

1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | ... | 14

1539 posts

Uber Geek

Trusted

#282206 14-Dec-2009 11:19

GJB21:
We want Telecom to reenable our CDMA phones and allow free calling on them until the XT network has been stable for say 6 months.

Sure only if your willing to pay for the millions of dollars lost from free calling....

codyc1515

1598 posts

Uber Geek
Inactive user

#282207 14-Dec-2009 11:20

This whole thing is probably somewhat like the AT&T iPhone MMS issue, how they did not want to release MMS to the general public on the iPhone due to network stability issues. No doubt this has something to do with bandwidth on XT network after they (practically) gave away all of those T-Sticks.

I guess they just didn't prepare ahead.. Who would know, when I rung them they just said "It's been out since early this morning, from just below Taupo down to the whole of the south island, we don't know when it will be fixed." However, he wouldn't comment on the cause of it all, or even speculate.

sbiddle

30853 posts

Uber Geek

Retired Mod

Trusted

Biddle Corp

Lifetime subscriber

#282208 14-Dec-2009 11:20

GJB21:No excuses - it may not be a single point of failure - but its still poor design if a fault can cascade and close down the system so completely, and there's obviously no recovery process that's been invoked.

Failure of a RNC is something that is the worst nightmare for a mobile operator. You can't simply easily have a backup RNC or a hot standby in case of a failure. Vodafone had several RNC failures earlier this year or late last year which affected the Auckland region.

What sort of "recovery process" do you expect? Telecom and ALU are working through the issue which is why sites are slowly being restored. There is no quick fix that can be implimented to resolve the issue.

oldmaknz

536 posts

Ultimate Geek

#282209 14-Dec-2009 11:22

No doubt this has something to do with bandwidth on XT network after they (practically) gave away all of those T-Sticks.

I highly doubt that.

rscole86

4974 posts

Uber Geek

Moderator

Trusted

Lifetime subscriber

#282210 14-Dec-2009 11:24

codyc1515: No doubt this has something to do with bandwidth on XT network after they (practically) gave away all of those T-Sticks.

I guess they just didn't prepare ahead..

LOL!!!

Oh you were serious?

stubbed

GIS Geek
128 posts

Master Geek

Trusted

Lifetime subscriber

#282213 14-Dec-2009 11:29

Latest email update:

Although the Christchurch RNC Gateway has been restored, voice and data traffic are still not being carried as intended.
Therefore, many customers will still be experiencing loss of XT Network services.
Further investigation is underway however there is no estimated resolution time.
Additional updates to follow as they become available.

https://gisgeek.app

NealR

426 posts

Ultimate Geek

Trusted

Lifetime subscriber

#282214 14-Dec-2009 11:30

Todays issues are in no way related to the TStick giveaway.

The comments I write on this forum do not necessarily reflect the views of my employer and as such cannot be taken as official statements of my employer.

Quic Broadband

Move to New Zealand's best fibre broadband service (affiliate link). Free setup code: R587125ERQ6VE. Note that to use Quic Broadband you must be comfortable with configuring your own router.

langi27

676 posts

Ultimate Geek

#282215 14-Dec-2009 11:31

I imagine, the boys from Alcatel-Lucent are running around like headless Chickens trying to get it back up.

Someone in Auckland will be getting kicked, Liquided Damages, finger pointing....you know the drill.

Now promise you won't do this again.........Promise!!!!!!

bbman

646 posts

Ultimate Geek

#282216 14-Dec-2009 11:32

sbiddle:
GJB21:No excuses - it may not be a single point of failure - but its still poor design if a fault can cascade and close down the system so completely, and there's obviously no recovery process that's been invoked.

Failure of a RNC is something that is the worst nightmare for a mobile operator. You can't simply easily have a backup RNC or a hot standby in case of a failure. Vodafone had several RNC failures earlier this year or late last year which affected the Auckland region.

What sort of "recovery process" do you expect? Telecom and ALU are working through the issue which is why sites are slowly being restored. There is no quick fix that can be implimented to resolve the issue.

Is the case of not having more RNC an issue, i.e. there is one for auckland or taupo north and one for the rest potentially problematic.

I am sure thats a really simplistic view from some one who doesnt know enough about the structure of UMTS, learning now, but maybe having more RNC's decreases the effected areas size. i.e large regions, say 5 in total, 3 in North, 2 in south island rather than two I believe right now, or is it 3?

Maybe thats just not efficent, googling to get an understanding but interested to know.

www.ultimatebroadband.co.nz
Delivering better broadband services

UFB fibre, Rural fibre on EA networks, RBI wireless, Ruralnet & Ultra wireless, wireless networks

nitrotech

1285 posts

Uber Geek

#282217 14-Dec-2009 11:38

sbiddle:
GJB21:No excuses - it may not be a single point of failure - but its still poor design if a fault can cascade and close down the system so completely, and there's obviously no recovery process that's been invoked.

Failure of a RNC is something that is the worst nightmare for a mobile operator. You can't simply easily have a backup RNC or a hot standby in case of a failure. Vodafone had several RNC failures earlier this year or late last year which affected the Auckland region.

What sort of "recovery process" do you expect? Telecom and ALU are working through the issue which is why sites are slowly being restored. There is no quick fix that can be implimented to resolve the issue.

I agtee, having worked in a remote support situation in the past I have seen how easily people want to blame others or offer 'helpful' suggestions.

This is simply not a good look for telecom and they will want this fixed as soon as possible to minimise impact on their reputation especially for business customers.

But technology is a funny thing it falls over at the most in opportune time and sometimes with disastrous results.

The tech's at telecom are world class and i'm sure they are doing everything they can

GJB21

55 posts

Master Geek

#282221 14-Dec-2009 11:43

sbiddle:
GJB21:No excuses - it may not be a single point of failure - but its still poor design if a fault can cascade and close down the system so completely, and there's obviously no recovery process that's been invoked.

Failure of a RNC is something that is the worst nightmare for a mobile operator. You can't simply easily have a backup RNC or a hot standby in case of a failure. Vodafone had several RNC failures earlier this year or late last year which affected the Auckland region.

What sort of "recovery process" do you expect? Telecom and ALU are working through the issue which is why sites are slowly being restored. There is no quick fix that can be implimented to resolve the issue.

If there was a proper recovery process there'd be a timetable for restoration. Its obviously "suck and see".
In your worst-case nightmare scenario you go back to rebuilding from scratch - at least you know how long that will take.

And re the free calling on CDMA - all I want is the old phones to be left enabled (a la prepay) so we can use them when the XT network is down to run our business - calls made on CDMA whenever the XT network is down to be free the same as inter-group calls are now - at other times its a normal prepay - after all we're paying for a XT service now that Telecom is not delivering so they've got the income!

stubbed

GIS Geek
128 posts

Master Geek

Trusted

Lifetime subscriber

#282222 14-Dec-2009 11:47

sbiddle: Failure of a RNC is something that is the worst nightmare for a mobile operator. You can't simply easily have a backup RNC or a hot standby in case of a failure.

For the UTMS newbs (Myself included) can you elaborate?

https://gisgeek.app

zl4plm

45 posts

Geek

#282223 14-Dec-2009 11:50

as I used to work for Vodafone in IT I would be asking rather a different question!

where is your redundany telecom?

a RNC should never be down in its entirety!

there should be redundancy in any network whether it be mobile or deep in a gateway .. there is always 2 of everything in an ability to re route round the issue.

the only way you have have a major outage is either failure of a core component that has no redundancy, poor design in that redundant systems fail ... or crappy upgrade plans.

ohhh and I don't accept that upgrades should ever cause loss of service for hours on a major providers network!

needless to say if this happened in Europe you would have the world falling on the providers head from government regulators.

get tough NZ and grow a pair and start kicking these clowns.

I run a business here and rely on mobile and Internet service....... and I have just made a formal complaint to telecom and expect compensated for loss.

my experience of telecom has been dire ....... poor billing... poor knowledge of their call centres.... unreliable network blah blah.

I went nuts this morning when I complained about the poor web performance this morning in CHCH which I am sure is related to the outage... when I get told by the customer service representative ... ohh yes I can see your on the big time plan you know this this traffic managed.... red rag to bull!

I would suggest if yr unhappy with telecoms service you call and log a complaint, log a complaint by the web form on line in telecoms site and then log that same complaint to the regulator.

a bit like ACC .. unless you shout and stamp poor service will continue .....

as for telecom ..... if I was senior mgt I would be asking for the head of the responsible persons which allowed at least 4 major network outages in the past month to occur on a new network!

bbman

646 posts

Ultimate Geek

#282224 14-Dec-2009 11:53

GJB21:
sbiddle:
GJB21:No excuses - it may not be a single point of failure - but its still poor design if a fault can cascade and close down the system so completely, and there's obviously no recovery process that's been invoked.

Failure of a RNC is something that is the worst nightmare for a mobile operator. You can't simply easily have a backup RNC or a hot standby in case of a failure. Vodafone had several RNC failures earlier this year or late last year which affected the Auckland region.

What sort of "recovery process" do you expect? Telecom and ALU are working through the issue which is why sites are slowly being restored. There is no quick fix that can be implimented to resolve the issue.

If there was a proper recovery process there'd be a timetable for restoration. Its obviously "suck and see".
In your worst-case nightmare scenario you go back to rebuilding from scratch - at least you know how long that will take.

And re the free calling on CDMA - all I want is the old phones to be left enabled (a la prepay) so we can use them when the XT network is down to run our business - calls made on CDMA whenever the XT network is down to be free the same as inter-group calls are now - at other times its a normal prepay - after all we're paying for a XT service now that Telecom is not delivering so they've got the income!

Tricky way of solving as there is a whole process involved, i guess though thats a discussion for the client manager within Telecom to have. If a black out happens it is usually under 4 hours long, so you could spend a a ton of organizing diversions to CDMA prepaid numbers in an event like this which then have to be removed when it comes online again.

Not worth it and I am sure that the motivation is there to improve on the reliablity of the network.

I have just been privy to a customer email that was a response to a client managers pro active email about the outage, comments included the statment,"when can Telecom give us a timetable for Inage as we seem to have outages all the time so it would be nice to have an inage report"

Telecom will be seriously working on this, yes it is embarressing but it is part of the whole new network thing which we all know. I just wish the marketing and number hungry retail management hadn't pushed as hard as they have for numbers, this may tarnish what is in essence a very good network.

www.ultimatebroadband.co.nz
Delivering better broadband services

UFB fibre, Rural fibre on EA networks, RBI wireless, Ruralnet & Ultra wireless, wireless networks

lambo

9 posts

Wannabe Geek

#282225 14-Dec-2009 12:00

Twitter:

XT Fault Update: Sorry guys there is still no confirmed resolution time, teams are working furiously to bring your XT service back. ^AC

1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | ... | 14