XT Network down (outage 14 DEC 2009)?

Forums › Spark New Zealand (including Skinny and BigPipe) › XT Network down (outage 14 DEC 2009)?

1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | ... | 14

646 posts

Ultimate Geek

#282265 14-Dec-2009 13:12

spronkey:
martinschcnz: Regardless of the what happened, or why, it is simply unacceptable for a national carrier such as Telecom to have a network that is this unreliable.

In our case not only are mobile communications mission critical they are LIFE critical. There should be no way a failure on ANY piece of equipment can have downstream effects such as this.

...

Simply not good enough Telecom.

While I agree, and don't take this as offense, but surely if your comms are life critical you would have rather serious redundancy?

Have to agree here,

This is no different to web based businesses who used ADSL in the early days and when they had slow speeds and outages went nuts at Telecom. In this case ADSL is a best effort service so no guarantees and those particular businesses should have been using dedicated internet services... they were just not prepared to spend the money required.

Telecom do have to work hard here but mobile is a best effort network also, if it is mission critical then surely a radio network or land based system is preferred?

www.ultimatebroadband.co.nz
Delivering better broadband services

UFB fibre, Rural fibre on EA networks, RBI wireless, Ruralnet & Ultra wireless, wireless networks

TrendMicro

Best TrendMicro deals for antivirus and malware protection(affiliate link).

ATinyChipmunk

136 posts

Master Geek

#282266 14-Dec-2009 13:17

put here in wellington cbd as of 15mins ago

bbman

646 posts

Ultimate Geek

#282270 14-Dec-2009 13:22

Finally online in CHCH, only prob is I had to turn off the phone in options and then turn back on, how many clients will know that?

www.ultimatebroadband.co.nz
Delivering better broadband services

UFB fibre, Rural fibre on EA networks, RBI wireless, Ruralnet & Ultra wireless, wireless networks

profrink

257 posts

Ultimate Geek

#282271 14-Dec-2009 13:24

bbman:
spronkey:
martinschcnz: Regardless of the what happened, or why, it is simply unacceptable for a national carrier such as Telecom to have a network that is this unreliable.

In our case not only are mobile communications mission critical they are LIFE critical. There should be no way a failure on ANY piece of equipment can have downstream effects such as this.

...

Simply not good enough Telecom.

While I agree, and don't take this as offense, but surely if your comms are life critical you would have rather serious redundancy?

Have to agree here,

This is no different to web based businesses who used ADSL in the early days and when they had slow speeds and outages went nuts at Telecom. In this case ADSL is a best effort service so no guarantees and those particular businesses should have been using dedicated internet services... they were just not prepared to spend the money required.

Telecom do have to work hard here but mobile is a best effort network also, if it is mission critical then surely a radio network or land based system is preferred?

Exactly.

It's all in the terms and conditions of any telecommunications company. Basically stating that they do not guarantee 100% network up time. It would be silly to guarantee that too.
From XT's T&C's

4. Provision of Services

2. We will try our best to provide quality Services to you at all times. But, because of the nature of mobile telecommunications, at times our ability to do so will be affected by factors outside of our control including, but not limited to, geographic conditions and physical obstructions, weather conditions, the number of people trying to use our Network at the same time or faults in our Network or other networks used by us to provide the Services to you. We cannot guarantee continuous or fault-free Services.

BaaaaD

29 posts

Geek

#282279 14-Dec-2009 13:33

Yay... Telecom just given all prepaid and on account XT network users $10 credit.

steve98

1381 posts

Uber Geek

Trusted

#282281 14-Dec-2009 13:34

martinschcnz: Regardless of the what happened, or why, it is simply unacceptable for a national carrier such as Telecom to have a network that is this unreliable.

In our case not only are mobile communications mission critical they are LIFE critical. There should be no way a failure on ANY piece of equipment can have downstream effects such as this.

XT has been over-hyped and over sold as the magic bullet for mobile communications.

Our experience to date on XT has been rather lack lustre.

We have experienced several complete outages, numerous smaller issues and ongoing issues with slow delivery of txt messages both intra and inter-network...

Quite frankly, if this is the state of the art in mobile networks then consumers have been grossly misled as o the availability and preformance of XT.

No excuses Telecom, it is YOUR fault, you are the carrier, you are the maintainers of the equipment, you perform the upgrades, we as consumers sit here and wait.... The promised calls from Account managers have not been received and the status page is long overdue for an update. (10:53am) was the last update.

Simply not good enough Telecom.

I would have serious concerns about any business or organisation that relied on mobile technology to support life-critical applications without having some sort of fail-over or backup plan in place. That would mean a very shoddy risk analysis of your operation has been carried out and quite frankly I'd be throwing the book at whoever has carried that out for you if today's network outage has put peoples lives in danger. That is what's not good enough here.

akia

489 posts

Ultimate Geek

Trusted

#282285 14-Dec-2009 13:38

Hey guys - just to let you know that the XT service is back up for some sites, but the service is unstable. Telecom is working furiously to get all XT services and sites restored as soon as we can, will let you know of any further updates.

sbiddle

30853 posts

Uber Geek

Retired Mod

Trusted

Biddle Corp

Lifetime subscriber

#282288 14-Dec-2009 13:44

langi27: I imagine, the boys from Alcatel-Lucent are running around like headless Chickens trying to get it back up.

Someone in Auckland will be getting kicked, Liquided Damages, finger pointing....you know the drill.

Now promise you won't do this again.........Promise!!!!!!

Alcatel Lucent vs Nokia Siemens reminds me of Airbus vs Boeing!

neverland

25 posts

Geek

#282290 14-Dec-2009 13:50

still not working in CHCH at the university despite being advised by the university computer support that the network is now fine (the University uses XT for mobile comms) ...

zonky

62 posts

Master Geek

#282292 14-Dec-2009 13:52

profrink:

Exactly.

It's all in the terms and conditions of any telecommunications company. Basically stating that they do not guarantee 100% network up time. It would be silly to guarantee that too.
From XT's T&C's

4. Provision of Services

2. We will try our best to provide quality Services to you at all times. But, because of the nature of mobile telecommunications, at times our ability to do so will be affected by factors outside of our control including, but not limited to, geographic conditions and physical obstructions, weather conditions, the number of people trying to use our Network at the same time or faults in our Network or other networks used by us to provide the Services to you. We cannot guarantee continuous or fault-free Services.

Read much? Telecom have already purportedly said the issue was within their control- see nbr.co.nz, for example. You've quote a T&C which mentions the opposite.

exportgoldman

1202 posts

Uber Geek

Trusted

#282294 14-Dec-2009 14:00

GJB21: If there was a proper recovery process there'd be a timetable for restoration. Its obviously "suck and see". In your worst-case nightmare scenario you go back to rebuilding from scratch - at least you know how long that will take.

Recovery plans are like battle plans, they never survive the first contact with the enemy.

They are good to have, but the reason you are in battle mode is that things have not gone to plan :( And I imagine no matter how big your lab, it's hard to have a duplicate setup of half of the XT network in the lab to run tests and drills on.

I'm not saying it doesn't suck or isn't a painful thing, and it does appear that the ChristChurch RNC is having issue after issue (and am only glad it wasn't put in Auckland hehe.) but Vodafone have had RNC issues (and IP core issues) as well this and last year, and from what I understand you cannot have hot spare RNC's in your network so they are a single point of failure - Although I assume they have hotswap parts and redundancy up the wazoo.

Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

Felix

344 posts

Ultimate Geek

Trusted

#282296 14-Dec-2009 14:01

steve98:
martinschcnz: Regardless of the what happened, or why, it is simply unacceptable for a national carrier such as Telecom to have a network that is this unreliable.

In our case not only are mobile communications mission critical they are LIFE critical. There should be no way a failure on ANY piece of equipment can have downstream effects such as this.

XT has been over-hyped and over sold as the magic bullet for mobile communications.

Our experience to date on XT has been rather lack lustre.

We have experienced several complete outages, numerous smaller issues and ongoing issues with slow delivery of txt messages both intra and inter-network...

Quite frankly, if this is the state of the art in mobile networks then consumers have been grossly misled as o the availability and preformance of XT.

No excuses Telecom, it is YOUR fault, you are the carrier, you are the maintainers of the equipment, you perform the upgrades, we as consumers sit here and wait.... The promised calls from Account managers have not been received and the status page is long overdue for an update. (10:53am) was the last update.

Simply not good enough Telecom.

I would have serious concerns about any business or organisation that relied on mobile technology to support life-critical applications without having some sort of fail-over or backup plan in place. That would mean a very shoddy risk analysis of your operation has been carried out and quite frankly I'd be throwing the book at whoever has carried that out for you if today's network outage has put peoples lives in danger. That is what's not good enough here.

Agreed... Try a paging network ;-)

gingereejit

89 posts

Master Geek

#282298 14-Dec-2009 14:07

zonky:
profrink:

Exactly.

It's all in the terms and conditions of any telecommunications company. Basically stating that they do not guarantee 100% network up time. It would be silly to guarantee that too.
From XT's T&C's

4. Provision of Services

2. We will try our best to provide quality Services to you at all times. But, because of the nature of mobile telecommunications, at times our ability to do so will be affected by factors outside of our control including, but not limited to, geographic conditions and physical obstructions, weather conditions, the number of people trying to use our Network at the same time or faults in our Network or other networks used by us to provide the Services to you. We cannot guarantee continuous or fault-free Services.

Read much? Telecom have already purportedly said the issue was within their control- see nbr.co.nz, for example. You've quote a T&C which mentions the opposite.

I'm not quite sure which part of the article alludes to TNZ said the issue was within their control - it refers often to a 'uplanned restart of the RNC' - which to me sounds like something they had not planned for.

exportgoldman

1202 posts

Uber Geek

Trusted

#282299 14-Dec-2009 14:07

BaaaaD: Yay... Telecom just given all prepaid and on account XT network users $10 credit.

From memory XT has 100,000 customers so thats a million dollar credit, plus the toll revenue lost, plus additional credit's to corporate customers plus loss of reputation etc.

How to lose a few million dollars on a Monday morning. The worse I did this morning was spill my coffee in the car :)

This is interesting from NBR http://www.nbr.co.nz/article/massive-outage-reveals-xt-s-achilles-heel-116425

"Telecom has two RNCs - one for Taupo and north, the other for everything south of Taupo, with no redundancy. Vodafone, by contrast, has five, with a sixth coming on stream during summer."

I wonder why Telecom decided it could get by with 1/3 of what Vodafone have as RNC's, unless they have duplicate RNC's for the 2G and 3G networks?

Although I don't believe that NBR is right in argueing that not keeping a 2G network around incase the 3G network is down is XT's 'achilles heel'

Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

Behodar

9238 posts

Uber Geek

Trusted

Lifetime subscriber

#282301 14-Dec-2009 14:09

BaaaaD: Yay... Telecom just given all prepaid and on account XT network users $10 credit.

Is this documented somewhere?

1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | ... | 14