Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.


View this topic in a long page with up to 500 replies per page Create new topic
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14
675 posts

Ultimate Geek

Trusted

Reply # 282397 14-Dec-2009 17:24

freitasm:
NealR: Just know that the restoration of todays service loss has been the focus of a very large number of people from all around the world.

Nobody wanted this to happen, nobody planned for this to happen, nobody "messed up" to cause this to happen, we don't have a "fundemental flaw" in our architecture and nobody was wasting time to resolve this issue. This was a serious event that could not have been planned for or tested around to develop quick and simple restoration plans.

Right now all I can do is appologise for the service interuption and praise the engineers who worked out the problem and resolved it.

Until you have been an engineer of a very complex technology that is responsible for the running of a large proportion of NZ businesses, as well as millions of dollars of revenue which when lost affects the pay packets of thousands of people, you cannot understand the preasure today's engineers had to deal with. Whether you like Telecom or it's suppliers, or it's technology choices, or the services it provides does not remove the fact that the engineers who resolved today's problems deserve respect.


Word!


QFT!

22 posts

Geek


  Reply # 282398 14-Dec-2009 17:34
Send private message

Ship happens, it's how we recover and take responsability.

I was floored today to be getting emails from [in a round about kind of way] Chris Quin, showing a commitment to not only resolving the issue, but taking responsability.

Great recovery!  (But hopefully the learnings mean it won't happen again)

 
 
 
 


BDFL - Memuneh
58719 posts

Uber Geek
+1 received by user: 10115

Administrator
Trusted
Geekzone
Subscriber

  Reply # 282399 14-Dec-2009 17:40
Send private message

peteinakl: Ship happens, it's how we recover and take responsability.



I used this in my blog post about today:





89 posts

Master Geek
+1 received by user: 5


  Reply # 282414 14-Dec-2009 18:16

NealR: Just know that the restoration of todays service loss has been the focus of a very large number of people from all around the world.

Nobody wanted this to happen, nobody planned for this to happen, nobody "messed up" to cause this to happen, we don't have a "fundemental flaw" in our architecture and nobody was wasting time to resolve this issue. This was a serious event that could not have been planned for or tested around to develop quick and simple restoration plans.

Right now all I can do is appologise for the service interuption and praise the engineers who worked out the problem and resolved it.

Until you have been an engineer of a very complex technology that is responsible for the running of a large proportion of NZ businesses, as well as millions of dollars of revenue which when lost affects the pay packets of thousands of people, you cannot understand the preasure today's engineers had to deal with. Whether you like Telecom or it's suppliers, or it's technology choices, or the services it provides does not remove the fact that the engineers who resolved today's problems deserve respect.




What Neal said.

I think TNZ are actually doing a very good job on the comms side, by having teams and individuals on forums such as this and twitter - it's appropriate and modern mode of communication.



203 posts

Master Geek
+1 received by user: 37

Trusted
Emergency Management

  Reply # 282419 14-Dec-2009 18:22
Send private message

NealR: Just know that the restoration of todays service loss has been the focus of a very large number of people from all around the world.

Nobody wanted this to happen, nobody planned for this to happen, nobody "messed up" to cause this to happen, we don't have a "fundemental flaw" in our architecture and nobody was wasting time to resolve this issue. This was a serious event that could not have been planned for or tested around to develop quick and simple restoration plans.

Right now all I can do is appologise for the service interuption and praise the engineers who worked out the problem and resolved it.

Until you have been an engineer of a very complex technology that is responsible for the running of a large proportion of NZ businesses, as well as millions of dollars of revenue which when lost affects the pay packets of thousands of people, you cannot understand the preasure today's engineers had to deal with. Whether you like Telecom or it's suppliers, or it's technology choices, or the services it provides does not remove the fact that the engineers who resolved today's problems deserve respect.




Nice, Smile its easy for people to sit here on their computers and flame the big corp for having an outage, I know there was a huge amount of work behind the scenes and I take my hat off for the team that did restore service, they worked there butts off to get it back up and running, also the front line teams thats delt with all the customers that had issues.

Like Neal, All we can do is appolgise for the service interuption and hope people can see that we did do our best to restore service.

My 2cents Smile

674 posts

Ultimate Geek
+1 received by user: 79

Subscriber

  Reply # 282421 14-Dec-2009 18:28
Send private message

underwatervrg: 
 its easy for people to sit here on their computers and flame the big corp for having an outage, I know there was a huge amount of work behind the scenes and I take my hat off for the team that did restore service, they worked there butts off to get it back up and running, also the front line teams thats delt with all the customers that had issues.

Like Neal, All we can do is appolgise for the service interuption and hope people can see that we did do our best to restore service.

My 2cents Smile



+1 amen.




Morgan French-Stagg

 

morgan.french.net.nz

 

 


Tel69
249 posts

Master Geek
+1 received by user: 4

Trusted
Subscriber

  Reply # 282431 14-Dec-2009 18:41
Send private message

NealR: Just know that the restoration of todays service loss has been the focus of a very large number of people from all around the world.

Nobody wanted this to happen, nobody planned for this to happen, nobody "messed up" to cause this to happen, we don't have a "fundemental flaw" in our architecture and nobody was wasting time to resolve this issue. This was a serious event that could not have been planned for or tested around to develop quick and simple restoration plans.

Right now all I can do is appologise for the service interuption and praise the engineers who worked out the problem and resolved it.

Until you have been an engineer of a very complex technology that is responsible for the running of a large proportion of NZ businesses, as well as millions of dollars of revenue which when lost affects the pay packets of thousands of people, you cannot understand the preasure today's engineers had to deal with. Whether you like Telecom or it's suppliers, or it's technology choices, or the services it provides does not remove the fact that the engineers who resolved today's problems deserve respect.


Totally agreed

1200 posts

Uber Geek
+1 received by user: 3

Trusted

  Reply # 282433 14-Dec-2009 18:43
Send private message

NealR: Just know that the restoration of todays service loss has been the focus of a very large number of people from all around the world.

Nobody wanted this to happen, nobody planned for this to happen, nobody "messed up" to cause this to happen, we don't have a "fundemental flaw" in our architecture and nobody was wasting time to resolve this issue. This was a serious event that could not have been planned for or tested around to develop quick and simple restoration plans.

Right now all I can do is appologise for the service interuption and praise the engineers who worked out the problem and resolved it.

Until you have been an engineer of a very complex technology that is responsible for the running of a large proportion of NZ businesses, as well as millions of dollars of revenue which when lost affects the pay packets of thousands of people, you cannot understand the preasure today's engineers had to deal with. Whether you like Telecom or it's suppliers, or it's technology choices, or the services it provides does not remove the fact that the engineers who resolved today's problems deserve respect.




First off I would like to say a word of thanks to the faceless Telecom engineers which were sweating raw emotion today... Every engineer worth his salt knows that blind fear when the big important thing is broken in ways unknown.

My understanding (and correct me if I'm wrong) is that although Radio Network Controllers  (RNC's) have hotswappable and redundant parts you cannot have hot spare or load balanced RNC's, so if a RNC goes down all the cell sites (node B's) off that site are down until the RNC comes back online or is rebuilt/replaced.

So how come Vodafone with it's 'inferior' network has 6 Radio Network Controllers (RNC's) in use for their 3G network so when a RNC outage occurs (and they do - both Vodafone and Telecom have had RNC failures in the last 12 months) Vodafone's RNC failures affect less customers than Telecom's, and Vodafone can move Cell Site's onto different RNC's if one RNC is 'running hot' so doesn't this leave Telecom with far less flexability and far greater risk or larger network outages.

Does Telecom still believe it has a better network design than Vodafone's? I get the impression that Vodafone's network is a trusty Toyota, not that fast, things break down but it keeps plodding along, Telecom's is the V8 but when it hits the wall it's in the pitt stop for a day or two.

Also for bonus points, since RNC's are a new concept in 3G networks (not around in 2G networks, their functions were spread out on different devices) why did the 3G network designers not engineer the ability to have load balanced RNC's? This seems like a shortfall in a industry which invented the term 'dial tone' or the uninterrupted availability of service.




Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

22 posts

Geek


  Reply # 282434 14-Dec-2009 18:44
Send private message

peteinakl: Ship happens, it's how we recover and take responsability.
that truely counts.

Yes, I am quoting myself - small addendum.

2748 posts

Uber Geek
+1 received by user: 114


  Reply # 282436 14-Dec-2009 18:47
Send private message

Wah, my cellphone didn't work for a few hours so my friends couldn't text me their mindless waffling and I deserve compensation.

Unless you're running a business that relies on it, then heed the words of Chopper Reid, and "Harden the f*** up". It was only half a day.

Cheers Neal et al.




1200 posts

Uber Geek
+1 received by user: 3

Trusted

  Reply # 282438 14-Dec-2009 18:49
Send private message

zonky:
Felix:

CDMA was a better technology than anything else available at the time. Granted the CDMA camp didn't have the marketing / cool devices etc but no one could have predicted which technology would win (though many claim they did now).


Matter of opinion i guess, but CDMA was always going to be hampered by the relative slow growth in North America of the mobile phones market at the time- some would say mainly due to the american pricing model (paying to recieve calls). GSM , logically being dominated by European manafactueres, were already experencing much quicker growth/penetration rates- you didn't need to be much of a soothsayer to see that GSM phones were where the rapid growth was and were going to compete to define the way people used mobile networks


I remember but can't be bothered googling it that the reason Telecom went CDMA is that it's biggest American shareholder forced them to go CDMA (CDMA is big in the states at the time)


Also here's a link for those that have not seen it for TV3's news video about the outage

http://www.3news.co.nz/XT-network-crash-leaves-90000-people-disconnected/tabid/369/articleID/134053/Default.aspx




Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

8 posts

Wannabe Geek


  Reply # 282440 14-Dec-2009 19:01
Send private message

I do not believe Telecom will offer 10$, or 5$ or anything similar to their subscribers. And I think they are right to do so. In my opinion, to people who really had their lives/bussinesses affected by this outage a 10$ reward would seem like a bad joke, while for the rest of the others who could not send SMSs/use facebook/watch MobileTV etc, any compensation like that would not be appropriate (and in my view necessary). If I were one of the users in the first category, I think I would be far more happy if Telecom kept their money and used them so that an event like this would be less likely in the future.


426 posts

Ultimate Geek
+1 received by user: 16

Trusted
Spark NZ
Subscriber

  Reply # 282451 14-Dec-2009 19:37
Send private message

For those who care we have simplified the issue by saying it was the Christchurch RNC. It was way more complex that this but the RNC platform was the focus. RNC is not a single box. It is a fully redundant, load sharing platform. The fact that we had a failure of this scale should indicate the magnitude of this situation.




The comments I write on this forum do not necessarily reflect the views of my employer and as such cannot be taken as official statements of my employer.

248 posts

Master Geek
+1 received by user: 13


  Reply # 282455 14-Dec-2009 19:55
Send private message

Hey Telecom (/NealR), thanks for the basic info you have just said,

If you can, more information on what happened would be an interesting read,
For instance, I am very interested in the inner workings of a cell networks, so the info on what and why went wrong would be very interesting.

Only if you're allowed to (don't wanna get anyone in trouble)




If you have to run heating in winter, you don’t own enough computers.


646 posts

Ultimate Geek
+1 received by user: 4


  Reply # 282459 14-Dec-2009 20:17
Send private message

Big ups to Nealr and others for their communication and of course the people who as quickly possible brought the outage to a conclusion.

Coming from a ground floor, coal face position that I have spent the last 12 years in I see it from there up and my concerns are leveled with top level management who are the first to duck for cover when this thing happens. I experienced the first 3-4 months as a dealer and was not happy with the way things where going, in saying that I am generally a happy XT punter as as a cellular geek I expect down time but today my business was effected by the inability to communicate and the extra cost of using a landline to call mobiles.

I have to say though, the networks guys do a flippen fantatsic job under some less than ideal circumstances.

Yay for having my phone working and I really do hope some investigating is done into how this could have been avoided.





www.ultimatebroadband.co.nz 
Delivering better broadband services

UFB fibre, Rural fibre on EA networks, RBI wireless, Ruralnet & Ultra wireless, wireless networks


1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14
View this topic in a long page with up to 500 replies per page Create new topic



Twitter »

Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





News »

FUJIFILM introduces X-E3 mirrorless camera with wireless connectivity
Posted 18-Sep-2017 13:53


Vodafone announces new plans with bigger data bundles
Posted 15-Sep-2017 10:51


Skinny launches phone with support for te reo Maori
Posted 14-Sep-2017 08:39


If Vodafone dropping mail worries you, you’re doing online wrong
Posted 11-Sep-2017 13:54


Vodafone New Zealand deploy live 400 gigabit system
Posted 11-Sep-2017 11:07


OPPO camera phones now available at PB Tech
Posted 11-Sep-2017 09:56


Norton Wi-Fi Privacy — Easy, flawed VPN
Posted 11-Sep-2017 09:48


Lenovo reveals new ThinkPad A Series
Posted 8-Sep-2017 14:37


Huawei passes Apple for the first time to capture the second spot globally
Posted 8-Sep-2017 10:45


Vodafone initiative enhances te reo Maori pronunciation on Google Maps
Posted 8-Sep-2017 10:40


Voyager Internet expand local internet phone services company with Conversant acquisition
Posted 6-Sep-2017 18:27


NOW Expands in to Tauranga
Posted 5-Sep-2017 18:16


Windows 10 Fall Creators Update coming Oct. 17
Posted 4-Sep-2017 14:10


Garmin introduce Garmin vivoactive 3
Posted 1-Sep-2017 18:38


Kiwibank wastes $90 million on software – Reseller News
Posted 1-Sep-2017 13:45



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.