Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.


juha

1317 posts

Uber Geek

Trusted

#57602 18-Feb-2010 16:19
Send private message

Just got this release...

February 18, 2010 STATEMENT FROM PAUL REYNOLDS ON XT

* Root cause of January outage identified
* Capacity and resilience improvements underway
* Analysys Mason appointed to undertake independent review

Root cause of January 27 outage identified
The cause of the recent issues with the XT Network South of Taupo is established and is due to traffic surges in the network overloading the Radio Network Controller in Christchurch. During the outage on 27 January, the traffic surge was caused by thousands of users suddenly re-registering after a separate network routing fault took down some cell sites.

Capacity, resilience and coverage improvements underway
Improving the quality and reliability of service on the XT Network is our absolute number one priority and the entire team is focussed on this. Since January 27, Telecom and its technology partner Alcatel Lucent have been working 24/7 to improve capacity and resilience.

The programme of work involves two key elements:

* The addition of processing capacity and surge protection measures in the Radio Network Controller to prevent surge issues from one part of the network affecting customers in other parts, along with adding further Radio Network Controllers.

 * Ongoing improvements to coverage and capacity, including new cell sites, the addition of signal amplification technology, and more fibre optic backhaul.

The additional processing power and surge protection is helping to protect overall service, but we have seen localised problems in recent days caused by local traffic peaks. We successfully migrated the first four cell sites to a new Radio Network Controller on Tuesday and last night we migrated a further 16 sites between 1.00am and 3.00am. Some of those sites showed problems and service degradation once the traffic load began to increase in the morning.

We have subsequently rolled back the migrations whilst we do urgent engineering work to understand the cause of this degradation. All sites were fully-restored by 2.30pm. Our plan is to install two new Radio Network Controllers by mid March.

Analysys Mason appointed to undertake independent review
Telecom has today appointed Analysys Mason to conduct its independent review of the XT mobile network. The Analysys Mason Group (AMG) is a highly respected UK-based international consultancy specialising in telecommunications, technology and media and bring world’s best practice around design, build and operations of mobile networks. AMG are bringing a multi-disciplined team of up to seven specialists who have in depth knowledge of the design, planning and operation of 3G and IP networks from a range of technology suppliers.

Their consultants were hand-picked for their knowledge of radio, core, and transport networks and their understanding of operational processes. AMG are often called on to undertake similar in-life network reviews for a range of clients. The review will commence next week and cover the design, build and operation of the XT mobile network. It is expected to be complete within eight weeks. During the review Analysys Mason will be engaging with Telecom staff, customers, Alcatel-Lucent and other key stakeholders.

Alcatel Lucent comment
Rajeev Singh-Molares, President Asia Pacific of Alcatel-Lucent, said today: “Resolving the XT Network problems in New Zealand is the single most important operational matter in the world for Alcatel-Lucent right now, our entire Executive and Senior Engineering team, including Global Chief Executive Ben Verwaayen are involved. “We apologise to the customers of Telecom New Zealand. The full resources of our global business are available to address these problems – we will do whatever it takes.”




View this topic in a long page with up to 500 replies per page Create new topic
 1 | 2
oldmaknz
536 posts

Ultimate Geek


  #300170 18-Feb-2010 16:36

Good to see Alcatel-Lucent taking a little responsibility. Hopefully they get it sorted.



naggyman
697 posts

Ultimate Geek


  #300174 18-Feb-2010 16:50
Send private message

Good on ya Alcatel-Lucent!
Wow, while Telecom trys to fix it it causes more issues!!!!!




Morgan French-Stagg

 

morgan.french.net.nz

 

 


johnr
19282 posts

Uber Geek
Inactive user


  #300175 18-Feb-2010 16:50
Send private message

Moving sites to a new RNC takes alot of planning. Vodafone NZ has a new RNC in Waikato (number 6) and we started migrating sites 4 weeks before Christmas and this is still on going today.

John



antoniosk
2358 posts

Uber Geek

ID Verified
Trusted
Lifetime subscriber

  #300180 18-Feb-2010 17:00
Send private message

Aaaa, those Analysys Mason guys are back again....




________

 

Antoniosk


exportgoldman
1202 posts

Uber Geek

Trusted

  #300182 18-Feb-2010 17:01
Send private message

johnr: Moving sites to a new RNC takes alot of planning. Vodafone NZ has a new RNC in Waikato (number 6) and we started migrating sites 4 weeks before Christmas and this is still on going today.

John


The joy of having a stable network which you can plan migrations and do testing before hand instead of the mad panic of outage motivated migrations.

From the look of the reports here and the press release Telecom are building and deploying the two new RNC's in record time and simply migrating sites ASAP with no real testing, which is why when they migrated their second batch of sites it worked initially in the depths of the night and then fell over with traffic loading and had to scramble to get  them back to the existing RNC.

Another 12 hour outage, this time the lucky winner is New Plymouth! You win a $20 credit and a blue squiggley logo

So much for the denials that 2 RNC's were enough for XT, and they had no capacity issues... Why spend another 24 Million (which was already budgeted for BTW buying two new RNC's and doubling the performance of the existing ones.) if the Telecom claims were correct.







Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

old3eyes
9119 posts

Uber Geek

Subscriber

  #300200 18-Feb-2010 18:23
Send private message

This is the same Alcatel that installed the not yet operating NGN. As this is about 6 years  behind schedule I hope that they do better with sorting out the XT network..




Regards,

Old3eyes


murray29
24 posts

Geek


  #300218 18-Feb-2010 19:46
Send private message

If my memory serves me correctly, my last post on the thread regarding the previous outage said something like "see you all next month"
and here we are...

see you all next month

 
 
 

Cloud spending continues to surge globally, but most organisations haven’t made the changes necessary to maximise the value and cost-efficiency benefits of their cloud investments. Download the whitepaper From Overspend to Advantage now.
yuxek
147 posts

Master Geek
Inactive user


  #300231 18-Feb-2010 21:24
Send private message

More problems for XT network:

http://www.nzherald.co.nz/business/news/article.cfm?c_id=3&objectid=10626962

New Plymouth, Upper Hutt, Wairarapa affected.

oxnsox
1923 posts

Uber Geek


  #300252 18-Feb-2010 22:49
Send private message

If I understand this statement correctly.....

Paul Said: Righto folks this is what went wrong, sorry about that, but we're paying some blokes a lot of money to tell you what went wrong.....

Ummmm... am I missing something or is there just a little bit of leading the horse by the nose here. Really, publicly does he expect them to say anything different now that he's put it on the record??

freitasm
BDFL - Memuneh
79254 posts

Uber Geek

Administrator
ID Verified
Trusted
Geekzone
Lifetime subscriber

#300254 18-Feb-2010 23:09
Send private message

yuxek: More problems for XT network:  [link to NZHerald] New Plymouth, Upper Hutt, Wairarapa affected.


You are late. This was being discussed already. People should REALLY READ THE FORUM to make sure they are not repeating posts.




Please support Geekzone by subscribing, or using one of our referral links: Samsung | AliExpress | Wise | Sharesies | Hatch | GoodSyncBackblaze backup


exportgoldman
1202 posts

Uber Geek

Trusted

  #300271 19-Feb-2010 07:59
Send private message


This article makes interesting reading in hindsight, it was published last year before XT went live


Can Alcatel deliver?
http://www.stuff.co.nz/technology/711898

A quote from the article

"ABN Amro analyst Geoff Zame noted in a research report in July that Alcatel-Lucent had had a hand in only one of the 24 W-850 networks built around the world – for AT&T in the United States.
"We remain uncertain about the merits of Telecom's reliance on [principally] one vendor and we believe Alcatel-Lucent's market position in W-850 increases execution risk," he said in a subsequent research report."

 




Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

BlakJak
1275 posts

Uber Geek

Trusted

  #300295 19-Feb-2010 09:20
Send private message

freitasm:
yuxek: More problems for XT network:  [link to NZHerald] New Plymouth, Upper Hutt, Wairarapa affected.


You are late. This was being discussed already. People should REALLY READ THE FORUM to make sure they are not repeating posts.


Whoa there, Mauricio.  You tweeted this URL. Folks are going to come straight here without reading the rest of the forum.  Makes it something slightly less offensive, perhaps?





No signature to see here, move along...

wongtop
563 posts

Ultimate Geek


  #300305 19-Feb-2010 09:45
Send private message

OK some questions for the cellular geeks - you know who your are (johnr, nealr and sbiddle).

In the course of the discussions I think we have established that you can't have fully redundant RNCs (e.g. an RNC on hot standby.

1.  The networks seem to operate with RNCs covering geographical areas.  Is there a reason (say handoffs) that cellsites cannot be allocated to RNCs in a checkerboard pattern instead?  That way (to use a cheese analogy) if you lost an RNC you would turn your coverage area into swiss cheese, rather than loosing half the block.  Since many of cellsites are built for capacity rather than coverage, if you lost half your cellsites in a checkerboard pattern you might still be left with say 2/3 of your coverage area.

2.  If the above is not possible, Telecom effectively operate 2 W-CDMA networks, one at 850 MHz and one at 2100 MHz.  I have read somewhere that the 2100 network is mainly for inbound roaming and not for capacity, thus the issues with having the 850 and 2100 networks on different RNCs may not be so bad.  Would it be possible to run the 850 and 2100 sites off different RNCs (e.g the Auckland RNC is responsible for the 850 sites north of Taupo and the 2100 sites south of Taupo, and the CHC RNC is the other way around), so that both the 850 and 2100 sites weren't subject to the same failures.  This would at least give some redundancy for the main centres.

The possible downfall of the above suggestions is that if traffic issues have caused the problems, then these could cascade in as there would be a significant traffic increase on the remaining RNC(s) when one RNC went down.

exportgoldman:
This article makes interesting reading in hindsight, it was published last year before XT went live


Can Alcatel deliver?
http://www.stuff.co.nz/technology/711898

A quote from the article

"ABN Amro analyst Geoff Zame noted in a research report in July that Alcatel-Lucent had had a hand in only one of the 24 W-850 networks built around the world – for AT&T in the United States.
"We remain uncertain about the merits of Telecom's reliance on [principally] one vendor and we believe Alcatel-Lucent's market position in W-850 increases execution risk," he said in a subsequent research report."


I think this is a red herring as none of the issues with the XT network as far as I know (apart from the initial interference issue which delayed the launch) have had anything to do with using 850 MHz on the air interface.  It would appear to me that RNCs are common to all W-CDMA networks (850, 900 and 2100).  The track record of ALU on all W-CDMA networks is probably of bigger relevance now, given the issues they seem to be having.

johnr
19282 posts

Uber Geek
Inactive user


  #300316 19-Feb-2010 10:30
Send private message

1) The transmission path from RNC to Node B is fixed (Iub)

2) The operating Freq is at Node B not the RNC having it split across 2 RNCs would be a nightmare inter RNC singnalling would be huge

John

sbiddle
30853 posts

Uber Geek

Retired Mod
Trusted
Biddle Corp
Lifetime subscriber

  #300332 19-Feb-2010 11:38
Send private message

wongtop: OK some questions for the cellular geeks - you know who your are (johnr, nealr and sbiddle).

In the course of the discussions I think we have established that you can't have fully redundant RNCs (e.g. an RNC on hot standby.

1.  The networks seem to operate with RNCs covering geographical areas.  Is there a reason (say handoffs) that cellsites cannot be allocated to RNCs in a checkerboard pattern instead?  That way (to use a cheese analogy) if you lost an RNC you would turn your coverage area into swiss cheese, rather than loosing half the block.  Since many of cellsites are built for capacity rather than coverage, if you lost half your cellsites in a checkerboard pattern you might still be left with say 2/3 of your coverage area.

2.  If the above is not possible, Telecom effectively operate 2 W-CDMA networks, one at 850 MHz and one at 2100 MHz.  I have read somewhere that the 2100 network is mainly for inbound roaming and not for capacity, thus the issues with having the 850 and 2100 networks on different RNCs may not be so bad.  Would it be possible to run the 850 and 2100 sites off different RNCs (e.g the Auckland RNC is responsible for the 850 sites north of Taupo and the 2100 sites south of Taupo, and the CHC RNC is the other way around), so that both the 850 and 2100 sites weren't subject to the same failures.  This would at least give some redundancy for the main centres.



Both your suggestions come against a fundamental problem - if you're handing a call between multiple RNCs it generates significant signalling traffic between them. Right now this only occurs when a phone changes from the Auckland RNC to the Christchurch RNC. Building a network where a significant number of handsets would be jumping between RNCs continually would require massive connectivity between RNCs, if it was even possible at all.

The iub's (the links from the cellites to the RNC) are essentially grouped together and load balanced when they connect to the RNC. This is why the outage last month affected customers in not only the South Island but also some sites in Wellington and both sites in Mt Ruapehu. If one of these "load balancers" fails then it's not necessarily a single geographic area that is taken out entirely.

The key thing here is that an RNC simply should not fail. Disaster recovery and risk mitigation have to be factored into everything, however you can only go so far.

 1 | 2
View this topic in a long page with up to 500 replies per page Create new topic





News and reviews »

Air New Zealand Starts AI adoption with OpenAI
Posted 24-Jul-2025 16:00


eero Pro 7 Review
Posted 23-Jul-2025 12:07


BeeStation Plus Review
Posted 21-Jul-2025 14:21


eero Unveils New Wi-Fi 7 Products in New Zealand
Posted 21-Jul-2025 00:01


WiZ Introduces HDMI Sync Box and other Light Devices
Posted 20-Jul-2025 17:32


RedShield Enhances DDoS and Bot Attack Protection
Posted 20-Jul-2025 17:26


Seagate Ships 30TB Drives
Posted 17-Jul-2025 11:24


Oclean AirPump A10 Water Flosser Review
Posted 13-Jul-2025 11:05


Samsung Galaxy Z Fold7: Raising the Bar for Smartphones
Posted 10-Jul-2025 02:01


Samsung Galaxy Z Flip7 Brings New Edge-To-Edge FlexWindow
Posted 10-Jul-2025 02:01


Epson Launches New AM-C550Z WorkForce Enterprise printer
Posted 9-Jul-2025 18:22


Samsung Releases Smart Monitor M9
Posted 9-Jul-2025 17:46


Nearly Half of Older Kiwis Still Write their Passwords on Paper
Posted 9-Jul-2025 08:42


D-Link 4G+ Cat6 Wi-Fi 6 DWR-933M Mobile Hotspot Review
Posted 1-Jul-2025 11:34


Oppo A5 Series Launches With New Levels of Durability
Posted 30-Jun-2025 10:15









Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.