Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.




Juha
1318 posts

Uber Geek
+1 received by user: 5

Trusted
Subscriber

Topic # 57602 18-Feb-2010 16:19
Send private message

Just got this release...

February 18, 2010 STATEMENT FROM PAUL REYNOLDS ON XT

* Root cause of January outage identified
* Capacity and resilience improvements underway
* Analysys Mason appointed to undertake independent review

Root cause of January 27 outage identified
The cause of the recent issues with the XT Network South of Taupo is established and is due to traffic surges in the network overloading the Radio Network Controller in Christchurch. During the outage on 27 January, the traffic surge was caused by thousands of users suddenly re-registering after a separate network routing fault took down some cell sites.

Capacity, resilience and coverage improvements underway
Improving the quality and reliability of service on the XT Network is our absolute number one priority and the entire team is focussed on this. Since January 27, Telecom and its technology partner Alcatel Lucent have been working 24/7 to improve capacity and resilience.

The programme of work involves two key elements:

* The addition of processing capacity and surge protection measures in the Radio Network Controller to prevent surge issues from one part of the network affecting customers in other parts, along with adding further Radio Network Controllers.

 * Ongoing improvements to coverage and capacity, including new cell sites, the addition of signal amplification technology, and more fibre optic backhaul.

The additional processing power and surge protection is helping to protect overall service, but we have seen localised problems in recent days caused by local traffic peaks. We successfully migrated the first four cell sites to a new Radio Network Controller on Tuesday and last night we migrated a further 16 sites between 1.00am and 3.00am. Some of those sites showed problems and service degradation once the traffic load began to increase in the morning.

We have subsequently rolled back the migrations whilst we do urgent engineering work to understand the cause of this degradation. All sites were fully-restored by 2.30pm. Our plan is to install two new Radio Network Controllers by mid March.

Analysys Mason appointed to undertake independent review
Telecom has today appointed Analysys Mason to conduct its independent review of the XT mobile network. The Analysys Mason Group (AMG) is a highly respected UK-based international consultancy specialising in telecommunications, technology and media and bring world’s best practice around design, build and operations of mobile networks. AMG are bringing a multi-disciplined team of up to seven specialists who have in depth knowledge of the design, planning and operation of 3G and IP networks from a range of technology suppliers.

Their consultants were hand-picked for their knowledge of radio, core, and transport networks and their understanding of operational processes. AMG are often called on to undertake similar in-life network reviews for a range of clients. The review will commence next week and cover the design, build and operation of the XT mobile network. It is expected to be complete within eight weeks. During the review Analysys Mason will be engaging with Telecom staff, customers, Alcatel-Lucent and other key stakeholders.

Alcatel Lucent comment
Rajeev Singh-Molares, President Asia Pacific of Alcatel-Lucent, said today: “Resolving the XT Network problems in New Zealand is the single most important operational matter in the world for Alcatel-Lucent right now, our entire Executive and Senior Engineering team, including Global Chief Executive Ben Verwaayen are involved. “We apologise to the customers of Telecom New Zealand. The full resources of our global business are available to address these problems – we will do whatever it takes.”




View this topic in a long page with up to 500 replies per page Create new topic
 1 | 2
536 posts

Ultimate Geek


  Reply # 300170 18-Feb-2010 16:36

Good to see Alcatel-Lucent taking a little responsibility. Hopefully they get it sorted.

674 posts

Ultimate Geek
+1 received by user: 79

Subscriber

  Reply # 300174 18-Feb-2010 16:50
Send private message

Good on ya Alcatel-Lucent!
Wow, while Telecom trys to fix it it causes more issues!!!!!




Morgan French-Stagg

 

morgan.french.net.nz

 

 


 
 
 
 


19282 posts

Uber Geek
+1 received by user: 2600
Inactive user


  Reply # 300175 18-Feb-2010 16:50
Send private message

Moving sites to a new RNC takes alot of planning. Vodafone NZ has a new RNC in Waikato (number 6) and we started migrating sites 4 weeks before Christmas and this is still on going today.

John

1478 posts

Uber Geek
+1 received by user: 247

Trusted
Subscriber

  Reply # 300180 18-Feb-2010 17:00
Send private message

Aaaa, those Analysys Mason guys are back again....




________
AK

1200 posts

Uber Geek
+1 received by user: 3

Trusted

  Reply # 300182 18-Feb-2010 17:01
Send private message

johnr: Moving sites to a new RNC takes alot of planning. Vodafone NZ has a new RNC in Waikato (number 6) and we started migrating sites 4 weeks before Christmas and this is still on going today.

John


The joy of having a stable network which you can plan migrations and do testing before hand instead of the mad panic of outage motivated migrations.

From the look of the reports here and the press release Telecom are building and deploying the two new RNC's in record time and simply migrating sites ASAP with no real testing, which is why when they migrated their second batch of sites it worked initially in the depths of the night and then fell over with traffic loading and had to scramble to get  them back to the existing RNC.

Another 12 hour outage, this time the lucky winner is New Plymouth! You win a $20 credit and a blue squiggley logo

So much for the denials that 2 RNC's were enough for XT, and they had no capacity issues... Why spend another 24 Million (which was already budgeted for BTW buying two new RNC's and doubling the performance of the existing ones.) if the Telecom claims were correct.







Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

7683 posts

Uber Geek
+1 received by user: 738

Subscriber

  Reply # 300200 18-Feb-2010 18:23
Send private message

This is the same Alcatel that installed the not yet operating NGN. As this is about 6 years  behind schedule I hope that they do better with sorting out the XT network..




Regards,

Old3eyes


24 posts

Geek


  Reply # 300218 18-Feb-2010 19:46
Send private message

If my memory serves me correctly, my last post on the thread regarding the previous outage said something like "see you all next month"
and here we are...

see you all next month

147 posts

Master Geek
Inactive user


  Reply # 300231 18-Feb-2010 21:24
Send private message

More problems for XT network:

http://www.nzherald.co.nz/business/news/article.cfm?c_id=3&objectid=10626962

New Plymouth, Upper Hutt, Wairarapa affected.

1923 posts

Uber Geek
+1 received by user: 139


  Reply # 300252 18-Feb-2010 22:49
Send private message

If I understand this statement correctly.....

Paul Said: Righto folks this is what went wrong, sorry about that, but we're paying some blokes a lot of money to tell you what went wrong.....

Ummmm... am I missing something or is there just a little bit of leading the horse by the nose here. Really, publicly does he expect them to say anything different now that he's put it on the record??

BDFL - Memuneh
59192 posts

Uber Geek
+1 received by user: 10423

Administrator
Trusted
Geekzone
Subscriber

Reply # 300254 18-Feb-2010 23:09
Send private message

yuxek: More problems for XT network:  [link to NZHerald] New Plymouth, Upper Hutt, Wairarapa affected.


You are late. This was being discussed already. People should REALLY READ THE FORUM to make sure they are not repeating posts.




1200 posts

Uber Geek
+1 received by user: 3

Trusted

  Reply # 300271 19-Feb-2010 07:59
Send private message


This article makes interesting reading in hindsight, it was published last year before XT went live


Can Alcatel deliver?
http://www.stuff.co.nz/technology/711898

A quote from the article

"ABN Amro analyst Geoff Zame noted in a research report in July that Alcatel-Lucent had had a hand in only one of the 24 W-850 networks built around the world – for AT&T in the United States.
"We remain uncertain about the merits of Telecom's reliance on [principally] one vendor and we believe Alcatel-Lucent's market position in W-850 increases execution risk," he said in a subsequent research report."

 




Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

598 posts

Ultimate Geek
+1 received by user: 53

Trusted

  Reply # 300295 19-Feb-2010 09:20
Send private message

freitasm:
yuxek: More problems for XT network:  [link to NZHerald] New Plymouth, Upper Hutt, Wairarapa affected.


You are late. This was being discussed already. People should REALLY READ THE FORUM to make sure they are not repeating posts.


Whoa there, Mauricio.  You tweeted this URL. Folks are going to come straight here without reading the rest of the forum.  Makes it something slightly less offensive, perhaps?





536 posts

Ultimate Geek
+1 received by user: 37


  Reply # 300305 19-Feb-2010 09:45
Send private message

OK some questions for the cellular geeks - you know who your are (johnr, nealr and sbiddle).

In the course of the discussions I think we have established that you can't have fully redundant RNCs (e.g. an RNC on hot standby.

1.  The networks seem to operate with RNCs covering geographical areas.  Is there a reason (say handoffs) that cellsites cannot be allocated to RNCs in a checkerboard pattern instead?  That way (to use a cheese analogy) if you lost an RNC you would turn your coverage area into swiss cheese, rather than loosing half the block.  Since many of cellsites are built for capacity rather than coverage, if you lost half your cellsites in a checkerboard pattern you might still be left with say 2/3 of your coverage area.

2.  If the above is not possible, Telecom effectively operate 2 W-CDMA networks, one at 850 MHz and one at 2100 MHz.  I have read somewhere that the 2100 network is mainly for inbound roaming and not for capacity, thus the issues with having the 850 and 2100 networks on different RNCs may not be so bad.  Would it be possible to run the 850 and 2100 sites off different RNCs (e.g the Auckland RNC is responsible for the 850 sites north of Taupo and the 2100 sites south of Taupo, and the CHC RNC is the other way around), so that both the 850 and 2100 sites weren't subject to the same failures.  This would at least give some redundancy for the main centres.

The possible downfall of the above suggestions is that if traffic issues have caused the problems, then these could cascade in as there would be a significant traffic increase on the remaining RNC(s) when one RNC went down.

exportgoldman:
This article makes interesting reading in hindsight, it was published last year before XT went live


Can Alcatel deliver?
http://www.stuff.co.nz/technology/711898

A quote from the article

"ABN Amro analyst Geoff Zame noted in a research report in July that Alcatel-Lucent had had a hand in only one of the 24 W-850 networks built around the world – for AT&T in the United States.
"We remain uncertain about the merits of Telecom's reliance on [principally] one vendor and we believe Alcatel-Lucent's market position in W-850 increases execution risk," he said in a subsequent research report."


I think this is a red herring as none of the issues with the XT network as far as I know (apart from the initial interference issue which delayed the launch) have had anything to do with using 850 MHz on the air interface.  It would appear to me that RNCs are common to all W-CDMA networks (850, 900 and 2100).  The track record of ALU on all W-CDMA networks is probably of bigger relevance now, given the issues they seem to be having.

19282 posts

Uber Geek
+1 received by user: 2600
Inactive user


  Reply # 300316 19-Feb-2010 10:30
Send private message

1) The transmission path from RNC to Node B is fixed (Iub)

2) The operating Freq is at Node B not the RNC having it split across 2 RNCs would be a nightmare inter RNC singnalling would be huge

John

25691 posts

Uber Geek
+1 received by user: 5447

Moderator
Trusted
Biddle Corp
Subscriber

  Reply # 300332 19-Feb-2010 11:38
Send private message

wongtop: OK some questions for the cellular geeks - you know who your are (johnr, nealr and sbiddle).

In the course of the discussions I think we have established that you can't have fully redundant RNCs (e.g. an RNC on hot standby.

1.  The networks seem to operate with RNCs covering geographical areas.  Is there a reason (say handoffs) that cellsites cannot be allocated to RNCs in a checkerboard pattern instead?  That way (to use a cheese analogy) if you lost an RNC you would turn your coverage area into swiss cheese, rather than loosing half the block.  Since many of cellsites are built for capacity rather than coverage, if you lost half your cellsites in a checkerboard pattern you might still be left with say 2/3 of your coverage area.

2.  If the above is not possible, Telecom effectively operate 2 W-CDMA networks, one at 850 MHz and one at 2100 MHz.  I have read somewhere that the 2100 network is mainly for inbound roaming and not for capacity, thus the issues with having the 850 and 2100 networks on different RNCs may not be so bad.  Would it be possible to run the 850 and 2100 sites off different RNCs (e.g the Auckland RNC is responsible for the 850 sites north of Taupo and the 2100 sites south of Taupo, and the CHC RNC is the other way around), so that both the 850 and 2100 sites weren't subject to the same failures.  This would at least give some redundancy for the main centres.



Both your suggestions come against a fundamental problem - if you're handing a call between multiple RNCs it generates significant signalling traffic between them. Right now this only occurs when a phone changes from the Auckland RNC to the Christchurch RNC. Building a network where a significant number of handsets would be jumping between RNCs continually would require massive connectivity between RNCs, if it was even possible at all.

The iub's (the links from the cellites to the RNC) are essentially grouped together and load balanced when they connect to the RNC. This is why the outage last month affected customers in not only the South Island but also some sites in Wellington and both sites in Mt Ruapehu. If one of these "load balancers" fails then it's not necessarily a single geographic area that is taken out entirely.

The key thing here is that an RNC simply should not fail. Disaster recovery and risk mitigation have to be factored into everything, however you can only go so far.

 1 | 2
View this topic in a long page with up to 500 replies per page Create new topic



Twitter »

Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





News »

Mobile market competition issues ComCom should watch
Posted 18-Dec-2017 10:52


New Zealand government to create digital advisory group
Posted 16-Dec-2017 08:47


Australia datum changes means whole country moving 1.8 metres north-east
Posted 16-Dec-2017 08:39


UAV Traffic Management Trial launching today in New Zealand
Posted 12-Dec-2017 16:06


UFB connections pass 460,000
Posted 11-Dec-2017 11:26


The Warehouse Group to adopt IBM Cloud to support digital transformation
Posted 11-Dec-2017 11:22


Dimension Data peeks into digital business 2018
Posted 11-Dec-2017 10:55


2018 Cyber Security Predictions
Posted 7-Dec-2017 14:55


Global Govtech Accelerator to drive public sector innovation in Wellington
Posted 7-Dec-2017 11:21


Stuff Pix media strategy a new direction
Posted 7-Dec-2017 09:37


Digital transformation is dead
Posted 7-Dec-2017 09:31


Fake news and cyber security
Posted 7-Dec-2017 09:27


Dimension Data New Zealand strengthens cybersecurity practice
Posted 5-Dec-2017 20:27


Epson NZ launches new Expression Premium Photo range
Posted 5-Dec-2017 20:26


Eventbrite and Twickets launch integration partnership in Australia and New Zealand
Posted 5-Dec-2017 20:23



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.