freitasm: Just received :

TELECOM UPDATE ON SERVICE ISSUES - 5.20PM

Telecom has now resolved a hardware failure and our call centres, website and other online customer support are now available. Telecom apologises to all those customers who have been affected in any way by today’s problems.

During today’s outage, calling, text, mobile data and most broadband services have continued as normal.

Earlier today Telecom experienced a significant hardware fault with a vital piece of equipment in our Auckland central data centre.  This fault meant that Telecom’s customer support channels – such as call centres, website, and other online services - were down for most of the day.  Services were restored progressively from early to mid-afternoon. 

The fault also limited internet access for some corporate customers. Some home broadband customers, whose network settings mean they also rely on the faulty piece of equipment to access the internet, may also have noticed their access to some internet sites was intermittent over the day.

Unfortunately, the fault coincided with upgrade work that Telecom carried out over the weekend on its prepaid mobile network.  This planned work meant that prepaid customers have been temporarily unable to access some Telecom services, such as the ability to top up their phone credit. That work was due to end this morning, but Telecom had to delay bringing these services back on when the equipment fault occurred. 

Telecom now intends to complete the upgrade work on the prepaid mobile system overnight tonight. Telecom assures prepaid customers who are due to top up that they can continue to use texts, call minutes and data as they usually would despite the upgrade issue. 

Telecom has rostered extra call centre staff on this evening to assist with any backlog in customer support needs.

UncleArthur: 

Word I've heard is that it was a switch failure..... pretty poor resiliency when 'a' switch failure can do this to a major ISP.....I would expect this from an SMB, not the countries largest telco.
Time for Simon Mutter to stop cutting to the bone quite so much me thinks.

eXDee: So my question is from a network resiliency point of view - is there any information on what actually went wrong - eg did a core switch/router/etc fail? And if so, what happened to redundancy? Multiple failures? Domino effect? Misconfiguration? What about being geographically independent?
Of course its likely we'll never know, but im genuinely curious - rather than trying to add blame and point the finger, as i know network downtime can happen to any company.

Can always learn lessons from issues like these.

Jase2985: im on post paid and still cant access my account, just keeps asking me to sign in

Dynamic: I looked for about 3 minutes this morning and could not find it either!

Telecom:  Please put a Service Status link on the top and bottom menus to make it easy to find.  Currently it is not intuitive to find, even if it is on the front page!

Jarsky:

Dynamic: I looked for about 3 minutes this morning and could not find it either!

Telecom:  Please put a Service Status link on the top and bottom menus to make it easy to find.  Currently it is not intuitive to find, even if it is on the front page!

I found the same thing actually when you mentioned it. Theres so much color going on, It took me about 20 seconds to actually find it even though its in plain sight.
I think theres just to much distraction with all the pictures and colors, or maybe we are just too use to the way it was before?