Telecom broadband down again 2012-12-11 21:45

Forums › Spark New Zealand (including Skinny and BigPipe) › Telecom broadband down again 2012-12-11 21:45

1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | ... | 13

1201 posts

Uber Geek
+1 received by user: 402

Lifetime subscriber

#731097 12-Dec-2012 08:55

surfisup1000:

My opinion is that this is a mistake on their behalf for poor planning -- it is not a routine hardware failure which we all accept can occur from time to time but still should be minimised through redundancy.

from the PR release above it seems they do have redundancy, but that is what is causing the problem :)

When the AKL server crashed due to an upgrade ( yesterdays AM outage) it become out of sync with the CHC server. Then when they tried to restore the redundancy ( so everyone was not trying to use the same server) overload occurred and the CHC server had to be taken offline.

With out the redundancy I would say that there would have been no internet for Telecom users at all yesterday AM :)

edit: fixed grammar and spelling, need coffee :)

sbiddle

30853 posts

Uber Geek
+1 received by user: 9996

Retired Mod

Trusted

Biddle Corp

Lifetime subscriber

#731102 12-Dec-2012 08:58

surfisup1000:
My opinion is that this is a mistake on their behalf for poor planning -- it is not a routine hardware failure which we all accept can occur from time to time but still should be minimised through redundancy.

While you're entitled to an opinion I can assure you that your viewpoint is entirely incorrect.

surfisup1000

5288 posts

Uber Geek
+1 received by user: 2159

#731107 12-Dec-2012 09:05

clinty:
surfisup1000:

My opinion is that this is a mistake on their behalf for poor planning -- it is not a routine hardware failure which we all accept can occur from time to time but still should be minimised through redundancy.

from the PR release above it seems they do have redundancy, but that is what is causing the problem :)

When the AKL server crashed due to any upgrade ( yesterdays AM outage) it become out of sync with the CHC server. Then when they tried to restore the restore the redundancy ( so everyone was not trying to use the same server) overload occurred and the CHC server had to be taken offline.

With out the redundancy I would say that there would have been no internet for Telecom suers at all yesterday AM :)

From what I understand , Telecom have 2 PPPOA authenticating servers and their redundancy plan is that if one goes down the other can cope.

But, it appears, they need 2 servers for normal operations. Any less, and the system breaks down.

So, telecom have made a planning error in thinking one server can cope.

Does that sound about right?

surfisup1000

5288 posts

Uber Geek
+1 received by user: 2159

#731109 12-Dec-2012 09:06

sbiddle:
surfisup1000:
My opinion is that this is a mistake on their behalf for poor planning -- it is not a routine hardware failure which we all accept can occur from time to time but still should be minimised through redundancy.

While you're entitled to an opinion I can assure you that your viewpoint is entirely incorrect.

Well, all I know is that my broadband is down again, and that some telecom migration is at fault.

Are you saying telecom have not made any error here?

clinty

1201 posts

Uber Geek
+1 received by user: 402

Lifetime subscriber

#731111 12-Dec-2012 09:11

surfisup1000:
clinty:
surfisup1000:

My opinion is that this is a mistake on their behalf for poor planning -- it is not a routine hardware failure which we all accept can occur from time to time but still should be minimised through redundancy.

from the PR release above it seems they do have redundancy, but that is what is causing the problem :)

When the AKL server crashed due to any upgrade ( yesterdays AM outage) it become out of sync with the CHC server. Then when they tried to restore the restore the redundancy ( so everyone was not trying to use the same server) overload occurred and the CHC server had to be taken offline.

With out the redundancy I would say that there would have been no internet for Telecom suers at all yesterday AM :)

From what I understand , Telecom have 2 PPPOA authenticating servers and their redundancy plan is that if one goes down the other can cope.

But, it appears, they need 2 servers for normal operations. Any less, and the system breaks down.

So, telecom have made a planning error in thinking one server can cope.

Does that sound about right?

I would say that one server would cope fine as it is did for around 6-7 hours yesterday according to the release. But the usual plan in these scenarios is to restore redundancy as soon as possible because lightning can strike twice :)

The issue is that when restoring the AKL server it appears to have caused an issue with the CHC server and crashing the system again.

I would expect that Telecom will now be reviewing how they restore the redundancy between these servers when something happens

GJB21

55 posts

Master Geek

#731116 12-Dec-2012 09:14

Yay, I think we are finally going on broadband at last (9.05am) but won't hold my breath....
Harks back to the XT debacles of a few Christmases ago - 3 major outages in 24 hours - we know Telecom considers it a "best efforts" service, but this hardly qualifies as even an "attempt" does it?

Geekzone support

Support Geekzone with one-off or recurring donations Donate via PressPatron.

clinty

1201 posts

Uber Geek
+1 received by user: 402

Lifetime subscriber

#731119 12-Dec-2012 09:16

Please note this is all from the PR release posted above, I have no actual knowledge of what is happening at Telecom :)

JonoNZ

274 posts

Ultimate Geek
+1 received by user: 43

#731121 12-Dec-2012 09:17

clinty:
surfisup1000:
clinty:
surfisup1000:

My opinion is that this is a mistake on their behalf for poor planning -- it is not a routine hardware failure which we all accept can occur from time to time but still should be minimised through redundancy.

from the PR release above it seems they do have redundancy, but that is what is causing the problem :)

When the AKL server crashed due to any upgrade ( yesterdays AM outage) it become out of sync with the CHC server. Then when they tried to restore the restore the redundancy ( so everyone was not trying to use the same server) overload occurred and the CHC server had to be taken offline.

With out the redundancy I would say that there would have been no internet for Telecom suers at all yesterday AM :)

From what I understand , Telecom have 2 PPPOA authenticating servers and their redundancy plan is that if one goes down the other can cope.

But, it appears, they need 2 servers for normal operations. Any less, and the system breaks down.

So, telecom have made a planning error in thinking one server can cope.

Does that sound about right?

I would say that one server would cope fine as it is did for around 6-7 hours yesterday according to the release. But the usual plan in these scenarios is to restore redundancy as soon as possible because lightning can strike twice :)

The issue is that when restoring the AKL server it appears to have caused an issue with the CHC server and crashing the system again.

I would expect that Telecom will now be reviewing how they restore the redundancy between these servers when something happens

I have some experience in enterprise network infrastructure and to only have two PPPOA authentication servers for an operation the size of Telecom seems a bit light to me. There are all manner of DR scenarios which can easily be predicted where you can run into problems. That is, if this really is how the infrastructure is architected.

JonoNZ

274 posts

Ultimate Geek
+1 received by user: 43

#731122 12-Dec-2012 09:18

Anyway, thank goodness XT isn't affected and the ability to tether over the iPad and iPhone!

Talkiet

4819 posts

Uber Geek
+1 received by user: 3934

Trusted

#731124 12-Dec-2012 09:18

GJB21: Yay, I think we are finally going on broadband at last (9.05am) but won't hold my breath....
Harks back to the XT debacles of a few Christmases ago - 3 major outages in 24 hours - we know Telecom considers it a "best efforts" service, but this hardly qualifies as even an "attempt" does it?

I can assure you that the people working on resolving this fault probably haven't worked this hard or under this much pressure in their life. While consumer broadband doesn't carry any SLAs with it, it is important, and when an outage affects this many people at once, EVERYONE involved downs tools on everything else and gives this their complete attention.

It is a complex fault, and people are working very very hard to identify the best way to bring the affected services back up and keep them stable.

To suggest this isn't even an "attempt" is disrespectful to those working on the issue.

Regards
N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

clinty

1201 posts

Uber Geek
+1 received by user: 402

Lifetime subscriber

#731125 12-Dec-2012 09:21

JonoNZ:
clinty:
surfisup1000:
clinty:
surfisup1000:

My opinion is that this is a mistake on their behalf for poor planning -- it is not a routine hardware failure which we all accept can occur from time to time but still should be minimised through redundancy.

from the PR release above it seems they do have redundancy, but that is what is causing the problem :)

When the AKL server crashed due to any upgrade ( yesterdays AM outage) it become out of sync with the CHC server. Then when they tried to restore the restore the redundancy ( so everyone was not trying to use the same server) overload occurred and the CHC server had to be taken offline.

With out the redundancy I would say that there would have been no internet for Telecom suers at all yesterday AM :)

From what I understand , Telecom have 2 PPPOA authenticating servers and their redundancy plan is that if one goes down the other can cope.

But, it appears, they need 2 servers for normal operations. Any less, and the system breaks down.

So, telecom have made a planning error in thinking one server can cope.

Does that sound about right?

I would say that one server would cope fine as it is did for around 6-7 hours yesterday according to the release. But the usual plan in these scenarios is to restore redundancy as soon as possible because lightning can strike twice :)

The issue is that when restoring the AKL server it appears to have caused an issue with the CHC server and crashing the system again.

I would expect that Telecom will now be reviewing how they restore the redundancy between these servers when something happens

I have some experience in enterprise network infrastructure and to only have two PPPOA authentication servers for an operation the size of Telecom seems a bit light to me. There are all manner of DR scenarios which can easily be predicted where you can run into problems. That is, if this really is how the infrastructure is architected.

I agree. I assume the PR release has been dumbed down a bit and the underlying architecture is actually more complex than one server in each city, but for discussion purposes i kept using the same terms :)

HP

Shop now for HP laptops and other devices (affiliate link).

surfisup1000

5288 posts

Uber Geek
+1 received by user: 2159

#731127 12-Dec-2012 09:24

JonoNZ:
I have some experience in enterprise network infrastructure and to only have two PPPOA authentication servers for an operation the size of Telecom seems a bit light to me. There are all manner of DR scenarios which can easily be predicted where you can run into problems. That is, if this really is how the infrastructure is architected.

I have no idea :) Telecom have printed a little bit of info about this.

User 'biddle' seems to know what has gone wrong.

Maybe it is just the within the normal range of occasional problems.

JonoNZ

274 posts

Ultimate Geek
+1 received by user: 43

#731128 12-Dec-2012 09:26

Talkiet:
GJB21: Yay, I think we are finally going on broadband at last (9.05am) but won't hold my breath....
Harks back to the XT debacles of a few Christmases ago - 3 major outages in 24 hours - we know Telecom considers it a "best efforts" service, but this hardly qualifies as even an "attempt" does it?

I can assure you that the people working on resolving this fault probably haven't worked this hard or under this much pressure in their life. While consumer broadband doesn't carry any SLAs with it, it is important, and when an outage affects this many people at once, EVERYONE involved downs tools on everything else and gives this their complete attention.

It is a complex fault, and people are working very very hard to identify the best way to bring the affected services back up and keep them stable.

To suggest this isn't even an "attempt" is disrespectful to those working on the issue.

Regards
N

Consumer broadband "important"? It almost sounds like it isn't all that important with a statement like that. Most of us don't care if it has an SLA or not, we just want it to work most of the time.

Some network monitoring is lacking because during this whole incident the status and page and twitter account were some of the last to know that the service was out. This morning we were assured via the status it was ok but it clearly was not.

We understand the pressure, many of are under pressure in IT everyday. Let's hope you get it resolved quickly or you'll be facing an unenviable PR disaster.

Talkiet

4819 posts

Uber Geek
+1 received by user: 3934

Trusted

#731131 12-Dec-2012 09:34

Just to clarify for all - the status updates on our website bear no relation to the operational staff that picked up and started working on this issue minutes after it (re)occurred. The service was under CONSTANT monitoring last night after it was initially restored.

I agree that the status updates were slack and have already asked the product manager to include this issue in the Post Issue Review process.

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

Asrafrate

276 posts

Ultimate Geek
+1 received by user: 19

#731132 12-Dec-2012 09:36

Talkiet not at all questioning the dedication the staff have to getting this fixed. A huge thanks to them for it :)

Zeb A.
Twitter: @asgard

1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | ... | 13