Orcon's 3PM outage

Forums › 2degrees (including Slingshot, Orcon, Flip, Stuff Fibre, MyRepublic, 2talk and Vocus) › Orcon's 3PM outage

1 | 2 | 3

35 posts

Geek

#209877 27-Apr-2009 19:05

jermsie:
I think that attitude is a bit unreasonable. Orcon identified the problem and did what they could to restore service as soon as possible. An ISP of their size is not likely to sit on their hands while customers remain offline. Cut them some slack and give them a ring if you have any more problems.

I don't think my attitude is unreasonable. They identified the problem and chose to keep from rebooting because a "small" (how small? how many?) number of customers were affected. So the customers that were affected (myself included) were kept offline for 23 hours+.

I doubt you would think this was reasonable if you were one of those affected customers. There were Orcon customers angry after only 1/2 hour of downtime... multiply that time period of 1/2 hour by 46 and you'll see where I'm coming from.

exportgoldman

1202 posts

Uber Geek

Trusted

#209878 27-Apr-2009 19:08

Sounddude: We ran into a Juniper bug in the software load that was running on one of the ERX's. It would stop forwarding traffic for some customers.

It took a lot of time to debug and was escalated to Juniper as a P1 fault.

In the end, a reboot was the only option to restore services. However it was only affecting a small number of customers, which is why we wanted to keep the LNS running for as long as we could, but in the end we had no choice.

This sounds a lot like the bug which has been haunting XNET's Juniper routers, stopping authentication and requiring a reboot. From the number of times XNET has gone offline they still are waiting for a resolution.

The Junipers seem to have a lot of P1 faults for enterprise class carrier grade kit.

Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

dauckland

290 posts

Ultimate Geek

Trusted

Lifetime subscriber

#209895 27-Apr-2009 19:38

Juniper..., bless the Layer7 shaping device then.
Looks like Orcon boys shaped some people's connections down to nil.
I think their whole business needs a bit of a reboot.

PS. This is not personal, I simply can't believe a company that behaves that way can call its service great.

freitasm

BDFL - Memuneh
79263 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#209897 27-Apr-2009 19:40

nexttothemoon: I doubt you would think this was reasonable if you were one of those affected customers. There were Orcon customers angry after only 1/2 hour of downtime... multiply that time period of 1/2 hour by 46 and you'll see where I'm coming from.

You can't make everyone happy.

nexttothemoon was affected by a bug and had no service. To fix the issue a reboot was needed, which obviously made dauckland unhappy...

Orcon is between a rock and a hard place. And I think the reboot would be the way to go.

So the balance is 30 minutes outage (duackland unhappy) or a late reboot (nexttothemoon unhappy)... There's no way for Orcon to win...

dauckland

290 posts

Ultimate Geek

Trusted

Lifetime subscriber

#209915 27-Apr-2009 20:10

Mauricio.

How about: well configured network, responsible technical team, NO reboot required and everyone's happy??!!
Too much to ask?

freitasm

BDFL - Memuneh
79263 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#209924 27-Apr-2009 20:19

dauckland, apparently this was a problem with one or more equipment... There are problems that you can't go around even with redundant equipments.

For example a couple of months ago a major hub in the U.S. came down because most of its traffic was rubish. Rebooting one at a time to keep the network wouldn't do the job because without a firmware update the routers started spilling out bad formed packets and immediately causing the other routers already rebooted to go bad again.

At some point a 30 minutes outage is not the worse thing in the world.

exportgoldman

1202 posts

Uber Geek

Trusted

#209930 27-Apr-2009 20:31

dauckland: Mauricio.

How about: well configured network, responsible technical team, NO reboot required and everyone's happy??!!

Too much to ask?

I love your wide eyed innocence, building large, reliable networks for the lowest retail costs with no downtime...

Sometimes you buy the best kit, have the smartest guys, and the most redundant network you can overbuild and still get bitten.

Juniper kit is high end super reliable gear. It's multipath redundant but there was a software bug. These switches take 30 minutes to reboot...

If a 30 minute outage is so critical to avoid your end, why not get two different connections, delivered over two different mediums (DSL/Wireless) via two different providers??

Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

Sharesies kids

Free kids accounts - trade shares and funds (NZ, US) with Sharesies (affiliate link).

dauckland

290 posts

Ultimate Geek

Trusted

Lifetime subscriber

#209991 27-Apr-2009 22:04

Why in deed, done that a long time ago. But one of our business DSL connections was Orcon, but not for long now.
I switched my home one to Snap and couldn't be happier with service and uptime.

And I understand that it takes time and effort to build a reliable network.
But still, I expect better from an ISP.
I have NEVER heard of a provider taking entire DSL network down during business hours.
And unfortunately this is not the first time Orcon made a mockery of the Juniper switch setup, and probably not the last one either (not sure I believe it was a software bug).

scottjpalmer

5973 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#210000 27-Apr-2009 22:27

dauckland: I have NEVER heard of a provider taking entire DSL network down during business hours.

I must have special Orcon interwebs, mine didn't go down at all

cr250bromo

222 posts

Master Geek

#210059 28-Apr-2009 07:37

Same, mine didn't go down either, possibly it was OK for people who were already authenticated and their connections didn't drop? (I am on a telecom wholesaled port).

dan

1134 posts

Uber Geek

Lifetime subscriber

#210070 28-Apr-2009 08:45

neither mine, nor any of my customers using orcon had any issue ( all LLU )

Sounddude

I fix stuff!
1928 posts

Uber Geek

Trusted

2degrees

Lifetime subscriber

#210071 28-Apr-2009 08:47

scottjpalmer:
dauckland: I have NEVER heard of a provider taking entire DSL network down during business hours.

I must have special Orcon interwebs, mine didn't go down at all

We have multiple BRAS's which support the DSL network. Only one was affected by the Software bug, which is why not everyone was hit by the bug.

We only had to reboot one of the BRAS's to restore the internet for the customers affected by the bug.

Last night we loaded a new software load onto rest of the BRAS's, to make sure we don't have to go though this again.

Again, apoligies for the trouble this has caused. It was a very hard problem to find the cause of, which is why it took so long to realise we had to do a reboot. We tried to hold off until the small hours, but in the end we have little choice.

More than happy to answer questions :)

xpd

Geek @ Coastguard NZ
13765 posts

Uber Geek

Retired Mod

ID Verified

Trusted

Lifetime subscriber

#210073 28-Apr-2009 08:48

It only affected users connected to Telecom equipment (not all however it seems) - if you were on Orcon+, you wernt affected.

Dauckland, get a grip TBH. Yes, downtime is a sod, but so's life at times, but dont see you going to ask for a new one of those. We had 6 of our offices go offline becuase of yesterdays outage, but we arent complaining. (And yes, the internet is fairly critical for them to operate)

Every time your car breaks down, do you throw it away for a new one ? Or complain to the manufacturer everytime ?

Gavin / xpd / FastRaccoon / Geek of Coastguard New Zealand

LinkTree

duncanblair

248 posts

Master Geek

Trusted

#210090 28-Apr-2009 09:44

Hi guys,

I think Simon (Sounddude) has done a pretty good job of clarifying things, just wanted to clarify a couple more things.

- The outage only affected those customers on Telecom Wholesale connections, not LLU connections
- The outage did not affect ALL Telecom Wholesale connections, which is why some of you experienced no outage
- We would never schedule planned maintenance for the middle of the day, the router reboot was only undertaken as a last resort because it was necessary to get the affected customers back online. Scheduled maintenance is generally carried out between 1am and 6am in the morning - our lowest traffic period - in order to cause the least possible disruption to peoples service.

Hope that clarifies things. If you have any more questions Simon or I will be happy to address them. Simon is a lot better at explaining the tech stuff than me :)

I would also like to sincerely apologise to all those who this problem affected - I know how frustrating it can be to have your connection to the world offline and so can completely understand how frustrated you are. Rest assured that Simon and the team are always doing everything that they can to ensure that the network is as robust and stable as possible. Unfortunately, as I am sure you are all aware, when it comes to dealing with technology there is no such thing as 100%.

scottjpalmer

5973 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#210098 28-Apr-2009 10:25

Thanks Simon and Duncan - the updates and answers are much appreciated by us all.

1 | 2 | 3