Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.


View this topic in a long page with up to 500 replies per page Create new topic
1 | 2 | 3
35 posts

Geek


  Reply # 209877 27-Apr-2009 19:05
Send private message

jermsie:
I think that attitude is a bit unreasonable. Orcon identified the problem and did what they could to restore service as soon as possible. An ISP of their size is not likely to sit on their hands while customers remain offline. Cut them some slack and give them a ring if you have any more problems.


I don't think my attitude is unreasonable. They identified the problem and chose to keep from rebooting because a "small" (how small? how many?) number of customers were affected. So the customers that were affected (myself included) were kept offline for 23 hours+.

I doubt you would think this was reasonable if you were one of those affected customers. There were Orcon customers angry after only 1/2 hour of downtime... multiply that time period of 1/2 hour by 46 and you'll see where I'm coming from.



1200 posts

Uber Geek
+1 received by user: 3

Trusted

  Reply # 209878 27-Apr-2009 19:08
Send private message

Sounddude: We ran into a Juniper bug in the software load that was running on one of the ERX's. It would stop forwarding traffic for some customers.

It took a lot of time to debug and was escalated to Juniper as a P1 fault.

In the end, a reboot was the only option to restore services. However it was only affecting a small number of customers, which is why we wanted to keep the LNS running for as long as we could, but in the end we had no choice.



This sounds a lot like the bug which has been haunting XNET's Juniper routers, stopping authentication and requiring a reboot. From the number of times XNET has gone offline they still are waiting for a resolution.

The Junipers seem to have a lot of P1 faults for enterprise class carrier grade kit.




Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.

 
 
 
 




266 posts

Ultimate Geek
+1 received by user: 17

Lifetime subscriber

  Reply # 209895 27-Apr-2009 19:38
Send private message

Juniper..., bless the Layer7 shaping device then.
Looks like Orcon boys shaped some people's connections down to nil.
I think their whole business needs a bit of a reboot.

PS. This is not personal, I simply can't believe a company that behaves that way can call its service great.

BDFL - Memuneh
61746 posts

Uber Geek
+1 received by user: 12411

Administrator
Trusted
Geekzone
Lifetime subscriber

  Reply # 209897 27-Apr-2009 19:40
Send private message

nexttothemoon: I doubt you would think this was reasonable if you were one of those affected customers. There were Orcon customers angry after only 1/2 hour of downtime... multiply that time period of 1/2 hour by 46 and you'll see where I'm coming from.


You can't make everyone happy.

nexttothemoon was affected by a bug and had no service. To fix the issue a reboot was needed, which obviously made dauckland unhappy...

Orcon is between a rock and a hard place. And I think the reboot would be the way to go.

So the balance is 30 minutes outage (duackland unhappy) or a late reboot (nexttothemoon unhappy)... There's no way for Orcon to win...







266 posts

Ultimate Geek
+1 received by user: 17

Lifetime subscriber

  Reply # 209915 27-Apr-2009 20:10
Send private message

Mauricio.

How about: well configured network, responsible technical team, NO reboot required and everyone's happy??!!
Too much to ask?

BDFL - Memuneh
61746 posts

Uber Geek
+1 received by user: 12411

Administrator
Trusted
Geekzone
Lifetime subscriber

  Reply # 209924 27-Apr-2009 20:19
Send private message

dauckland, apparently this was a problem with one or more equipment... There are problems that you can't go around even with redundant equipments.

For example a couple of months ago a major hub in the U.S. came down because most of its traffic was rubish. Rebooting one at a time to keep the network wouldn't do the job because without a firmware update the routers started spilling out bad formed packets and immediately causing the other routers already rebooted to go bad again.

At some point a 30 minutes outage is not the worse thing in the world.





1200 posts

Uber Geek
+1 received by user: 3

Trusted

  Reply # 209930 27-Apr-2009 20:31
Send private message

dauckland: Mauricio.



How about: well configured network, responsible technical team, NO reboot required and everyone's happy??!!

Too much to ask?



I love your wide eyed innocence, building large, reliable networks for the lowest retail costs with no downtime...

Sometimes you buy the best kit, have the smartest guys, and the most redundant network you can overbuild and still get bitten.

Juniper kit is high end super reliable gear. It's multipath redundant but there was a software bug. These switches take 30 minutes to reboot...

If a 30 minute outage is so critical to avoid your end, why not get two different connections, delivered over two different mediums (DSL/Wireless) via two different providers??




Tyler - Parnell Geek - iPhone 3G - Lenovo X301 - Kaseya - Great Western Steak House, these are some of my favourite things.



266 posts

Ultimate Geek
+1 received by user: 17

Lifetime subscriber

  Reply # 209991 27-Apr-2009 22:04
Send private message

Why in deed, done that a long time ago. But one of our business DSL connections was Orcon, but not for long now.
I switched my home one to Snap and couldn't be happier with service and uptime.

And I understand that it takes time and effort to build a reliable network.
But still, I expect better from an ISP.
I have NEVER heard of a provider taking entire DSL network down during business hours.
And unfortunately this is not the first time Orcon made a mockery of the Juniper switch setup, and probably not the last one either (not sure I believe it was a software bug).

Minimalist
5500 posts

Uber Geek
+1 received by user: 431

Moderator
Trusted
Lifetime subscriber

  Reply # 210000 27-Apr-2009 22:27
Send private message

dauckland: I have NEVER heard of a provider taking entire DSL network down during business hours.


I must have special Orcon interwebs, mine didn't go down at all

222 posts

Master Geek
+1 received by user: 81


  Reply # 210059 28-Apr-2009 07:37
Send private message

Same, mine didn't go down either, possibly it was OK for people who were already authenticated and their connections didn't drop? (I am on a telecom wholesaled port).

dan

986 posts

Ultimate Geek
+1 received by user: 89


  Reply # 210070 28-Apr-2009 08:45
Send private message

neither mine, nor any of my customers using orcon had any issue ( all LLU )

I fix stuff!
1711 posts

Uber Geek
+1 received by user: 381

Trusted
Vocus
Subscriber

  Reply # 210071 28-Apr-2009 08:47
Send private message

scottjpalmer:
dauckland: I have NEVER heard of a provider taking entire DSL network down during business hours.


I must have special Orcon interwebs, mine didn't go down at all



We have multiple BRAS's which support the DSL network. Only one was affected by the Software bug, which is why not everyone was hit by the bug.

We only had to reboot one of the BRAS's to restore the internet for the customers affected by the bug.


Last night we loaded a new software load onto rest of the BRAS's, to make sure we don't have to go though this again.

Again, apoligies for the trouble this has caused. It was a very hard problem to find the cause of, which is why it took so long to realise we had to do a reboot. We tried to hold off until the small hours, but in the end we have little choice.


More than happy to answer questions :)




 

xpd

Chief Trash Bandit
9140 posts

Uber Geek
+1 received by user: 1440

Mod Emeritus
Trusted
Lifetime subscriber

  Reply # 210073 28-Apr-2009 08:48
Send private message

It only affected users connected to Telecom equipment (not all however it seems) - if you were on Orcon+, you wernt affected.

Dauckland, get a grip TBH. Yes, downtime is a sod, but so's life at times, but dont see you going to ask for a new one of those. We had 6 of our offices go offline becuase of yesterdays outage, but we arent complaining. (And yes, the internet is fairly critical for them to operate)

Every time your car breaks down, do you throw it away for a new one ? Or complain to the manufacturer everytime ?







XPD / Gavin / DemiseNZ

 

Server : i3-3240 @ 3.40GHz  16GB RAM  Win 10 Pro    Workstation : i5-xxxx @ x.xxGHz  16GB RAM  Win 10 pro    Console : Xbox One

 

https://www.xpd.co.nz - Games, geeks, and more.    


248 posts

Master Geek

Trusted

  Reply # 210090 28-Apr-2009 09:44
Send private message

Hi guys,

I think Simon (Sounddude) has done a pretty good job of clarifying things, just wanted to clarify a couple more things.

- The outage only affected those customers on Telecom Wholesale connections, not LLU connections
- The outage did not affect ALL Telecom Wholesale connections, which is why some of you experienced no outage
- We would never schedule planned maintenance for the middle of the day, the router reboot was only undertaken as a last resort because it was necessary to get the affected customers back online. Scheduled maintenance is generally carried out between 1am and 6am in the morning - our lowest traffic period - in order to cause the least possible disruption to peoples service.

Hope that clarifies things. If you have any more questions Simon or I will be happy to address them. Simon is a lot better at explaining the tech stuff than me :)

I would also like to sincerely apologise to all those who this problem affected - I know how frustrating it can be to have your connection to the world offline and so can completely understand how frustrated you are. Rest assured that Simon and the team are always doing everything that they can to ensure that the network is as robust and stable as possible. Unfortunately, as I am sure you are all aware, when it comes to dealing with technology there is no such thing as 100%.


Minimalist
5500 posts

Uber Geek
+1 received by user: 431

Moderator
Trusted
Lifetime subscriber

  Reply # 210098 28-Apr-2009 10:25
Send private message

Thanks Simon and Duncan - the updates and answers are much appreciated by us all.

1 | 2 | 3
View this topic in a long page with up to 500 replies per page Create new topic



Twitter »

Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.