Another major outage with Orcon

Forums › 2degrees (including Slingshot, Orcon, Flip, Stuff Fibre, MyRepublic, 2talk and Vocus) › Another major outage with Orcon

openmedia

3332 posts

Uber Geek

Trusted

#69252 5-Oct-2010 12:07

As some of you might have already seen on Twitter - Orcon have posted.

We are experiencing another outage. I don't know the details yet, you'll be the first to know when I do.

Second major outage in less than 12 hours. I can't hit their DNS or any services hosted at their Northcote Colo.

Generally known online as OpenMedia, now working for Red Hat APAC as a Technology Evangelist and Portfolio Architect. Still playing with MythTV and digital media on the side.

1 | 2

879 posts

Ultimate Geek

ID Verified

#388145 5-Oct-2010 12:11

Would explain the 3 calls I just got from customers trying to get me to fix their email... :)

LoJac

17 posts

Geek

#388158 5-Oct-2010 12:26

My Orcon connection has been pretty bad over the past week, slow and frequent dropouts. Using my phone to post this because of this most recent outage...

wazzageek

1093 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#388165 5-Oct-2010 12:37

A bit of a pain - Orcon's main website still works

Ramjet007

319 posts

Ultimate Geek

#388192 5-Oct-2010 12:57

Their webmail and account pages dont work.

getting this now

Error, cannot connect to database: Lost connection to MySQL server at 'reading initial communication packet', system error: 113

wazzageek

1093 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#388242 5-Oct-2010 13:58

Power is off in the cohosting again (From what I understand it's the B circuits without power)

wazzageek

1093 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#388278 5-Oct-2010 14:43

Orcon makes the news over their outage : http://bit.ly/99FipV

robbyp

1199 posts

Uber Geek

#388294 5-Oct-2010 15:07

wazzageek: Orcon makes the news over their outage : http://bit.ly/99FipV

Was that major outage in the early morning caused by Orcon? I thought it was somewhere further upstream, as I'm with Xnet, and they also had the outage with international websites. It was almost as though NZs internet was cut off from teh rest of the world.

Sharesies

Trade NZ and US shares and funds with Sharesies (affiliate link).

Kyanar

4089 posts

Uber Geek

ID Verified

Trusted

#388762 6-Oct-2010 14:30

robbyp:
wazzageek: Orcon makes the news over their outage : http://bit.ly/99FipV

Was that major outage in the early morning caused by Orcon? I thought it was somewhere further upstream, as I'm with Xnet, and they also had the outage with international websites. It was almost as though NZs internet was cut off from teh rest of the world.

Orcon's tweet indicated it was a Vector connectivity outage.

Zeon

3916 posts

Uber Geek

Trusted

#389160 7-Oct-2010 13:24

Any further details on what happened with their own power problem? I think I saw on twitter one of their power distribution boards went down? Do the racks have power from multiple distribution boards?

Would make sense as if you had a server with 2+ PSUs you could plug into a different distribution board so if one failed you could at least stay online.

Reminds me of their problems at Vector's DC that damaged their reputation a few years ago. Worrying as we are looking at getting a 1/3rd rack at their DC!

Speedtest 2019-10-14

freitasm

BDFL - Memuneh
79296 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#389175 7-Oct-2010 13:41

This is what ICONZ sent out to customers today:

ICONZ Vector Fibre and International Connectivity Issues 05/10/2010 00:20 ? 02:30

Incident Time: 12.20am ? 02.30am (approx) Incident Duration: 2 hours 10 minutes (approx)
Impact: Loss of international connectivity (30 mins), Loss of service for Vector Fibre customers (2 hours)

Incident Summary:

On the 5th of October, at approximately 12.20am, ICONZ monitoring systems presented alerts for international connectivity and several Vector Fibre services. The issue was immediately escalated to ICONZ network engineers, who began fault diagnostics.

At approximately 12.50am ICONZ engineers were advised of a power-related fault at a major Vector node in Auckland CBD. This caused service issues for several Vector customers, including ICONZ?s primary supplier of international connectivity, Vocus. The impact to Vocus caused ICONZ to lose international service. Additionally, some ICONZ customers who are connected via Vector experienced a loss of connectivity.

At this time, ICONZ engineers migrated international connectivity to our secondary provider (Global Gateway), restoring international connectivity.

Vector technicians resolved their fault at approximately 2.30am, restoring service to Vocus and other Vector Fibre customers. Upon confirmation of service restoration, ICONZ engineers migrated international connectivity back to Vocus.

ICONZ are liaising with both Vocus and Vector, with a view to preventing further issues of a similar nature. In the meantime we apologise for any inconvenience caused by this unscheduled service outage.

It should be noted that customers may have been unable to access various services hosted on other Service Provider?s networks ? several other providers were also affected by this outage.

Kind Regards,

ICONZ Corporate Support

wazzageek

1093 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#389444 8-Oct-2010 09:36

Zeon: Any further details on what happened with their own power problem? I think I saw on twitter one of their power distribution boards went down? Do the racks have power from multiple distribution boards?

Would make sense as if you had a server with 2+ PSUs you could plug into a different distribution board so if one failed you could at least stay online.

Reminds me of their problems at Vector's DC that damaged their reputation a few years ago. Worrying as we are looking at getting a 1/3rd rack at their DC!

Still no official word from Orcon as to what happened - It may have been one of my tweets you saw :-)

I can only speak from my experience with Orcon - one power distribution board. I understand that you can get a second power distribution board, however whether that is one a second circuit is unknown.

adw

175 posts

Master Geek

#389458 8-Oct-2010 10:14

It also doesn't explain their other outages or their poor service. Is anyone else experiencing major delays in calls being answered? I appreciate they're understaffed at the moment but we've been hearing that for months now. It gets frustrating when you can't get back to your own clients because Orcon aren't answering your questions and start to play the 'we'll ignore it and maybe the problem will go away' - not a great business model guys.

CutCutCut

1039 posts

Uber Geek

#389630 8-Oct-2010 16:20

I had the phone stop working yesterday. The internet was going, at about 1/4 speed. They sent a tech out and it was fixed about 4 hours later. Pretty good service I thought. Although I did have to wait on hold for about 15mins which is annoying.
When Orcon rang to check everything was working ok again, I asked what the tech had done and apparetnly he had repaired a FBV (or a similar assortment of letters) whcih the guy ringing me had no idea what it meant. Possibly related to some of these other issues Orcon are having? I'm on the hibiscus coast.

wazzageek

1093 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#390338 11-Oct-2010 09:16

CutCutCut: I had the phone stop working yesterday. The internet was going, at about 1/4 speed. They sent a tech out and it was fixed about 4 hours later. Pretty good service I thought. Although I did have to wait on hold for about 15mins which is annoying.
When Orcon rang to check everything was working ok again, I asked what the tech had done and apparetnly he had repaired a FBV (or a similar assortment of letters) whcih the guy ringing me had no idea what it meant. Possibly related to some of these other issues Orcon are having? I'm on the hibiscus coast.

That sounds suspiciously like a faulty wiring / CPE device at your premises. Plus, 4hours is probably OK for consumer grade stuff ... but given the total lack of communication from Orcon to it's Datacenter customers ... really was poor of Orcon.

CutCutCut

1039 posts

Uber Geek

#390339 11-Oct-2010 09:24

What's a CPE? How would that come about? I wasn't home but my wife said he was fiddling with the grey tube down by the road and went and then came back a couple of times before disappearing for good. He never came inside. I'm always interested to know what the fault is with things like this/power/water etc even if I don't fully understand it.

1 | 2