Large Chorus Outage affecting all ISPs 19.05.14

Forums › New Zealand Broadband › Large Chorus Outage affecting all ISPs 19.05.14

BigPipeNZ

1170 posts

Uber Geek

Trusted

BigPipe

# 145468 19-May-2014 09:38

Surprised nobody has posted this here yet, but in the interest of transparency I think it is useful to note that chorus is having some major issues right now with provisioning and faults systems, which means no new connections can be processed and no faults can be logged or worked on. This is affecting all ISPs who interact with Chorus - which is basically everybody.

Sucks for us as an ISP, and sucks for anybody in the midst of transitioning broadband providers as this will either be delayed, or, if not done properly, cannot be escalated and fixed.

Email from Chorus:

Good morning, Chorus continue to experience an outage of various internal systems, including OFM, SPM, SSP, Chorus Portal and our inbound phone queues company-wide. Currently we are unable to complete any provisioning or fault restoration work. Please phone through any medical emergencies via the following escalation channels;

We do not have an ETA for service restoration, however our technical teams are investigating and we endeavour to keep you updated

ETA: Confirmed that this is indeed infrastructure that Chorus and Telecom still share post demerger that has gone haywire, so it's definitely related.

Not trying to bag Chorus here, just trying to be transparent about what things aren't working and why.
Looks like the issue is (mostly) fixed now anyway (as at 5.50pm Monday 19.05).

(we're not impacted by the Telecom component of the outage as we are on totally separate systems, but we are impacted by this Chorus outage)

bigpipe.co.nz
https://www.facebook.com/BigPipeNZ
https://twitter.com/BigPipeNZ

dan

1148 posts

Uber Geek

Lifetime subscriber

# 1047831 19-May-2014 09:42

hmm, got a line and broadband install happening in 20 mins, chrous tech didnt mention anything about it

BigPipeNZ

1170 posts

Uber Geek

Trusted

BigPipe

# 1047833 19-May-2014 09:45

dan: hmm, got a line and broadband install happening in 20 mins, chrous tech didnt mention anything about it

I'm guessing here, but physical work already planned can likely still go ahead (i.e. techs in vans doing stuff) but faults cannot be logged if he stuffs it up, and new orders cannot be placed.

Also note that the techs aren't directly employed by Chorus, they are employed by a sub-contractor (e.g. Visionstream) who have their own systems, so potentially may not be affected as much.

bigpipe.co.nz
https://www.facebook.com/BigPipeNZ
https://twitter.com/BigPipeNZ

sbiddle

28574 posts

Uber Geek

Moderator

Trusted

Biddle Corp

Lifetime subscriber

# 1047834 19-May-2014 09:45

2 people support this post

It's worth noting it's not actually a Chorus fault - it's a total meltdown of Telecom's internal network which has taken Telecom pretty much offline.

Many Chorus systems are still legacy and hosted by Telecom hence the problem.

Inphinity

2547 posts

Uber Geek

# 1047836 19-May-2014 09:46

Sounds like someone's gonna be crying themselves to sleep over their stuffup :P

DjShadow

3104 posts

Uber Geek

Trusted

# 1047839 19-May-2014 09:49

Would it be related to that upgrade in the weekend affecting mobile?

sbiddle

28574 posts

Uber Geek

Moderator

Trusted

Biddle Corp

Lifetime subscriber

# 1047842 19-May-2014 09:51

DjShadow: Would it be related to that upgrade in the weekend affecting mobile?

No idea. Telecom internally has pretty much ground to a halt with pretty much every system down from what I've heard.

freitasm

BDFL - Memuneh
65304 posts

Uber Geek

Administrator

Trusted

Geekzone

Lifetime subscriber

# 1047843 19-May-2014 09:52

sbiddle: It's worth noting it's not actually a Chorus fault - it's a total meltdown of Telecom's internal network which has taken Telecom pretty much offline.

Many Chorus systems are still legacy and hosted by Telecom hence the problem.

Please follow the Telecom Meltdown here.

surfisup1000

4470 posts

Uber Geek

# 1047844 19-May-2014 09:53

Maybe you mean all subscribers connected via chorus?

I'm with UFF (via snap) and they seem just fine.

sbiddle

28574 posts

Uber Geek

Moderator

Trusted

Biddle Corp

Lifetime subscriber

# 1047847 19-May-2014 09:57

surfisup1000:
Maybe you mean all subscribers connected via chorus?

I'm with UFF (via snap) and they seem just fine.

The issue is nothing to do with Chorus provided ISP services. If you have internet through a Chorus wholesale connection you will not be affected unless you're with Telecom where you may be affected due to their DNS outage.

The issue is with Telecom's internal network affecting Chorus backend systems. Most of the Chorus systems are legacy products still run on Telecom infrastructure so are affected by this outage.

DarthKermit

5066 posts

Uber Geek

Trusted

# 1047950 19-May-2014 12:29

13 people support this post

Have Telecom installed a master splitter yet? That'll fix everything.

Whatifthespacekeyhadneverbeeninvented?

BigPipeNZ

1170 posts

Uber Geek

Trusted

BigPipe

# 1048268 19-May-2014 17:49

sbiddle:
surfisup1000:
Maybe you mean all subscribers connected via chorus?

I'm with UFF (via snap) and they seem just fine.

The issue is nothing to do with Chorus provided ISP services. If you have internet through a Chorus wholesale connection you will not be affected unless you're with Telecom where you may be affected due to their DNS outage.

The issue is with Telecom's internal network affecting Chorus backend systems. Most of the Chorus systems are legacy products still run on Telecom infrastructure so are affected by this outage.

Yep, confirmed that this is the case. It's infrastructure that Chorus and Telecom still share post demerger that has gone haywire.

Not trying to bag Chorus here, just trying to be transparent about what things aren't working and why.

Looks like the issue is (mostly) fixed now.

bigpipe.co.nz
https://www.facebook.com/BigPipeNZ
https://twitter.com/BigPipeNZ