Chorus Fibre Outage at 1am

Forums › New Zealand Broadband › Chorus Fibre Outage at 1am

ssamjh

358 posts

Ultimate Geek
+1 received by user: 174

ID Verified

Trusted

#289903 7-Oct-2021 13:56

Did anyone see a Chorus fibre outage just afer 1am?

https://home.sjh.at/smokeping/?displaymode=n;start=2021-10-06%2007:53;end=now;target=DNS.CloudflareDNS1

https://home.sjh.at/worksmokeping/?displaymode=n;start=2021-09-27%2013:57;end=now;target=DNS.CloudflareDNS1 - at work the fibre dropped and it started using the backup DSL

https://home.sjh.at/dcsmokeping/?displaymode=n;start=2021-10-06%2007:53;end=now;target=DNS.CloudflareDNS1 - Alexandra

It affected Voyager, Spark and 2degrees and from what I could tell it affected the Coromandel (Thames and Whitianga), Auckland and even Alexandra down south?

I was watching a TeamViewer account and around the same time computers all around NZ suddenly came back online.

Can't see anything in the news, was this some major coincidence?

Whitianga, Coromandel - Quic Rocket1G - Ref (free setup): R893192EGADLZ

Smokeping: Quic / Voyager / Starlink

Coromandel Webcam Network

1 | 2 | 3 | 4 | 5 | 6 | ... | 7

518 posts

Ultimate Geek
+1 received by user: 196

ID Verified

Trusted

#2791237 7-Oct-2021 13:57

Planned work would be my bet

OzoneNZ

102 posts

Master Geek
+1 received by user: 79

ID Verified

#2791267 7-Oct-2021 14:45

No outage for me on 2d fibre last night, North Shore

Had a few friends affected by the outage covering most of Albany today though

Lias

5655 posts

Uber Geek
+1 received by user: 3978

ID Verified

Trusted

Lifetime subscriber

#2791289 7-Oct-2021 14:56

I vaguely recall one of the network engineers at a former job grumbling about this sort of thing. If memory serves me correctly, anything not on a business grade plan with SLA is subject to a default Chorus maintenance window of 00:00-06:00 or something rather extreme.

I'm a geek, a gamer, a dad, a Quic user, and an IT Professional. I have a full rack home lab, size 15 feet, an epic beard and Asperger's. I'm a bit of a Cypherpunk, who believes information wants to be free and the Net interprets censorship as damage and routes around it. If you use my Quic signup you can also use the code R570394EKGIZ8 for free setup. Opinions are my own and not the views of my employer.

ProbablyAGeek

26 posts

Geek
+1 received by user: 8

#2791297 7-Oct-2021 15:22

It will be planned maintenance, likely a automated software upgrade on a lot of network nodes based on the scope you saw.

Lias:

I vaguely recall one of the network engineers at a former job grumbling about this sort of thing. If memory serves me correctly, anything not on a business grade plan with SLA is subject to a default Chorus maintenance window of 00:00-06:00 or something rather extreme.

It's pretty standard for all ISPs & LFCs to perform maintenance between midnight and 6am. If you need an uptime of 5x9s or more then you engineer your network with that in mind.

Valcor

97 posts

Master Geek
+1 received by user: 37

#2791316 7-Oct-2021 15:45

Yep... Most of Wellington was down at 1pm

Wellingtondave

160 posts

Master Geek
+1 received by user: 88

#2791330 7-Oct-2021 16:06

Valcor:

Yep... Most of Wellington was down at 1pm

What is interesting and not at all helpful is you can use the Spark mobile app to do a connection test on your broadband connection as you're logged into the app with your account (testing using cellular data / another provider obviously) and the connection test says "Fibre network OK, Fibre connection OK, Modem OK" - even if the network is down (PPOE was down this morning, the fibre network was down yesterday).

Planned maintenance or even unplanned events should be listed in the app.

Quic Broadband

Move to New Zealand's best fibre broadband service (affiliate link). Free setup code: R587125ERQ6VE. Note that to use Quic Broadband you must be comfortable with configuring your own router.

ssamjh

358 posts

Ultimate Geek
+1 received by user: 174

ID Verified

Trusted

#2791331 7-Oct-2021 16:09

Wellingtondave:
Valcor:

Yep... Most of Wellington was down at 1pm

What is interesting and not at all helpful is you can use the Spark mobile app to do a connection test on your broadband connection as you're logged into the app with your account (testing using cellular data / another provider obviously) and the connection test says "Fibre network OK, Fibre connection OK, Modem OK" - even if the network is down (PPOE was down this morning, the fibre network was down yesterday).

Planned maintenance or even unplanned events should be listed in the app.

I agree, I couldn't find mention of this outage anywhere.

Whitianga, Coromandel - Quic Rocket1G - Ref (free setup): R893192EGADLZ

Smokeping: Quic / Voyager / Starlink

Coromandel Webcam Network

Linux

12173 posts

Uber Geek
+1 received by user: 8469

Trusted

Lifetime subscriber

#2791334 7-Oct-2021 16:18

Valcor:

Yep... Most of Wellington was down at 1pm

@Valcor No planned work would happen 1pm

Valcor

97 posts

Master Geek
+1 received by user: 37

#2791346 7-Oct-2021 16:42

Sorry I mean 1 am this morning.

There was a reddit post in r/Wellington with people reporting their net had dropped and was from all different providers.

https://www.reddit.com/r/Wellington/comments/q2jd6u/problems_with_spark/

And there was no outage listed the entire time it was down or any planned work on the spark site.

Lias

5655 posts

Uber Geek
+1 received by user: 3978

ID Verified

Trusted

Lifetime subscriber

#2791349 7-Oct-2021 16:53

ProbablyAGeek:

It's pretty standard for all ISPs & LFCs to perform maintenance between midnight and 6am. If you need an uptime of 5x9s or more then you engineer your network with that in mind.

It might be standard, it doesn't really mean it would be acceptable to most consumers or small businesses if they actually knew. Telco's and people in the industry may not want to acknowledge it, but pretty much everyone expects their internet to have zero downtime. That's probably an unreal expectation, but in my experience it's still the expectation nonetheless. Have had actual conversations along the lines of "Power might go out once a year, water once a decade, why isn't internet the same"?

Linux

12173 posts

Uber Geek
+1 received by user: 8469

Trusted

Lifetime subscriber

#2791365 7-Oct-2021 17:11

1am planned work end of story!

What is the purpose of this thread?

Dell

Shop now for Dell laptops and other devices (affiliate link).

dolsen

1483 posts

Uber Geek
+1 received by user: 319

Trusted

Lifetime subscriber

#2791366 7-Oct-2021 17:12

I too spent some time this morning trying to work out what the issue was with my connection, before switching to mobile data.

I don't have too much of an issue with the fact that maintenance was done at that time. What I do have an issue with is no indication that there is maintenance going on / planned at that time. I checked Chorus, Vocus and Sfuff Fibre sites and none of them had any mention of either an outage, or, planned maintenance.

https://www.vocus.co.nz/networkstatus

https://www.chorus.co.nz/outages

https://support.stuff-fibre.co.nz/hc/en-us/articles/360009754473-NETWORK-STATUS-OUTAGE-INFORMATION

No real issue if outages are communicated, but just taking it down with no communication isn't a great experience.

dolsen

1483 posts

Uber Geek
+1 received by user: 319

Trusted

Lifetime subscriber

#2791367 7-Oct-2021 17:16

Linux: 1am planned work end of story!

What is the purpose of this thread?

To express displeasure with the lack of notification given and to hopefully influence who ever undertakes / plans such work in future to better communicate their changes.

The fact that most people weren't using the connection at 1 AM does not mean that no one was. It wasn't a 5 minute outage either.

Linux

12173 posts

Uber Geek
+1 received by user: 8469

Trusted

Lifetime subscriber

#2791389 7-Oct-2021 17:36

dolsen:
Linux: 1am planned work end of story!

What is the purpose of this thread?

To express displeasure with the lack of notification given and to hopefully influence who ever undertakes / plans such work in future to better communicate their changes.

The fact that most people weren't using the connection at 1 AM does not mean that no one was. It wasn't a 5 minute outage either.

Planned work needs to happen so you really need to move on!

This thread needs to be locked by a mod as itt is 100% pointless

dolsen

1483 posts

Uber Geek
+1 received by user: 319

Trusted

Lifetime subscriber

#2791403 7-Oct-2021 17:57

Linux:

Planned work needs to happen so you really need to move on!

Planned work - should be communicated to the affected parties as part of the planning! To not do so is either, poor planning, or, represents a poor attitude towards service delivery.

Whilst I do admit, communicating with all people prior to an outage such as this is not feasible, at least bothering to put it on the scheduled maintenance pages, like I listed in a previous post, would have let people affected know that it is an upstream issue and not to bother local troubleshooting.

Something with this much of an impact doesn't (shouldn't) get released on a whim. There would have been planning, testing, rollback plans long before it was pushed to the network. It is too late for this outage, however, hopefully someone reading this will also include updating the scheduled maintenance pages available to the public for future releases.

1 | 2 | 3 | 4 | 5 | 6 | ... | 7