Back up may need a long reboot of router (recommendation is upto 10mins).

Status page says pretty much all should be up and working again now, or restart your router if not.

I'm still waiting for my DHCP session to be torn down. Have had WAN unplugged for a while.

Guess I'll kill the ont too.

Edit: finally got a new session

Still dead

Did an ONT and network reboot, no connection here when all back up and running, so I have turned network off and will try the 10mins like the status page recommends. . . 

No reboots required here (Synology router). IPv6 isn't working any more, but I'm just going to leave it alone and not panic until tomorrow.

Edit: Working again as at Friday morning.

It doesn't matter what ISP you go to, they all have their fair share of issues every now and then. You can basically look in any ISP thread here and find that for yourself.

Not sure what happened here seeing I’ve been out and have only got home but for those thinking of leaving after an outage then that’s totally fine. Who left ANZ after yesterday’s internet banking outage? Who left Kiwibank after theirs the other day also? Who’s not shopped at Spotlight today as their site was down? You can shove a whole lot of money into being as stable as possible but even Cloudflare, Facebook & Google have issues - it doesn’t matter how big you are, you can fall.

As for where the next outage will be? Well, it may be whatever ISP people may move to, who knows. ISP networks are complex.

Looks to be all coming up now though. 

Waited for two sessions of 5 mins, no connection.

Waited for another solid 10 mins, connection is back.

I do notice unstable connections every now and then on quic nowadays though.

My latency's a bit up and down too, but again I'll leave it until tomorrow.

I'd imagine there'd be a bit of reconvergence of routes happening within the Quic network, so expect to see some weirdness until it all settles down :)

Hi all,

Just popped on to say that obviously we've had a crap night, and a bad week to be fair.

There's a lot we need to digest through before putting a postmortem out, and I've personally never seen something so big as this in my life - very proud of the teams efforts tonight, but we're all understandably devastated in what has happened in the first place.

We'll be doing all we can to make this right - but for now, if you're still without internet, please turn off your ONT for 10 minutes before powering back on, as well as restarting (not resetting) your router.

Many Thanks for your ongoing support. Once we've fully digested and reviewed, a postmortem will be available.

🧡

Thanks QUIC :) hopefully your team can sleep tonight and re-group over the next few days.

Cheers for the update. All in good time :)

Never much fun these things! Sending #hugops ;)

How bizarre.. for someone who usually spends all his time in front of a PC I managed to miss this entirely. Looks like it started around the time I decided to do something not involving sitting in front of my PC (rare!) and was restored enough that when I went to watch a movie with my kids after dinner the only weirdness I noticed was a remote plex server being down.

Big P1's always a rough day at the office in IT, hope things settle down for you lads.

how bizarre... I've been using the internet all day and did not have an outage here (north shore Auckland)... well, that's a good thing I guess :)