michaelmurfy:

 

At 1pm yesterday there was a key rollover for the ac.nz domain and it appears providers are noting some issues with net.nz. I'm seeing issues with co.nz currently on DNS servers unrelated to 2degrees. It should all hopefully be resolved soon if this is the cause: https://status.internetnz.nz/ 

 

Direct link to this fault: InternetNZ Status - DNSSEC chain validation issue for .nz 2LD

ISPs still showing this problem need to clear their caches.

Considering the whopping great charges they have for .NZ domains, you would think the least they could do is at least have some kind of monitoring in-place. It is wholly unacceptable to have virtually every .NZ domain down for 8~ hours.

boosacnoodle:

 

Considering the whopping great charges they have for .NZ domains, you would think the least they could do is at least have some kind of monitoring in-place. It is wholly unacceptable to have virtually every .NZ domain down for 8~ hours.

 

What actually happened is a lot more complicated than just requiring monitoring. The actual issue has been resolved for some time - the problem lies with providers supplying DNS recursors to their customers, and the TTLs involved with DNS. it's complicated. and without the full picture, it's unwise to make armchair accusations.

I noticed a lot of posts on Facebook from various companies about their phone systems being down yesterday. Some I think are still experiencing issues today. Could that also be related to this?

SMTP2GO was having delivery issues that look to be linked to this according to them, flowing again now but holy hell what a mess

InternetNZ apologises for security mishap disrupting access to many .nz websites - https://www.stuff.co.nz/business/132185814/internetnz-apologises-for-security-mishap-disrupting-access-to-many-nz-websites

Internet NZ has a write-up of the incident here.

DNSSEC is essentially a way of verifying that the DNS record you have gotten (usually from your ISP in most cases) is actually the same one the authoritative name server (in this case, InternetNZ for any domain under .nz ie. co.nz, ac.nz etc; which is the first step towards getting domains you are used to seeing: trademe.co.nz, massey.ac.nz) has ok'ed using the magic of cryptography.

What happened was a routine process involving replacing a particular crypto-key called the Key-signing Key (KSK) for a new one got borked, so it invalidated any legit records. Unfortunately with DNS, this isn't a case of clicking the undo button: the horse has bolted and is already rampaging around town. 

Imagine... you are buying some beer from Countdown and as you are getting ID checked, the supervisor notices your driver's license actually has a misprint and is issued by the New Zaland Government. The supervisor says "hey pal, you look old enough to buy beer, but I can't actually verify that that ID is legit, so no dice".

You go home, temporarily defeated and grumbling about bureaucracy... so you contact the NZTA, get them to issue you a new license, and come back to buy that same beer just to wave it in that jumped-up rapscallion's face, to which they are like "whatever" and you finally walk out with your beer, triumphant over the tyranny of Countdown.

I'm sure someone could drive a bus through the holes in this metaphor, but... you get the idea

michaelmurfy:

 

I actually suspect something else is going on. I had my monitoring host alert that a bunch of sites were down (couldn't resolve DNS). Corrected by re-configuring that host to use Cloudflare DNS. Note, this isn't on 2degrees.

At 1pm yesterday there was a key rollover for the ac.nz domain and it appears providers are noting some issues with net.nz. I'm seeing issues with co.nz currently on DNS servers unrelated to 2degrees. It should all hopefully be resolved soon if this is the cause: https://status.internetnz.nz/ 

 

Thanks for your reply last night, I tried to reply via mobile 4G but even that stopped working it seems in the early hours.