Spark DNS Issues - Amazing - Broadband Service Alert

Forums › Spark New Zealand (including Skinny and BigPipe) › Spark DNS Issues - Amazing - Broadband Service Alert

1 | ... | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | ... | 21

310 posts

Ultimate Geek
+1 received by user: 22

#1123578 7-Sep-2014 11:35

sbiddle:
techmeister:
Kirdog: Very slow and laggy ingress play on Spark mobile data today and yesterday evening. :(
Just out of Curiosity: Why Voda customers are not affected by that? ( http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=11320100 < some very strange excuse detected here, as my opinion)

Also someone mentioned that its "paid attack", it can be/why not...

Yes I agree, I smell a rat.
Why only Spark customers?

How about the sample fact they have over 50% of NZers as broadband customers, hence they're more than twice the size of the next biggest network? This makes something on this scale so much harder to contain.

I the herald article the Spark guy said a handful of customers did this and they all happened to be Spark customers?

PaulBags

809 posts

Ultimate Geek
+1 received by user: 184
Inactive user

#1123581 7-Sep-2014 11:45

Spark initially (after 13-15hours of "not knowing what the problem was" overnight) just said a handful of customers with malware sending data overseas; clearly not the case but Spark certainly fumbled the ball PR wise. Then it continued, when they did come up with a fix, switching DNS servers, they NEVER put working addresses up on their service status; and blamed their customers instead of their own security or even just the nature of the business.

pohutukawa

197 posts

Master Geek
+1 received by user: 9

#1123589 7-Sep-2014 12:04

sbiddle:
pohutukawa:
The answer to all of this is that if Spark are NZ's largest ISP then they are also making the most money and should be providing for occurrences like this.

Such a comment shows a complete lack of understanding of DNS amplification and the fact it is the scariest thing to hit an ISP.

Exactly how should they be "providing for an occurrence like this" and how should such an attack be prevented? You've clearly got the answers so could be very rick this week.

At the end of the day there are going to be a lot of people trashing Spark and thinking they're experts on the matter. The scary thing is there is absolutely nothing stopping this exact same thing from happening to Spark again next week, or happening to any other ISP in NZ.

I predict a day in the not too distant future where the global internet goes down for a period of time. It's not a matter of if this will happen, but when this will happen.

I both understand the DNS amplification DDOS vector and how to mitigate in as much as such attacks can be. I concede that the issues are not simple and extend to wider questions such as source IP validation, etc.

However, the Spark response has been pretty dismal IMO, in particular keeping customers informed. I acknowledge that I should have made clear was my main point.

pohutukawa ... turning with the seasons ...

dwl

371 posts

Ultimate Geek
+1 received by user: 46

#1123608 7-Sep-2014 13:17

PaulBags:
dwl: ... Workarounds like officially suggesting Google is a brave move.

Except they aren't, service status still suggests to use the dns servers that are the ones actually having issues...

The service status advice was rather inappropriate. For those who use Facebook they will have seen this post which I assume was officially sanctioned:

There is now an update telling users to change back to the addresses which were in that service status advice. However, not everyone will see this so there may be a slight remaining legacy from this issue.

mattwnz

20520 posts

Uber Geek
+1 received by user: 4798

#1123609 7-Sep-2014 13:27

dwl:
PaulBags:
dwl: ... Workarounds like officially suggesting Google is a brave move.

Except they aren't, service status still suggests to use the dns servers that are the ones actually having issues...

The service status advice was rather inappropriate. For those who use Facebook they will have seen this post which I assume was officially sanctioned:

There is now an update telling users to change back to the addresses which were in that service status advice. However, not everyone will see this so there may be a slight remaining legacy from this issue.

Somewhat ironically, it was international traffic that was affected, so I couldn't access facebook anyway while it was out. Don't think they had that info on their own status page, but I believe different people operate the social media vs the actual status page.

I just can't understand why no other ISPs were affected by this, NZ or even around they world. Does telecom aka spark just have the worst luck of any ISP/phone provider? It seems like they get the the worst luck when it comes to Murphies Law, and the name change hasn't helped stop it. Vodafone must have almost a similar number of customers these days, and would suspect that proportionally, they would have the same number of infected computers on their network. I wonder if other ISPs will come out in the next few days to say that they are working on stopping the same sort of thing happening on their networks.

Talkiet

4819 posts

Uber Geek
+1 received by user: 3934

Trusted

#1123611 7-Sep-2014 13:32

I'm hesitant to go into too much detail at this stage since the issue is still under (very) active investigation, but I can say that we have seen symptoms (in terms of DNS response times from all DNS servers and DNS lookup results) abate to normal, expected performance since approx 4am this morning.

We are still seeing a significant amount of unexpected traffic that we are working towards being able to identify as selectively as possible so as not to hurt legitimate users, but at the moment performance is as expected (from our distributed test network stats and from server based metrics like load balancer stats, DNS server CPU/RAM/QPS stats).

There will be no official comment made at this stage about the root cause (at least through this pretty direct technical channel) because it looks like the attack has evolved over time to avoid some of the mitigations put in place early on during the attack.

At this stage, if you're the sort of person comfortable flicking between DNS servers, I would personally recommend switching back to Sparks auto assigned DNS servers, but be ready to flick back to the Google servers should the attack escalate or find a way to dodge current mitigations.

From the (many) people working on this (some of whom have been working for ~32 of the last ~40 hours), we appreciate the understanding that many on this board have displayed in the face of one of the longest sustained attacks we've seen. There's certainly a HUGE amount less of the purely emotional and frankly insulting comments seen on FB and stuff.co.nz comments sections. Many thanks for that.

If anyone using the Spark DNS servers is currently seeing regular DNS lookup failures, I'd appreciate a PM highlighting that.

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

HP

Shop now for HP laptops and other devices (affiliate link).

clinty

1201 posts

Uber Geek
+1 received by user: 402

Lifetime subscriber

#1123612 7-Sep-2014 13:44

Talkiet:There will be no official comment made at this stage about the root cause (at least through this pretty direct technical channel) because it looks like the attack has evolved over time to avoid some of the mitigations put in place early on during the attack.

Which begs the questions, was this a trial run for a larger attack on an bigger overseas ISP or govt? :)

Clint

paulchinnz

Circumspice
796 posts

Ultimate Geek
+1 received by user: 223

Trusted

Lifetime subscriber

#1123615 7-Sep-2014 13:55

Is it just me or are things not entirely back to normal yet... I'm still using Google's DNS, tried going back to automatic DNS and it's still broken.

mikesmac

17 posts

Geek
+1 received by user: 8

#1123616 7-Sep-2014 13:57

The DNS DDOS attack that i have seen involves incorrect configured customer modems. Quite a few modems are shipped with a factory default of having their inbuilt resolver accessible both from the internal network (which is why your pc normally sets your modem/router to be your DNS) as well as the external network. This means that as someone external to your network I can set your modem to be my DNS server -or if i am evil, I can send a large number of DNS requests to your modem. Your modem being what it is, will then forward these queries to your ISPs DNS server.

Basically this means that I can sit in another country send thousands of request to your customers ip address range and watch things fall apart. The ISP DNS server in most cases is not the target, it is just an victim of the additional queries (which are often to non-existent hosts, in existent domains).

As an ISP there are two simple fixes. 1. Fix your customer modems (or regularly check them, then notify them if they have an "open-resolver") 2. Block incoming DNS requests for all customer addresses except those that specifically run/host DNS services.

You can check your Modem to see if it responds to external DNS queries at www.openresolverproject.org

Talkiet

4819 posts

Uber Geek
+1 received by user: 3934

Trusted

#1123617 7-Sep-2014 13:58

paulchinnz: Is it just me or are things not entirely back to normal yet... I'm still using Google's DNS, tried going back to automatic DNS and it's still broken.

We are seeing normal response times against all our DNS servers, and 100% successful lookups. If you're seeing a problem still, please be specific. What is failing?

Cheers -N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

Talkiet

4819 posts

Uber Geek
+1 received by user: 3934

Trusted

#1123618 7-Sep-2014 14:01

mikesmac: [snip]
You can check your Modem to see if it responds to external DNS queries at www.openresolverproject.org

If anyone sees themselves operating as an open resolver, and they DO NOT HAVE PORT 25 UNBLOCKING ENABLED, please let me know.

So if you have Port 25 unblocking enabled, AND you are running an open resolver, you may show up as an open resolver. If either of those clauses are untrue then you shouldn't be showing up.

Cheers - N

ps. Yes I know port 25 isn't DNS.

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

Lego

Shop now for Lego sets and other gifts (affiliate link).

cyril7

9075 posts

Uber Geek
+1 received by user: 2499

ID Verified

Trusted

Subscriber

#1123637 7-Sep-2014 15:18

Hi, just to report back, returned to ns1/2.xtra.co.nz and all normal, if your still having issues then its something YOU are doing wrong.

Neil and the team thanks for getting this backup and running, I am sure you will be doing a lot of corporate/team soul searching as to how best to mitigate this in future, and I am sure other local ISPs will be checking the rigging in a similar manner as this is a big wake up for all.

As for those that that seem keen on kicking the team while down, you clearly don't have any idea of what is being dealt with here, this is serious stuff, some could say more serious than the rants and wants of politics, as TCP is a fundamental to the internet, there is a lot of trust that all players are reasonable and honest, as time moves on this is clearly not the case, look over your shoulder if you have any public facing networks, this applies to any ISP, corporate or independent.

I suspect most other ISPs are just as vunerable, if not more as they have less resources to stack the odds against attack.

Cyril

sbiddle

30853 posts

Uber Geek
+1 received by user: 9996

Retired Mod

Trusted

Biddle Corp

Lifetime subscriber

#1123658 7-Sep-2014 15:44

mikesmac: The DNS DDOS attack that i have seen involves incorrect configured customer modems.

And it's not just some older modems that respond to lookups, there have been plenty of people on here post issues where they've seen excessive traffic because they've decided to disable their firewall.

There is one thread in particular from a year or two from a guy who posted on here who had purposely disabled his firewall and saw no reason why it should be enabled because he had a firewall enabled on his PC.

Jarsky

142 posts

Master Geek
+1 received by user: 39

#1123680 7-Sep-2014 16:17

Talkiet: There's certainly a HUGE amount less of the purely emotional and frankly insulting comments seen on FB and stuff.co.nz comments sections. Many thanks for that.

Sometimes its nice to see there are some appreciative and understanding customers :)

chevrolux

4962 posts

Uber Geek
+1 received by user: 2638
Inactive user

#1123792 7-Sep-2014 19:59

Jarsky:
Talkiet: There's certainly a HUGE amount less of the purely emotional and frankly insulting comments seen on FB and stuff.co.nz comments sections. Many thanks for that.

Sometimes its nice to see there are some appreciative and understanding customers :)

That's nice to see. I was getting so angry seeing the comments on Spark's status on facebook. I realise changing your DNS is not simple for some but holy balls some people were just plain rude.

I feel sorry for the engineers, what a cr4p thing to happen on a weekend!!

1 | ... | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | ... | 21