Spark DNS Issues - Amazing - Broadband Service Alert

Forums › Spark New Zealand (including Skinny and BigPipe) › Spark DNS Issues - Amazing - Broadband Service Alert

1 | ... | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | ... | 21

14941 posts

Uber Geek
+1 received by user: 13956

ID Verified

#1123480 7-Sep-2014 08:03

Do you have some basis for that comment?

pohutukawa

197 posts

Master Geek
+1 received by user: 9

#1123481 7-Sep-2014 08:13

quickymart: Do you have some basis for that comment?

Well currently Spark's DNS is severely hampering Spark's customers' ability to use their internet connection, and nothing since the issue arose as given me any indication that they know what they're doing.

For example, the "help" and "status" info is basically useless.

It would be probably better if they had a domain (hosted by somebody other than Spark) where they just had a message "Something's wrong and we're trying to fix it. Meanwhile, here are two ways to use alternative DNS, depending on your platform" (examples with screenshots, etc.) and an up-to-date status dashboard.

So the "basis" for my comment is (a) Spark is NZ's ISP and they still can't mitigate issues like this despite huge resources and (b) Sparks previous SMTP and email issues have shown clearly that they are second or third-rate as a company and basically are still coasting on the inertia of them being a huge, government-owned monopoly (which is good from a commercial sense, but not for customers).

Also, I have personal experience (and evidence) of a totally broken attitude towards customers, that persists to this day in Spark.

Your experiences?

See for example: https://twitter.com/BenRoss_AKL/status/508346736352325632

pohutukawa ... turning with the seasons ...

sbiddle

30853 posts

Uber Geek
+1 received by user: 9996

Retired Mod

Trusted

Biddle Corp

Lifetime subscriber

#1123483 7-Sep-2014 08:23

pohutukawa:
The answer to all of this is that if Spark are NZ's largest ISP then they are also making the most money and should be providing for occurrences like this.

Such a comment shows a complete lack of understanding of DNS amplification and the fact it is the scariest thing to hit an ISP.

Exactly how should they be "providing for an occurrence like this" and how should such an attack be prevented? You've clearly got the answers so could be very rick this week.

At the end of the day there are going to be a lot of people trashing Spark and thinking they're experts on the matter. The scary thing is there is absolutely nothing stopping this exact same thing from happening to Spark again next week, or happening to any other ISP in NZ.

I predict a day in the not too distant future where the global internet goes down for a period of time. It's not a matter of if this will happen, but when this will happen.

itxtme

2102 posts

Uber Geek
+1 received by user: 557

#1123497 7-Sep-2014 09:06

Does anyone know why this is only affecting Spark and not other ISPs in NZ? From all the information around its a global attack thats basically lead to a DOS on their network, but I am wondering why nobody else seems to be affected like Spark?

MikeB4

MikeB4
18776 posts

Uber Geek
+1 received by user: 12767

ID Verified

Trusted

Subscriber

#1123499 7-Sep-2014 09:07

Lessons....

Don't look for porn.
Educate before you click.
Don't click.

Here is a crazy notion, lets give peace a chance.

dwl

371 posts

Ultimate Geek
+1 received by user: 46

#1123513 7-Sep-2014 09:37

I am sure hindsight and reviews will provide answers (maybe not that public) as to why Spark seemed more affected than others. I totally agree with those posting about this being a complex issue and in many cases it can just be luck how resilient a service is to specific attacks. I don't understand the details but I have heard other scary stories and vulnerabilities that may exist in those other networks that have not yet been exploited.

I think we need to think of those within Spark who have had a very stressful 24+ hours. From my location Smokeping DNS requests I started last night seemed to have found much better performance, noting that last night maybe only 1 in 5 requests were getting through (5 second timeout) in the case where a result was shown - other times no answer at all:

Ultimately customers have the right to complain and chose their provider. What this forum hopefully shows is these are complex systems. Workarounds like officially suggesting Google is a brave move. Earlier this year such DNS requests may have resulted in pulling CDN lots of content from offshore rather than the local CDN and you might imagine the impact in international bandwidth (assuming the local caches don't deliver). Google now seems to better understand Spark addressing but even now it might still point offshore for some local cached content or to a Spark CDN cluster at the other end of the country.

Spark now has a lot of inappropriate DNS entries that will continue to work but not necessarily deliver the best outcome for the customer or their network. I am quite happy to cut them some slack for this type of issue. Sorry guys but I'm glad I wasn't on call for Spark starting from Friday.

Geekzone support

Support Geekzone with one-off or recurring donations Donate via PressPatron.

OnlyJoe

10 posts

Wannabe Geek
+1 received by user: 9

#1123521 7-Sep-2014 10:09

itxtme: Does anyone know why this is only affecting Spark and not other ISPs in NZ? From all the information around its a global attack thats basically lead to a DOS on their network, but I am wondering why nobody else seems to be affected like Spark?

My guess is that Spark didn't have adequate filtering on their DNS servers, and other ISPs do.

With a DNS amplification attack, someone sends a DNS request to a DNS server as a UDP packet with a fake return IP address, this fake IP points back at the server you are trying to attack. This is so you can basically multiply the amount of traffic you are sending at the attacking server, because a DNS response is a lot bigger than a DNS request. Telecom says the attacks are going overseas, so the fake IP addresses will be for overseas sites. What should normally happen on a private DNS server is that any packets with IP addresses that are not controlled by them should be filtered out and disregarded. As a private DNS should only be sending responses to its users, not to addresses overseas.
This filtering should happen at a very early stage, and not even let the packet hit the actual DNS server. But the tricky part is that because DNS uses UDP it s very very hard to work out where the traffic actually came from, so you basically just have to absorb it on the server. Which means that if you were getting a really really large amount it might overload your filter, which could stop it working for everyone. Spark is saying that they have a lot of new fiber customers who have a massive amount of upstream bandwidth, this means these customers could send an enormous amount of requests compared to someone on regular broadband. So this leads me to think that for Spark the amount of requests they were getting overloaded their filtering system, where as for other ISPs it didn't. So basically it is in part Sparks fault for not having the capacity to deal with these kind of problems, which as been amplified by the increase in viruses from people taking advantage of nude celebrity leak. Or Spark didn't have filtering at all. So last night they basically increased the capacity, or added filtering, to a level where they can handle requests again.

PaulBags

809 posts

Ultimate Geek
+1 received by user: 184
Inactive user

#1123536 7-Sep-2014 10:34

dwl: ... Workarounds like officially suggesting Google is a brave move.

Except they aren't, service status still suggests to use the dns servers that are the ones actually having issues...

mdav056

616 posts

Ultimate Geek
+1 received by user: 160

Subscriber

#1123556 7-Sep-2014 10:41

As a not-very-geeky person, I wondered: Most people's DNS server addresses are set on Auto (I've just put mine back to that from Google), but is this "auto" set by the modem or by Spark servers? If the latter, then why could Spark not set everyone's machines to use Google or the business DNS server addresses? In the interim, at least? Just asking...

gml

GregV

932 posts

Ultimate Geek
+1 received by user: 304

#1123562 7-Sep-2014 10:46

They are set by Spark, but if they changed them, the malicious traffic would then be directed at these other DNS servers. I'm assuming - dangerous I know - that the business DNS server wouldn't have fared any better. Would have been interesting to see Google's response though

hio77

'That VDSL Cat'
13036 posts

Uber Geek
+1 received by user: 3896

ID Verified

Trusted

Lizard Networks

Subscriber

#1123566 7-Sep-2014 10:53

GregV: They are set by Spark, but if they changed them, the malicious traffic would then be directed at these other DNS servers. I'm assuming - dangerous I know - that the business DNS server wouldn't have fared any better. Would have been interesting to see Google's response though

Exactly this.

Ild say Google are hit on a daily basis, and probably have some form of a filter designed specially to identify and block dns attacks.

OnlyJoe:
which as been amplified by the increase in viruses from people taking advantage of nude celebrity leak.

what a novel distribution method that would be!

#include <std_disclaimer>

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.

Dyson

Shop now for Dyson appliances (affiliate link).

PaulBags

809 posts

Ultimate Geek
+1 received by user: 184
Inactive user

#1123570 7-Sep-2014 11:17

gujal: They also have a set of Business customer DNS Servers. These are quite nice to use rather than Google's DNS Servers which are in US and have to go over Southern Cross cable for something simple as DNS lookup
ns1.netgate.net.nz 202.37.245.17
ns2.netgate.net.nz 202.37.245.20

Stopped working for me about 11am, although I can still ping them. Back to OpenDNS then...

techmeister

310 posts

Ultimate Geek
+1 received by user: 22

#1123571 7-Sep-2014 11:23

Kirdog: Very slow and laggy ingress play on Spark mobile data today and yesterday evening. :(
Just out of Curiosity: Why Voda customers are not affected by that? ( http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=11320100 < some very strange excuse detected here, as my opinion)

Also someone mentioned that its "paid attack", it can be/why not...

Yes I agree, I smell a rat.
Why only Spark customers?

sbiddle

30853 posts

Uber Geek
+1 received by user: 9996

Retired Mod

Trusted

Biddle Corp

Lifetime subscriber

#1123573 7-Sep-2014 11:25

techmeister:
Kirdog: Very slow and laggy ingress play on Spark mobile data today and yesterday evening. :(
Just out of Curiosity: Why Voda customers are not affected by that? ( http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=11320100 < some very strange excuse detected here, as my opinion)

Also someone mentioned that its "paid attack", it can be/why not...

Yes I agree, I smell a rat.
Why only Spark customers?

How about the sample fact they have over 50% of NZers as broadband customers, hence they're more than twice the size of the next biggest network? This makes something on this scale so much harder to contain.

Curiosity

7 posts

Wannabe Geek
+1 received by user: 1

#1123576 7-Sep-2014 11:32

Something doesn't quite make sense to me. If it's a DNS amplification attack on an overseas target by Spark's customers, using Spark's recursive resolvers (the ones that fell over), that implies that packets sent by customers with seriously spoofed source IP addresses (not even within Spark's address range) can reach the DNS servers. I don't have an easy way to test it at the moment, but I thought most ISPs blocked source IP address spoofing pretty close to the customer?

1 | ... | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | ... | 21