Spark DNS Issues - Amazing - Broadband Service Alert

Forums › Spark New Zealand (including Skinny and BigPipe) › Spark DNS Issues - Amazing - Broadband Service Alert

1 | ... | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | ... | 21

276 posts

Ultimate Geek
+1 received by user: 10

#1124319 8-Sep-2014 13:59

Hmm interesting...

==================================

C:\Users\Admin>ping 122.56.237.1

Pinging 122.56.237.1 with 32 bytes of data:

Request timed out.

Ping statistics for 122.56.237.1:

Packets: Sent = 1, Received = 0, Lost = 1 (100% loss),

Control-C

C:\Users\Admin>tracert 122.56.237.1

Tracing route to ns1.xtra.co.nz [122.56.237.1]

over a maximum of 30 hops:

1 1 ms <1 ms <1 ms 192.168.10.1

2 58 ms 21 ms 20 ms 122-59-232-1.jetstream.xtra.co.nz [122.59.232.1]

3 * 36 ms 27 ms 122.56.238.242

4 * * * Request timed out.

5 * * * Request timed out.

6 * * * Request timed out.

7     *        *        *     Request timed out.

Trace complete.

C:\Users\Admin>nslookup 122.56.237.1
Server: google-public-dns-a.google.com
Address: 8.8.8.8

Name:    ns1.xtra.co.nz
Address: 122.56.237.1

C:\Users\Admin>nslookup
Default Server: google-public-dns-a.google.com
Address: 8.8.8.8

> server 122.56.237.1
Default Server: ns1.xtra.co.nz
Address: 122.56.237.1

> google.com
Server: ns1.xtra.co.nz
Address: 122.56.237.1

Non-authoritative answer:
Name:    google.com
Addresses: 2404:6800:4006:804::1000
          74.125.237.137
          74.125.237.132
          74.125.237.134
          74.125.237.131
          74.125.237.130
          74.125.237.136
          74.125.237.129
          74.125.237.128
          74.125.237.133
          74.125.237.142
          74.125.237.135

> stuff.co.nz
Server: ns1.xtra.co.nz
Address: 122.56.237.1

Non-authoritative answer:
Name:    stuff.co.nz
Addresses: 203.57.145.102
          202.21.128.102

==================================

....and its working now (I guess I should be happy)

OnlyJoe

10 posts

Wannabe Geek
+1 received by user: 9

#1124321 8-Sep-2014 14:01

I just did a real basic test on some ISP DNS servers, basically can I resolve addresses from their DNS using a server in the USA. Is there any filtering on their DNS.

Here are the results:

Spark (122.56.237.1): no - request timed out
Orcon (121.98.0.1): no response from server
Slingshot (202.180.64.10): Query refused
Woosh (202.74.207.253): Query refused
Actrix (203.96.16.35): Query refused
Digital Island (123.100.67.135): Query refused
Inspire (203.114.168.2): Query refused

Vodafone (old 203.109.129.67) resolves fine, just like a public DNS
Vodafone (new 203.109.191.1) resolves fine, just like a public DNS
So why is the Vodafone DNS open to connections from the USA?? It is like they are running a public DNS server.

Talkiet

4819 posts

Uber Geek
+1 received by user: 3934

Trusted

#1124324 8-Sep-2014 14:03

gished: Hmm interesting...

==================================

C:\Users\Admin>ping 122.56.237.1 Pinging 122.56.237.1 with 32 bytes of data: Request timed out. Ping statistics for 122.56.237.1: Packets: Sent = 1, Received = 0, Lost = 1 (100% loss), Control-C ^C C:\Users\Admin>tracert 122.56.237.1 Tracing route to ns1.xtra.co.nz [122.56.237.1] over a maximum of 30 hops: 1 1 ms <1 ms <1 ms 192.168.10.1 2 58 ms 21 ms 20 ms 122-59-232-1.jetstream.xtra.co.nz [122.59.232.1] 3 * 36 ms 27 ms 122.56.238.242 4 * * * Request timed out. 5 * * * Request timed out. 6 * * * Request timed out. 7 * * * Request timed out.

Trace complete.

search for tcping, install...

tcping 122.56.237.1 53

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

pohutukawa

197 posts

Master Geek
+1 received by user: 9

#1124333 8-Sep-2014 14:19

Talkiet:
pohutukawa: Interesting article in the Herald (the bastion of all that is good and true in the world, of course [irony alert]

http://www.nzherald.co.nz/business/news/article.cfm?c_id=3&objectid=11320778

"Spark hasn't so far provided any technical detail on what happened, but as the rest of its network appeared to be working, it's unlikely that a DNS amplification attack was the culprit."

Anybody here from Spark that can comment?

Spark people have been very critical of anybody laying the slightest bit of responsibility at Spark's door.

I'll make the personal comment that the esteemed author of that article quoted ME without talking to me, or even sending a request for further info. I might have been able to tell him I was staring at packet captures of insanely large DNS answers that were clearly part of a DNS amplification attack.

My personal comment is that Juha's article is a hastily written, non fact checked article with a picture of Jennifer Lawrence at the top of it.

Cheers - N (in a personal capacity)

edit: " Are we to believe that "a handful" of malware infected users were able to overwhelm that, for three whole days?" Um, it was 33 hours from first detection to restoration of normal DNS server performance. (There would have been some users impacted for lonbger because they were identified as being contributing to the issue and they were knocked off - but that was a tiny fraction of users)

Hence "Irony alert".

So the Spark position is that the release is correct and definitive?

pohutukawa ... turning with the seasons ...

sbiddle

30853 posts

Uber Geek
+1 received by user: 9996

Retired Mod

Trusted

Biddle Corp

Lifetime subscriber

#1124336 8-Sep-2014 14:22

OnlyJoe:
freitasm: Just received:

Media Advisory 12:10 Monday 8 September 2014 Update on Spark New Zealand DNS service issue ....

Also makes sense as to why they now block external access on Port 53, as per the other thread. Do other ISPs already do this?

In the last few months a number that previously allowed this (incl Telecom who I believe changed their policy earlier in the year) have disabled lookups from outside their networks.

pohutukawa

197 posts

Master Geek
+1 received by user: 9

#1124338 8-Sep-2014 14:24

cbrpilot: Sorry Pohutukawa, I'm not going to get into a war of words over a NZherald article. We don't come on here to throw dirt at other organisations.
All I think we can say is that a little knowledge is a dangerous thing, and quotes have been attributed to Spark that did not come from Spark at all.
Please see Mauricio's post above for a good explanation of what occurred.

You come in here and say you don't want to get into a war of words and then go on to add to the discussion.

OK then.

"A little knowledge is a dangerous thing". Don't think anybody here would disagree.

You need to understand that geekzone isn't just somewhere where you can come and have your say and not expect questions to be asked and even positions challenged. It's an exchange of views and provides the opportunity to go into a little more depth about technical factors that would be a complete waste of time discussing with your person on the street.

And you don't need to take everything personally, either.

pohutukawa ... turning with the seasons ...

Dyson

Shop now for Dyson appliances (affiliate link).

Talkiet

4819 posts

Uber Geek
+1 received by user: 3934

Trusted

#1124391 8-Sep-2014 15:10

pohutukawa: [snip]
So the Spark position is that the release is correct and definitive?

That press release is a wrap up of the major points written at a level understandable by many people and journalists. There are certain technical bits we won't release because they would be confusing, misleading or simply beyond the comprehension of people that haven't been involved in running a large ISP network.

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

PaulBags

809 posts

Ultimate Geek
+1 received by user: 184
Inactive user

#1124452 8-Sep-2014 16:00

Media Advisory 12:10 Monday 8 September 2014

Update on Spark New Zealand DNS service issue

Nicely put, balms almost all of my angst; except I'm left wondering why this isn't sparks service status/linked to in the service status? Service status is, strangely enough, my first port of call to find the status of services, but the info it's provided throughout this whole saga has been woefully mismanaged to the point of being inflammatory.

cbrpilot

964 posts

Ultimate Geek
+1 received by user: 555

Trusted

Spark NZ

#1124453 8-Sep-2014 16:00

pohutukawa:
cbrpilot: Sorry Pohutukawa, I'm not going to get into a war of words over a NZherald article. We don't come on here to throw dirt at other organisations.
All I think we can say is that a little knowledge is a dangerous thing, and quotes have been attributed to Spark that did not come from Spark at all.
Please see Mauricio's post above for a good explanation of what occurred.

You come in here and say you don't want to get into a war of words and then go on to add to the discussion.

OK then.

"A little knowledge is a dangerous thing". Don't think anybody here would disagree.

You need to understand that geekzone isn't just somewhere where you can come and have your say and not expect questions to be asked and even positions challenged. It's an exchange of views and provides the opportunity to go into a little more depth about technical factors that would be a complete waste of time discussing with your person on the street.

And you don't need to take everything personally, either.

??? Wow.
I will reiterate what I have said. As a Spark employee I am not going to critique an editorial article by a news publication. Some may disagree with my decision not to get drawn into that discussion, but that's life.

I have no issues with people asking questions about what we have done or said on GZ.

And for what it's worth, nothing being taken personally here.

My views are my own, and may not necessarily represent those of my employer.

freitasm

BDFL - Memuneh
80658 posts

Uber Geek
+1 received by user: 41072

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#1124454 8-Sep-2014 16:02

I hope this doesn't turn into a Facebook-level exchange. Heads roll when that happens.

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.

Martin_NZ

133 posts

Master Geek
+1 received by user: 26

Trusted

#1124459 8-Sep-2014 16:07

pohutukawa:
Martin_NZ:
dcole13:
michaelmurfy:
mattwnz:
A lot if those are kids or nutters who will say it online but would never have guts to say it in person. Social media like Facebook and Twitter can bring out the worst in humans.

Minecraft - that explains it, kids these days have not been bought up without the internet like many others were so possibly a 12 year old having a temper tantrum not being able to play his precious Minecraft.

I think it's more people who think they can make a living off playing "professional" Call of Duty, I sent something to Tim about someone that lost $5 cause he couldn't join an Xbox live game...

At least 12 year old kids playing mine craft are not looking at dodgy porn sites and supposedly bringing down national grade isps. Nor are they doing any large number of recursive dns lookups. Don't judge people by how they choose to use their Internet connection. Regardless of how people actually use the internet it is always important to them - the end user.

So you say
At least 12 year old kids playing mine craft are not looking at dodgy porn sites
and then follow up with
Don't judge people by how they choose to use their Internet connection.

You appear to have no idea what 12 year-old kids look at on the internet, and seem to be judging people by how they use their internet connection.

I'm very aware of what my 12 year olds look at on their Internet connection. My checkpoint firewall provides very good insight.

Geekzone support

Support Geekzone with one-off or recurring donations Donate via PressPatron.

dcole13

534 posts

Ultimate Geek
+1 received by user: 107

#1124461 8-Sep-2014 16:09

Martin_NZ:
pohutukawa:
Martin_NZ:
dcole13:
michaelmurfy:
mattwnz:
A lot if those are kids or nutters who will say it online but would never have guts to say it in person. Social media like Facebook and Twitter can bring out the worst in humans.

Minecraft - that explains it, kids these days have not been bought up without the internet like many others were so possibly a 12 year old having a temper tantrum not being able to play his precious Minecraft.

I think it's more people who think they can make a living off playing "professional" Call of Duty, I sent something to Tim about someone that lost $5 cause he couldn't join an Xbox live game...

At least 12 year old kids playing mine craft are not looking at dodgy porn sites and supposedly bringing down national grade isps. Nor are they doing any large number of recursive dns lookups. Don't judge people by how they choose to use their Internet connection. Regardless of how people actually use the internet it is always important to them - the end user.

So you say
At least 12 year old kids playing mine craft are not looking at dodgy porn sites
and then follow up with
Don't judge people by how they choose to use their Internet connection.

You appear to have no idea what 12 year-old kids look at on the internet, and seem to be judging people by how they use their internet connection.

I'm very aware of what my 12 year olds look at on their Internet connection. My checkpoint firewall provides very good insight.

I can tell you that a lot of 12 year olds do look at adult content.

Home ADSL: School:

pohutukawa

197 posts

Master Geek
+1 received by user: 9

#1124512 8-Sep-2014 16:45

cbrpilot:
pohutukawa:
cbrpilot: Sorry Pohutukawa, I'm not going to get into a war of words over a NZherald article. We don't come on here to throw dirt at other organisations.
All I think we can say is that a little knowledge is a dangerous thing, and quotes have been attributed to Spark that did not come from Spark at all.
Please see Mauricio's post above for a good explanation of what occurred.

You come in here and say you don't want to get into a war of words and then go on to add to the discussion.

OK then.

"A little knowledge is a dangerous thing". Don't think anybody here would disagree.

You need to understand that geekzone isn't just somewhere where you can come and have your say and not expect questions to be asked and even positions challenged. It's an exchange of views and provides the opportunity to go into a little more depth about technical factors that would be a complete waste of time discussing with your person on the street.

And you don't need to take everything personally, either.

??? Wow.
I will reiterate what I have said. As a Spark employee I am not going to critique an editorial article by a news publication. Some may disagree with my decision not to get drawn into that discussion, but that's life.

I have no issues with people asking questions about what we have done or said on GZ.

And for what it's worth, nothing being taken personally here.

That's great.

Let's just say that when it comes to spinning the media, the company formerly known as TelecomNZ are up there with the most eager.

Most of these modems were not supplied by Spark and tend to be older or lower-end modems.

So old, low-end modems were the vector to attack Sparks DNS infrastructure?

How many were Spark modems?

Is Spark undertaking any source IP validation measures, or are attacks of this sort (DNS DDoS amplification) going to be commonplace for Spark?

Put another way, if there is nothing you can do about this apart from what you have, then Spark customers just need to live with this reality.

I note that previously historically there were significant issues with SPAM being sent via Telecom SMTP servers due to the lack of authentication but there was little appetite on Telecom's behalf to do anything relatively serious about it until very recently.

pohutukawa ... turning with the seasons ...

Tel69

Tel69
261 posts

Ultimate Geek
+1 received by user: 4

Trusted

Lifetime subscriber

#1124533 8-Sep-2014 17:00

The modem at the place I'm staying at is one of the old Technicolor TG582n's
Default settings, no password for the Administrator account and this.
{Administrator}=>dns server config
domain   : telecom
timeout : 15s
suppress : 0
state    : enabled
trace    : disabled
syslog   : disabled
spoofing : enabled
spoof ip : 198.18.1.0

Ummm, the modems running it's own DNS server with spoofing enabled (Oh and no logging)??

Am I reading that right?
(Also is that DNS server running an Open Recursive DNS, http://support.zen.co.uk/kb/Knowledgebase/Broadband-Securing-your-Technicolor-TG582n-against-Open-Recursive-DNS indicates it might be)

EDIT added the last question in brackets as I just thought of it.

Talkiet

4819 posts

Uber Geek
+1 received by user: 3934

Trusted

#1124534 8-Sep-2014 17:02

pohutukawa: snip

I don't know whether it's deliberate, but your questions are coming across as pretty aggressive and inflammatory.

I can promise you, Spark is doing a hell of a lot to improve the response to any future attacks of this nature. I can also promise you that there's no way we're going to tell random people on message boards exactly what we are doing from a resiliency and security design point of view.

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

1 | ... | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | ... | 21