Spark DNS Issues - Amazing - Broadband Service Alert

Forums › Spark New Zealand › Spark DNS Issues - Amazing - Broadband Service Alert

1 | ... | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21

48 posts

Geek
+1 received by user: 4

Trusted

Reply # 1124135 8-Sep-2014 11:04

One person supports this post

dcole13:
michaelmurfy:
mattwnz:
A lot if those are kids or nutters who will say it online but would never have guts to say it in person. Social media like Facebook and Twitter can bring out the worst in humans.

Minecraft - that explains it, kids these days have not been bought up without the internet like many others were so possibly a 12 year old having a temper tantrum not being able to play his precious Minecraft.

I think it's more people who think they can make a living off playing "professional" Call of Duty, I sent something to Tim about someone that lost $5 cause he couldn't join an Xbox live game...

At least 12 year old kids playing mine craft are not looking at dodgy porn sites and supposedly bringing down national grade isps. Nor are they doing any large number of recursive dns lookups. Don't judge people by how they choose to use their Internet connection. Regardless of how people actually use the internet it is always important to them - the end user.

OnlyJoe

10 posts

Wannabe Geek
+1 received by user: 9

Reply # 1124137 8-Sep-2014 11:09

Martin_NZ:
TimA: http://www.digitalattackmap.com/#anim=1&color=0&country=ALL&time=16320&view=map
Reckons about 5Gb/s to NZ.

5 gbps is hardly a large DoS attack considering the scale of recent attacks on the likes of cloud flare and spamhaus which were in the order of 400 and 300Gbps respectively. Given that even in nz networks like KAREN offer 10Gbps links between universities and research institutes it seems odd that a 5gbps attack can bring down a national isp. Something still doesn't add up as far as I'm concerned. Maybe they are running dns off a 386 laptop on a jets tart connection as was suggested by someone previously.

I am guessing here. But it is probably more to do with the quantity of the requests that were hitting their DNS server, rather than the bandwidth itself. If your DNS server is getting requests faster than it can process them, eventually it will overflow the buffers for pending requests. This will in the best situation just cause the server to completely disregard incoming packets, and in the worst case make the server crash due to lack of memory (depending how the software is managing this issue).

wasabi2k

2091 posts

Uber Geek
+1 received by user: 848

Reply # 1124138 8-Sep-2014 11:10

Post on reddit, linked to an article claiming this was all due to people looking at the nude celeb leaks - wut?

God the articles are terrible:
http://www.stuff.co.nz/business/industries/10468669/Naked-pics-link-to-internet-problems - just what? How are the two issues related?
http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=11320100 - "A frenzy over fake or leaked nude celebrity photos possibly sparked this weekend's disastrous internet meltdown." - No, it didn't

Good lord - how can such tripe be published?

michaelmurfy

Meow
7787 posts

Uber Geek
+1 received by user: 3847

Moderator

Trusted

Lifetime subscriber

Reply # 1124152 8-Sep-2014 11:19

wasabi2k: Post on reddit, linked to an article claiming this was all due to people looking at the nude celeb leaks - wut?

God the articles are terrible:
http://www.stuff.co.nz/business/industries/10468669/Naked-pics-link-to-internet-problems - just what? How are the two issues related?
http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=11320100 - "A frenzy over fake or leaked nude celebrity photos possibly sparked this weekend's disastrous internet meltdown." - No, it didn't

Good lord - how can such tripe be published?

It was people getting Malware by searching up such things that sparked a DDoS on Sparks DNS servers, so is correct.

Michael Murphy | https://murfy.nz
A quick guide to picking the right ISP | The Router Guide | Community UniFi Cloud Controller | Ubiquiti Edgerouter Tutorial

gjen

75 posts

Master Geek
+1 received by user: 7

Reply # 1124166 8-Sep-2014 11:31

Talkiet: I think it's time to go to bed. It's a work day tomorrow.

:-)

N

The Grand Prix was about to start how could you retire for the evening

dwl

363 posts

Ultimate Geek
+1 received by user: 43

Reply # 1124185 8-Sep-2014 12:01

gished: 8am Monday and still DNS issues with Spark DNS servers,

Auto config picks up 210.55.11.13 (GW), 122.56.237.1 (dns1) and 210.55.111.1 (dns2)

Gateway works fine but ns1 and ns2 aren't responding to ping, assuming ICMP isn't disabled. Ironically enough I didn't even notice the issue until I read the Stuff article on Saturday night which was about a hour or two before my manual dns servers went belly up (dnsc1/dnsc2.xtra.co.nz). Currently working off Googles dns atm.

The Spark website says pings are disabled:

The distributed nature of the DNS might mean you are affected differently to others. Do you get any response using nslookup:
C:\WINDOWS\system32>nslookup www.stuff.co.nz 122.56.237.1
Server: ns1.xtra.co.nz
Address: 122.56.237.1
Non-authoritative answer:
Name: a1784.g.akamai.net
Addresses: 219.88.187.34
219.88.187.32
Aliases: www.stuff.co.nz
www.stuff.co.nz.edgesuite.net

P.S. I have just had a call from what sounds like overseas saying they are from Spark and will need to disable my Internet connection as per the warning on the Spark website - with the publicity I think there could be an increase and some may get caught

pristle

73 posts

Master Geek
+1 received by user: 7

Reply # 1124190 8-Sep-2014 12:16

I had a client txt me last night to say that she'd received a call from a Spark rep. "They say that my modem has been affected by the virus an they will replace it. They have shut our service down."

When I spoke her today she elaborated further saying the rep told her that their connection had been identified as one contributing to the DDOS. I'll be visiting the family later to see what's up.

EDIT: included an inadvertently deleted sentence

pohutukawa

176 posts

Master Geek
+1 received by user: 9

Reply # 1124210 8-Sep-2014 12:37

Martin_NZ:
dcole13:
michaelmurfy:
mattwnz:
A lot if those are kids or nutters who will say it online but would never have guts to say it in person. Social media like Facebook and Twitter can bring out the worst in humans.

Minecraft - that explains it, kids these days have not been bought up without the internet like many others were so possibly a 12 year old having a temper tantrum not being able to play his precious Minecraft.

I think it's more people who think they can make a living off playing "professional" Call of Duty, I sent something to Tim about someone that lost $5 cause he couldn't join an Xbox live game...

At least 12 year old kids playing mine craft are not looking at dodgy porn sites and supposedly bringing down national grade isps. Nor are they doing any large number of recursive dns lookups. Don't judge people by how they choose to use their Internet connection. Regardless of how people actually use the internet it is always important to them - the end user.

So you say

At least 12 year old kids playing mine craft are not looking at dodgy porn sites

and then follow up with

Don't judge people by how they choose to use their Internet connection.

You appear to have no idea what 12 year-old kids look at on the internet, and seem to be judging people by how they use their internet connection.

freitasm

BDFL - Memuneh
61192 posts

Uber Geek
+1 received by user: 11974

Administrator

Trusted

Geekzone

Lifetime subscriber

Reply # 1124217 8-Sep-2014 12:46

One person supports this post

Just received:

Media Advisory 12:10 Monday 8 September 2014

Update on Spark New Zealand DNS service issue

What has happened?

Cyber criminals based overseas appear to have been attacking web addresses in Eastern Europe, and were bouncing the traffic off Spark customer connections, in what is known as a distributed denial of service (DDoS) attack.

The DDoS attack was dynamic, predominantly taking the shape of an ‘amplified DNS attack’ which means an extremely high number of connection requests – in the order of thousands per second - were being sent to a number of overseas web addresses with the intention of overwhelming and crashing them. Each of these requests, as it passes through our network, queries our DNS server before it passes on – so our servers were bearing the full brunt of the attack.

While the Spark network did not crash, we did experience extremely high traffic loads hitting our DNS servers which meant many customers had either slow or at times no connectivity (as their requests were timing out). There were multiple attacks, which were dynamic in nature. They began on Friday night, subsided, and then began again early Saturday, continuing over the day. By early Sunday morning traffic levels were back to normal and have remained so since. We did see the nature of the attack evolve over the period, possibly due to the cyber criminals monitoring our response and modifying their attack to circumvent our mitigation measures – in a classic ‘whack a mole’ scenario.

How did they get access through the Spark Network?

Since the attacks began we have had people working 24/7 to identify the root causes, alongside working to get service back to normal. During the attack, we observed that a small number of customer connections were involved in generating the vast majority of the traffic. This was consistent with customers having malware on their devices and the timing coincided with other DNS activity related to malware in other parts of the world.

However, while we’re not ruling out malware as a factor, we have also identified that cyber criminals have been accessing vulnerable customer modems on our network. These modems have been identified as having “open DNS resolver” functionality, which means they can be used to carry out internet requests for anyone on the internet. This makes it easier for cyber criminals to ‘bounce’ an internet request off them (making it appear that the NZ modem was making the request, whereas it actually originates from an overseas source). Most of these modems were not supplied by Spark and tend to be older or lower-end modems.

What remains clear is that good end user security remains an important way to combat these attacks. With the proliferation of devices in households, that means both the security within your device and the security of your modem.

What did Spark do?

We have now disconnected those modems from our network and are contacting all the affected customers. We have also taken steps at a network level to mitigate this modem vulnerability. We are now in the process of scanning our entire broadband customer base to identify any other customers who may be using modems with similar vulnerabilities and will be contacting those identified customers in due course to advise them on what they should do.

With respect to malware we continue to strongly encourage our customers to keep their internet device security up to date, conduct regular scans and regularly update the operating software and firmware on their home network. We also continue to advise customers not to click on suspicious links or download files when they are not sure of the contents.

We have also taken steps at the network level to make it more difficult for cyber criminals to exploit the DNS open resolver modem vulnerability and we’re using the latest technology to strengthen our network monitoring and management capabilities. For security reasons we can’t detail these steps, however this is an ongoing battle to stay one step ahead of cyber criminals who are continually using more and more sophisticated tactics.

Why only Spark?

We can’t say what other networks experienced. However, cyber criminals often look for clusters of IP addresses to use in any particular DDoS attack. That makes it more likely that these IP addresses belong to the customers of a single ISP – even more likely with a large ISP like Spark.

Backblaze cloud backup (personal and business) | Geekzone broadband switch | Amazon (Geekzone aff) | MightyApe (Geekzone aff) |

My technology disclosure | Business Transformation | Enterprise Content Management | Customer Relationship Management

OnlyJoe

10 posts

Wannabe Geek
+1 received by user: 9

Reply # 1124223 8-Sep-2014 12:50

One person supports this post

freitasm: Just received:

Media Advisory 12:10 Monday 8 September 2014

Update on Spark New Zealand DNS service issue

....

Also makes sense as to why they now block external access on Port 53, as per the other thread. Do other ISPs already do this?

pohutukawa

176 posts

Master Geek
+1 received by user: 9

Reply # 1124300 8-Sep-2014 13:38

Interesting article in the Herald (the bastion of all that is good and true in the world, of course [irony alert]

http://www.nzherald.co.nz/business/news/article.cfm?c_id=3&objectid=11320778

"Spark hasn't so far provided any technical detail on what happened, but as the rest of its network appeared to be working, it's unlikely that a DNS amplification attack was the culprit."

Anybody here from Spark that can comment?

Spark people have been very critical of anybody laying the slightest bit of responsibility at Spark's door.

Bee

593 posts

Ultimate Geek
+1 received by user: 109

Reply # 1124305 8-Sep-2014 13:42

waiting for Yahoo to be mentioned in all of this...

pohutukawa

176 posts

Master Geek
+1 received by user: 9

Reply # 1124308 8-Sep-2014 13:44

Bee: waiting for Yahoo to be mentioned in all of this...

"Yahoo attacks own clients, who blame misfortunes on porn, malware and home users"

Talkiet

3658 posts

Uber Geek
+1 received by user: 2178

Trusted

Spark NZ

Reply # 1124312 8-Sep-2014 13:48

pohutukawa: Interesting article in the Herald (the bastion of all that is good and true in the world, of course [irony alert]

http://www.nzherald.co.nz/business/news/article.cfm?c_id=3&objectid=11320778

"Spark hasn't so far provided any technical detail on what happened, but as the rest of its network appeared to be working, it's unlikely that a DNS amplification attack was the culprit."

Anybody here from Spark that can comment?

Spark people have been very critical of anybody laying the slightest bit of responsibility at Spark's door.

I'll make the personal comment that the esteemed author of that article quoted ME without talking to me, or even sending a request for further info. I might have been able to tell him I was staring at packet captures of insanely large DNS answers that were clearly part of a DNS amplification attack.

My personal comment is that Juha's article is a hastily written, non fact checked article with a picture of Jennifer Lawrence at the top of it.

Cheers - N (in a personal capacity)

edit: " Are we to believe that "a handful" of malware infected users were able to overwhelm that, for three whole days?" Um, it was 33 hours from first detection to restoration of normal DNS server performance. (There would have been some users impacted for lonbger because they were identified as being contributing to the issue and they were knocked off - but that was a tiny fraction of users)

cbrpilot

676 posts

Ultimate Geek
+1 received by user: 222

Trusted

Spark NZ

Reply # 1124314 8-Sep-2014 13:51

Sorry Pohutukawa, I'm not going to get into a war of words over a NZherald article. We don't come on here to throw dirt at other organisations.
All I think we can say is that a little knowledge is a dangerous thing, and quotes have been attributed to Spark that did not come from Spark at all.
Please see Mauricio's post above for a good explanation of what occurred.

My views are my own, and may not necessarily represent those of my employer.

1 | ... | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21