Spark DNS Issues - Amazing - Broadband Service Alert

Forums › Spark New Zealand (including Skinny and BigPipe) › Spark DNS Issues - Amazing - Broadband Service Alert

PeteS

40 posts

Geek
+1 received by user: 18

#151805 5-Sep-2014 21:55

Well somethings improved, Sparky & the Sparquettes are on the case in record time and only 1.5 hours after this was a major issue, I note an alert appeared. The cynic in me normally reckons half the country will have to be out for any hint of an admission.
BROADBAND SERVICE STATUS Updated at 8:30pm, 05 September 2014 Broadband Outage - Browsing Issue Some customers may be experiencing problems browsing websites. Our technicians are currently working on this problem. At this stage our representatives are unable to provide any additional information. We apologise for any inconvenience.

This is a filtered page: currently showing replies marked as answers. Click here to see full discussion.

Talkiet

4819 posts

Uber Geek
+1 received by user: 3934

Trusted

#1122970 6-Sep-2014 10:45

As many have worked out (and I just got off a technical audio about the issue), this is a DNS amplification attack and we have had people working all night trying to mitigate the issue as best we can.

In the interim, the suggestion here to change your DNS servers to 8.8.8.8 and 8.8.4.4 is appropriate.

This should restore browsing. Please remember to check back to see when the issue is resolved - the performance of some sites with the above alternate DNS servers will not be as good as with our DNS servers (When they are fully available again)

At the moment, the workaround on the Spark status page is wrong. I am having it changed.

Sorry about this everyone

Neil G

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

Talkiet

4819 posts

Uber Geek
+1 received by user: 3934

Trusted

#1123049 6-Sep-2014 12:32

I don't have time to respond to everything here but...

- It's a DNS amplification attack, it doesn't need much of a footprint to have a big impact
- People (many) are working on multiple streams to mitigate and fix

Suggesting the GSCB or FBI look into this I can only assume is an attempt at humour.

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

Talkiet

4819 posts

Uber Geek
+1 received by user: 3934

Trusted

#1123611 7-Sep-2014 13:32

I'm hesitant to go into too much detail at this stage since the issue is still under (very) active investigation, but I can say that we have seen symptoms (in terms of DNS response times from all DNS servers and DNS lookup results) abate to normal, expected performance since approx 4am this morning.

We are still seeing a significant amount of unexpected traffic that we are working towards being able to identify as selectively as possible so as not to hurt legitimate users, but at the moment performance is as expected (from our distributed test network stats and from server based metrics like load balancer stats, DNS server CPU/RAM/QPS stats).

There will be no official comment made at this stage about the root cause (at least through this pretty direct technical channel) because it looks like the attack has evolved over time to avoid some of the mitigations put in place early on during the attack.

At this stage, if you're the sort of person comfortable flicking between DNS servers, I would personally recommend switching back to Sparks auto assigned DNS servers, but be ready to flick back to the Google servers should the attack escalate or find a way to dodge current mitigations.

From the (many) people working on this (some of whom have been working for ~32 of the last ~40 hours), we appreciate the understanding that many on this board have displayed in the face of one of the longest sustained attacks we've seen. There's certainly a HUGE amount less of the purely emotional and frankly insulting comments seen on FB and stuff.co.nz comments sections. Many thanks for that.

If anyone using the Spark DNS servers is currently seeing regular DNS lookup failures, I'd appreciate a PM highlighting that.

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

freitasm

BDFL - Memuneh
80658 posts

Uber Geek
+1 received by user: 41072

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#1124217 8-Sep-2014 12:46

Just received:

Media Advisory 12:10 Monday 8 September 2014

Update on Spark New Zealand DNS service issue

What has happened?

Cyber criminals based overseas appear to have been attacking web addresses in Eastern Europe, and were bouncing the traffic off Spark customer connections, in what is known as a distributed denial of service (DDoS) attack.

The DDoS attack was dynamic, predominantly taking the shape of an ‘amplified DNS attack’ which means an extremely high number of connection requests – in the order of thousands per second - were being sent to a number of overseas web addresses with the intention of overwhelming and crashing them. Each of these requests, as it passes through our network, queries our DNS server before it passes on – so our servers were bearing the full brunt of the attack.

While the Spark network did not crash, we did experience extremely high traffic loads hitting our DNS servers which meant many customers had either slow or at times no connectivity (as their requests were timing out). There were multiple attacks, which were dynamic in nature. They began on Friday night, subsided, and then began again early Saturday, continuing over the day. By early Sunday morning traffic levels were back to normal and have remained so since. We did see the nature of the attack evolve over the period, possibly due to the cyber criminals monitoring our response and modifying their attack to circumvent our mitigation measures – in a classic ‘whack a mole’ scenario.

How did they get access through the Spark Network?

Since the attacks began we have had people working 24/7 to identify the root causes, alongside working to get service back to normal. During the attack, we observed that a small number of customer connections were involved in generating the vast majority of the traffic. This was consistent with customers having malware on their devices and the timing coincided with other DNS activity related to malware in other parts of the world.

However, while we’re not ruling out malware as a factor, we have also identified that cyber criminals have been accessing vulnerable customer modems on our network. These modems have been identified as having “open DNS resolver” functionality, which means they can be used to carry out internet requests for anyone on the internet. This makes it easier for cyber criminals to ‘bounce’ an internet request off them (making it appear that the NZ modem was making the request, whereas it actually originates from an overseas source). Most of these modems were not supplied by Spark and tend to be older or lower-end modems.

What remains clear is that good end user security remains an important way to combat these attacks. With the proliferation of devices in households, that means both the security within your device and the security of your modem.

What did Spark do?

We have now disconnected those modems from our network and are contacting all the affected customers. We have also taken steps at a network level to mitigate this modem vulnerability. We are now in the process of scanning our entire broadband customer base to identify any other customers who may be using modems with similar vulnerabilities and will be contacting those identified customers in due course to advise them on what they should do.

With respect to malware we continue to strongly encourage our customers to keep their internet device security up to date, conduct regular scans and regularly update the operating software and firmware on their home network. We also continue to advise customers not to click on suspicious links or download files when they are not sure of the contents.

We have also taken steps at the network level to make it more difficult for cyber criminals to exploit the DNS open resolver modem vulnerability and we’re using the latest technology to strengthen our network monitoring and management capabilities. For security reasons we can’t detail these steps, however this is an ongoing battle to stay one step ahead of cyber criminals who are continually using more and more sophisticated tactics.

Why only Spark?

We can’t say what other networks experienced. However, cyber criminals often look for clusters of IP addresses to use in any particular DDoS attack. That makes it more likely that these IP addresses belong to the customers of a single ISP – even more likely with a large ISP like Spark.

Referral links: Quic Broadband (free setup code: R587125ERQ6VE) | Samsung | AliExpress | Wise | Sharesies

Support Geekzone by subscribing (browse ads-free), or making a one-off or recurring donation through PressPatron.