Spark DNS Issues - Amazing - Broadband Service Alert

Forums › Spark New Zealand (including Skinny and BigPipe) › Spark DNS Issues - Amazing - Broadband Service Alert

1 | ... | 15 | 16 | 17 | 18 | 19 | 20 | 21

20 posts

Geek

#1125819 10-Sep-2014 15:20

I was advised to check the DNS entries in the router, it seems they may have retired / disabled some DNS servers over the weekend.
I called 3 of the sites effected, they all had Dlink DSL-2642B a little white box with a single aerial.

Primary 122.56.237.1 ns1.xtra.co.nz Secondary 210.55.111.1 ns2.xtra.co.nz http://www.spark.co.nz/help/internet/manually-change-your-dns-server-setting/

Talkiet

4792 posts

Uber Geek

Trusted

#1125821 10-Sep-2014 15:21

insane:
pristle:

The client has already asked about other provider options.

If the client is actively taking part in the cause of the issue, then perhaps they should be looking inwards before looking outwards? As far as I know every ISP has rights to kick users off who are affecting their service/core infrastructure.

Unless I've misunderstood what your issue is.

Without getting involved in every individual customer issue, I should clarify what in this case 'blocking' or 'kick users off' means.

If users were specifically identified as sending a lot of DNS lookups to OUR DNS servers for specific (and very precise/odd/uncommon) domain names which had huge responses, their access to our primary DNS infrastructure was blocked. THOSE users could still use Google DNS etc.

There are a few types of router on the network (not large numbers / NOT Telecom/Spark supplied) that refuse to play nice with one of the network level mitigations we carried out over the weekend. This is because those old routers do not implement their DNS relay in according to current (or even moderately old) best practises. We are speaking with the vendors involved right now. These users will need to put DNS settings on their client devices directly in order to access the Internet. We can't release the list of affected routers right now.

There's a crazy chance you might fall into both categories- I haven't done the corss reference, but it's unlikely.

We have ABSOLUTLEY NO INTENTION of kicking people off the service permanently as a result of the issues, or the inadvertent participation in the amplification attack. We've only done what we've done to stabilise the network for everyone.

I understand that if you have been hit by either of these mitigation processes it will be annoying, but the helpdesk DOES have processes to help in either case.

Cheers - N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

Talkiet

4792 posts

Uber Geek

Trusted

#1125822 10-Sep-2014 15:22

Keef: I was advised to check the DNS entries in the router, it seems they may have retired / disabled some DNS servers over the weekend.

Primary 122.56.237.1 ns1.xtra.co.nz Secondary 210.55.111.1 ns2.xtra.co.nz http://www.spark.co.nz/help/internet/manually-change-your-dns-server-setting/

This is not true. We have not retired or disabled any DNS servers over the weekend.

The DNS servers you list above are correct and in most cases will work. As per my above post, a very small numebr of users may have been blocked from these servers temporarily.

Cheers _ N

Please note all comments are from my own brain and don't necessarily represent the position or opinions of my employer, previous employers, colleagues, friends or pets.

pristle

158 posts

Master Geek

Subscriber

#1125842 10-Sep-2014 15:40

While the client concedes there may well have been issues at their end, they can only go on the say so of the providers rep. As I've already stated I performed extensive malware checks on the household PCs, which did reveal and resolve several issues.

The brickbats the client has are: the time frame for their services to be restored after their environment had been secured; the poor customer service they've received (in terms of not keeping to agreed call backs etc.) . She is a cancer sufferer, who needs access to her medical data and specialist online. Her boys both have homework assignments that are filed online. Husband works from home and relies heavily on email.

It's fair for them to have lost some faith in Spark. Given that they've been stalwart Telecom customers for their entire history, I doubt that they'll change the status quo anyway.

Txt just came in, they have service.

toxicbubble

44 posts

Geek

#1125843 10-Sep-2014 15:40

Talkiet:

Without getting involved in every individual customer issue, I should clarify what in this case 'blocking' or 'kick users off' means.

If users were specifically identified as sending a lot of DNS lookups to OUR DNS servers for specific (and very precise/odd/uncommon) domain names which had huge responses, their access to our primary DNS infrastructure was blocked. THOSE users could still use Google DNS etc.

There are a few types of router on the network (not large numbers / NOT Telecom/Spark supplied) that refuse to play nice with one of the network level mitigations we carried out over the weekend. This is because those old routers do not implement their DNS relay in according to current (or even moderately old) best practises. We are speaking with the vendors involved right now. These users will need to put DNS settings on their client devices directly in order to access the Internet. We can't release the list of affected routers right now.

There's a crazy chance you might fall into both categories- I haven't done the corss reference, but it's unlikely.

We have ABSOLUTLEY NO INTENTION of kicking people off the service permanently as a result of the issues, or the inadvertent participation in the amplification attack. We've only done what we've done to stabilise the network for everyone.

I understand that if you have been hit by either of these mitigation processes it will be annoying, but the helpdesk DOES have processes to help in either case.

Cheers - N

Thanks heaps for clarifying the multiple situations which were mixing around around confusing the thread.

As a sidenote: I've ordered a myself new modem to upgrade the old Dynalink, as was pointed out earlier, it's fairly old and likely should be updated anyway. For now the Google DNS workaround is working for me, should hold us over until the new modem arrives.

cbrpilot

955 posts

Ultimate Geek

Trusted

Spark NZ

#1125965 10-Sep-2014 17:42

pristle:

Blaming the non 'Telecom"/"Spark" routers is a bit of a cop out. If there was a vulnerablility in them, why was this not picked up in the Telepermit process?

Just a point of clarification I would like to add here so as people understand what a Telepermit is and what it is not.

A Telepermit is a Permit to Connect (PTC). What that means is that it has been tested and proved that it will not electrically interfere with or damage the infrastructure that it is directly connected to - in the case of DSL, the copper network, and the Chorus DSLAMs. The device is not tested to make sure it works. It is not tested to make sure that it has no security vulnerabilities etc. It just means that it is safe to connect to the network. If you connect a non-Telepermitted device to the network, and it damages a Chorus DSLAM, you would be legally liable for that damage. If it was a Telepermitted device, I understand that in the same circumstance, you would not be legally liable.

So from the point of view of device operations (i.e. that it actually works) and security, that responsibility rests solely with the supplier of that device.

My views are my own, and may not necessarily represent those of my employer.

OnlyJoe

10 posts

Wannabe Geek

#1125977 10-Sep-2014 18:05

cbrpilot:
pristle:

Blaming the non 'Telecom"/"Spark" routers is a bit of a cop out. If there was a vulnerablility in them, why was this not picked up in the Telepermit process?

Just a point of clarification I would like to add here so as people understand what a Telepermit is and what it is not.

A Telepermit is a Permit to Connect (PTC). What that means is that it has been tested and proved that it will not electrically interfere with or damage the infrastructure that it is directly connected to - in the case of DSL, the copper network, and the Chorus DSLAMs. The device is not tested to make sure it works. It is not tested to make sure that it has no security vulnerabilities etc. It just means that it is safe to connect to the network. If you connect a non-Telepermitted device to the network, and it damages a Chorus DSLAM, you would be legally liable for that damage. If it was a Telepermitted device, I understand that in the same circumstance, you would not be legally liable.

So from the point of view of device operations (i.e. that it actually works) and security, that responsibility rests solely with the supplier of that device.

I would have thought that all the routers on the new Ultra Fibre network are going to be pretty new routers, so it is surprising that these "old routers" that has been mentioned have such an open security fault.

Quic Broadband

Move to New Zealand's best fibre broadband service (affiliate link). Note that to use Quic Broadband you must be comfortable with configuring your own router.

quickymart

13932 posts

Uber Geek

ID Verified

#1125986 10-Sep-2014 18:20

So...do I need to do anything with Netcomm NF4V, or is the dropping DNS issue I'm still intermittently experiencing unrelated?

Tel69

Tel69
261 posts

Ultimate Geek

Trusted

Lifetime subscriber

#1126017 10-Sep-2014 18:55

God, I forgot what a PITA the Technicolor TG582n can be.
You can't change the DNS settings for the dhcp pool in the modem unless everything is off the wireless interface.
Even if your machine is outside the DCHP pool (fixed IP, lets say 192.168.1.10 with the DCHP pool starts at 192.168.1.64), and dhcp is turned off, because you are wireless it refuses to allow you access to edit the DHCP pool (for adding DNS servers is what I'm after so I can turn the internal modem one off).

Since it's not my modem or internet connection I probably should not be doing it, but hey, I already picked a better channel for them to use for wireless.
Just looking at what I would look at if I was home for hardening, and man I know why I customised my Technicolor TG582n firmware as soon as it was up when I had one.

dwl

371 posts

Ultimate Geek

#1126201 10-Sep-2014 23:29

Talkiet:
While using the Google DNS servers is a perfectly reasonable short term fix, you will likely be directed offshore for Akamai content, although Youtube content should still come from an optimal location.

I believed this was the case not very long ago but a quick check now suggests that Google has done some catch-up and doing a better job recognising Spark address space and pointing to a Spark Akamai cluster for some sites sampled - an example:

$ dig @8.8.8.8 download.tvnz.co.nz
a1093.g.akamai.net. 19 IN A 219.88.186.89
a1093.g.akamai.net. 19 IN A 219.88.186.97

$ dig @ns1.xtra.co.nz download.tvnz.co.nz
a1093.g.akamai.net. 18 IN A 219.88.187.33
a1093.g.akamai.net. 18 IN A 219.88.187.35

$ dig @ns2.xtra.co.nz download.tvnz.co.nz
a1093.g.akamai.net. 20 IN A 219.88.186.97
a1093.g.akamai.net. 20 IN A 219.88.186.89

Related matches for www.stuff.co.nz. The Google DNS queries are a bit slower but seems they may not be pointing offshore quite the way they used to.

plambrechtsen

1948 posts

Uber Geek
Inactive user

#1126220 11-Sep-2014 06:51

cyril7: Just like to update that with the help of Spark staff who frequent here (thanks guys) the schools router was removed from the blacklist, it would seem that a machine in the school seems to be infested with malware that took part in an amplification attack, naturally I have requested the site admin take a look into that aspect.

Cheers
Cyril

You're welcome :). You could have just emailed me and I would have helped you out Cyril.

pristle

158 posts

Master Geek

Subscriber

#1126632 11-Sep-2014 16:31

This just arrived (although I don't know who Pardaman is) :

" Good afternoon Pardaman

Thank you for contacting Spark Broadband with regards to the recent outage over the weekend. By now your broadband should have resumed normal service. Our tests on your connection show it is now functioning normally. If you require further assistance, please call The 24/7 Broadband Helpdesk on 0800 225598.

You may be wondering what and how this happened and what we here at Spark are doing about, please read the following to get an understanding of the cause of the outage:
Cyber criminals based overseas appear to have been attacking web addresses in Eastern Europe, and were bouncing the traffic off Spark customer connections, in what is known as a distributed denial of service (DDoS) attack.

The DDoS attack was dynamic, predominantly taking the shape of an ‘amplified DNS attack’ which means an extremely high number of connection requests – in the order of thousands per second - were being sent to a number of overseas web addresses with the intention of overwhelming and crashing them. Each of these requests, as it passes through our network, queries our DNS server before it passes on – so our servers were bearing the full brunt of the attack.

While the Spark network did not crash, we did experience extremely high traffic loads hitting our DNS servers which meant many customers had either slow or at times no connectivity (as their requests were timing out). There were multiple attacks, which were dynamic in nature. They began on Friday night, subsided, and then began again early Saturday, continuing over the day. By early Sunday morning traffic levels were back to normal and have remained so since. We did see the nature of the attack evolve over the period, possibly due to the cyber criminals monitoring our response and modifying their attack to circumvent our mitigation measures – in a classic ‘whack a mole’ scenario.

How did they get access through the Spark Network?
Since the attacks began we have had people working 24/7 to identify the root causes, alongside working to get service back to normal. During the attack, we observed that a small number of customer connections were involved in generating the vast majority of the traffic. This was consistent with customers having malware on their devices and the timing coincided with other DNS activity related to malware in other parts of the world.

However, while we’re not ruling out malware as a factor, we have also identified that cyber criminals have been accessing vulnerable customer modems on our network. These modems have been identified as having “open DNS resolver” functionality, which means they can be used to carry out internet requests for anyone on the internet. This makes it easier for cyber criminals to ‘bounce’ an internet request off them (making it appear that the NZ modem was making the request, whereas it actually originates from an overseas source). Most of these modems were not supplied by Spark and tend to be older or lower-end modems.

What remains clear is that good end user security remains an important way to combat these attacks. With the proliferation of devices in households, that means both the security within your device and the security of your modem.

What did Spark do?
We have now disconnected those modems from our network and are contacting all the affected customers. We have also taken steps at a network level to mitigate this modem vulnerability. We are now in the process of scanning our entire broadband customer base to identify any other customers who may be using modems with similar vulnerabilities and will be contacting those identified customers in due course to advise them on what they should do.

With respect to malware we continue to strongly encourage our customers to keep their internet device security up to date, conduct regular scans and regularly update the operating software and firmware on their home network. We also continue to advise customers not to click on suspicious links or download files when they are not sure of the contents.

We have also taken steps at the network level to make it more difficult for cyber criminals to exploit the DNS open resolver modem vulnerability and we’re using the latest technology to strengthen our network monitoring and management capabilities. For security reasons we can’t detail these steps, however this is an ongoing battle to stay one step ahead of cyber criminals who are continually using more and more sophisticated tactics.

Why only Spark?
We can’t say what other networks experienced. However, it’s typical that cyber criminals look for clusters of IP addresses to use in any particular denial of service attack. That makes it more likely that these IP addresses belong to the customers of a single ISP – even more likely with a large ISP like Spark. They do this because it’s then easier for them to monitor the steps the ISP is taking to mitigate the attack and change their tactics accordingly. We definitely saw this happening over the weekend.

Is Spark offering Compensation?
We are very sorry for the inconvenience and hassle these issues have caused. We appreciate how important the internet is to you and assure you we take our services to you seriously, however we are not offering compensation to customers for the events over the weekend.

DDoS (Distributed Denial of Service) attacks are happening all the time all over the world and all ISPs like us must constantly deal with them and what we do is manage incidents to our best ability, as and when they happen. Normally we monitor and can deal with them day to day, however the difference this weekend was the huge volume of traffic generated by the attack passing through our network for overseas destinations. Our internet service is best efforts and while we are committed to providing a consistent and reliable service, these services can occasionally go down from time to time.

I hope that this information helps."

nickt1

28 posts

Geek

#1126734 11-Sep-2014 19:38

Finally back online after both hard reset of router and Modem yesterday.
I am just thankful I am not running a major business enterprise otherwise I would be feeling really Sparked off!
Thanks Pahutukawa and Jarsky for your help-much appreciated.

quickymart

13932 posts

Uber Geek

ID Verified

#1126801 11-Sep-2014 21:37

nickt1: I am just thankful I am not running a major business enterprise otherwise...

...you would be using a business-grade connection?

BarTender

3606 posts

Uber Geek

ID Verified

Trusted

Lifetime subscriber

#1126878 12-Sep-2014 01:09

quickymart:
nickt1: I am just thankful I am not running a major business enterprise otherwise...

...you would be using a business-grade connection?

...and probably running a modem from the ISP that was made within the last 6+ years
...and would have anti-virus / malware software running on your computers
...and have a firewall with threat protection that prevents any malware infested machines from connecting to the internet.

1 | ... | 15 | 16 | 17 | 18 | 19 | 20 | 21