Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.


Filter this topic showing only the reply marked as answer View this topic in a long page with up to 500 replies per page Create new topic
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | ... | 21
310 posts

Ultimate Geek
+1 received by user: 22


  Reply # 1123578 7-Sep-2014 11:35
Send private message

sbiddle:
techmeister:
Kirdog:  Very slow and laggy ingress play on Spark mobile data today and yesterday evening. :(
Just out of Curiosity: Why Voda customers are not affected by that? ( http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=11320100 < some very strange excuse detected here, as my opinion) 

Also someone mentioned that its "paid attack", it can be/why not...


Yes I agree, I smell a rat.
Why only Spark customers?


How about the sample fact they have over 50% of NZers as broadband customers, hence they're more than twice the size of the next biggest network? This makes something on this scale so much harder to contain.






I the herald article the Spark guy said a handful of customers did this and they all happened to be Spark customers?




Now on 2talk Network and it's better.



810 posts

Ultimate Geek
+1 received by user: 191
Inactive user


  Reply # 1123581 7-Sep-2014 11:45
Send private message

Spark initially (after 13-15hours of "not knowing what the problem was" overnight) just said a handful of customers with malware sending data overseas; clearly not the case but Spark certainly fumbled the ball PR wise. Then it continued, when they did come up with a fix, switching DNS servers, they NEVER put working addresses up on their service status; and blamed their customers instead of their own security or even just the nature of the business.

 
 
 
 


176 posts

Master Geek
+1 received by user: 9


  Reply # 1123589 7-Sep-2014 12:04
Send private message

sbiddle:
pohutukawa:
The answer to all of this is that if Spark are NZ's largest ISP then they are also making the most money and should be providing for occurrences like this.


Such a comment shows a complete lack of understanding of DNS amplification and the fact it is the scariest thing to hit an ISP.

Exactly how should they be "providing for an occurrence like this" and how should such an attack be prevented? You've clearly got the answers so could be very rick this week.

At the end of the day there are going to be a lot of people trashing Spark and thinking they're experts on the matter. The scary thing is there is absolutely nothing stopping this exact same thing from happening to Spark again next week, or happening to any other ISP in NZ.

I predict a day in the not too distant future where the global internet goes down for a period of time. It's not a matter of if this will happen, but when this will happen.






I both understand the DNS amplification DDOS vector and how to mitigate in as much as such attacks can be. I concede that the issues are not simple and extend to wider questions such as source IP validation, etc.

However, the Spark response has been pretty dismal IMO, in particular keeping customers informed. I acknowledge that I should have made clear was my main point.

dwl

363 posts

Ultimate Geek
+1 received by user: 43


  Reply # 1123608 7-Sep-2014 13:17
Send private message

PaulBags:
dwl: ... Workarounds like officially suggesting Google is a brave move.

Except they aren't, service status still suggests to use the dns servers that are the ones actually having issues...

The service status advice was rather inappropriate.  For those who use Facebook they will have seen this post which I assume was officially sanctioned:


There is now an update telling users to change back to the addresses which were in that service status advice.  However, not everyone will see this so there may be a slight remaining legacy from this issue.

14447 posts

Uber Geek
+1 received by user: 1898


  Reply # 1123609 7-Sep-2014 13:27
Send private message

dwl:
PaulBags:
dwl: ... Workarounds like officially suggesting Google is a brave move.

Except they aren't, service status still suggests to use the dns servers that are the ones actually having issues...

The service status advice was rather inappropriate.  For those who use Facebook they will have seen this post which I assume was officially sanctioned:


There is now an update telling users to change back to the addresses which were in that service status advice.  However, not everyone will see this so there may be a slight remaining legacy from this issue.


Somewhat ironically, it was international traffic that was affected, so I couldn't access facebook anyway while it was out. Don't think they had that info on their own status page, but I believe different people operate the social media  vs the actual status page. 

I just can't understand why no other ISPs were affected by this, NZ or even around they world. Does telecom aka spark just have the worst luck of any ISP/phone provider? It seems like they get the the worst luck when it comes to Murphies Law, and the name change hasn't helped stop it. Vodafone must have almost a similar number of customers these days, and would suspect that proportionally, they would have the same number of infected computers on their network. I wonder if other ISPs will come out in the next few days to say that they are working on stopping the same sort of thing happening on their networks.


3758 posts

Uber Geek
+1 received by user: 2274

Trusted
Spark NZ

  Reply # 1123611 7-Sep-2014 13:32
9 people support this post
Send private message

I'm hesitant to go into too much detail at this stage since the issue is still under (very) active investigation, but I can say that we have seen symptoms (in terms of DNS response times from all DNS servers and DNS lookup results) abate to normal, expected performance since approx 4am this morning.

We are still seeing a significant amount of unexpected traffic that we are working towards being able to identify as selectively as possible so as not to hurt legitimate users, but at the moment performance is as expected (from our distributed test network stats and from server based metrics like load balancer stats, DNS server CPU/RAM/QPS stats).

There will be no official comment made at this stage about the root cause (at least through this pretty direct technical channel) because it looks like the attack has evolved over time to avoid some of the mitigations put in place early on during the attack. 

At this stage, if you're the sort of person comfortable flicking between DNS servers, I would personally recommend switching back to Sparks auto assigned DNS servers, but be ready to flick back to the Google servers should the attack escalate or find a way to dodge current mitigations.

From the (many) people working on this (some of whom have been working for ~32 of the last ~40 hours), we appreciate the understanding that many on this board have displayed in the face of one of the longest sustained attacks we've seen. There's certainly a HUGE amount less of the purely emotional and frankly insulting comments seen on FB and stuff.co.nz comments sections. Many thanks for that.

If anyone using the Spark DNS servers is currently seeing regular DNS lookup failures, I'd appreciate a PM highlighting that.

Cheers - N


546 posts

Ultimate Geek
+1 received by user: 106


Reply # 1123612 7-Sep-2014 13:44
Send private message

Talkiet:There will be no official comment made at this stage about the root cause (at least through this pretty direct technical channel) because it looks like the attack has evolved over time to avoid some of the mitigations put in place early on during the attack. 

Which begs the questions, was this a trial run for a larger attack on an bigger overseas ISP or govt? :)

Clint

Circumspice
515 posts

Ultimate Geek
+1 received by user: 119

Trusted
Lifetime subscriber

  Reply # 1123615 7-Sep-2014 13:55
Send private message

Is it just me or are things not entirely back to normal yet... I'm still using Google's DNS, tried going back to automatic DNS and it's still broken.

11 posts

Geek
+1 received by user: 4


  Reply # 1123616 7-Sep-2014 13:57
One person supports this post
Send private message

The DNS DDOS attack that i have seen involves incorrect configured customer modems.  Quite a few modems are shipped with a factory default of having their inbuilt resolver accessible both from the internal network (which is why your pc normally sets your modem/router to be your DNS) as well as the external network.  This means that as someone external to your network I can set your modem to be my DNS server -or if i am evil, I can send a large number of DNS requests to your modem.  Your modem being what it is, will then forward these queries to your ISPs DNS server. 

Basically this means that I can sit in another country send thousands of request to your customers ip address range and watch things fall apart.  The ISP DNS server in most cases is not the target, it is just an victim of the additional queries (which are often to non-existent hosts, in existent domains).

As an ISP there are two simple fixes.  1. Fix your customer modems (or regularly check them, then notify them if they have an "open-resolver") 2. Block incoming DNS requests for all customer addresses except those that specifically run/host DNS services.

You can check your Modem to see if it responds to external DNS queries at www.openresolverproject.org



3758 posts

Uber Geek
+1 received by user: 2274

Trusted
Spark NZ

  Reply # 1123617 7-Sep-2014 13:58
Send private message

paulchinnz: Is it just me or are things not entirely back to normal yet... I'm still using Google's DNS, tried going back to automatic DNS and it's still broken.


We are seeing normal response times against all our DNS servers, and 100% successful lookups. If you're seeing a problem still, please be specific. What is failing?

Cheers -N


3758 posts

Uber Geek
+1 received by user: 2274

Trusted
Spark NZ

  Reply # 1123618 7-Sep-2014 14:01
Send private message

mikesmac: [snip]
You can check your Modem to see if it responds to external DNS queries at www.openresolverproject.org


If anyone sees themselves operating as an open resolver, and they DO NOT HAVE PORT 25 UNBLOCKING ENABLED, please let me know.

So if you have Port 25 unblocking enabled, AND you are running an open resolver, you may show up as an open resolver. If either of those clauses are untrue then you shouldn't be showing up.

Cheers - N

ps. Yes I know port 25 isn't DNS.

6359 posts

Uber Geek
+1 received by user: 316

Trusted
Subscriber

  Reply # 1123637 7-Sep-2014 15:18
6 people support this post
Send private message

Hi, just to report back, returned to ns1/2.xtra.co.nz and all normal, if your still having issues then its something YOU are doing wrong.

Neil and the team thanks for getting this backup and running, I am sure you will be doing a lot of corporate/team soul searching as to how best to mitigate this in future, and I am sure other local ISPs will be checking the rigging in a similar manner as this is a big wake up for all.

As for those that that seem keen on kicking the team while down, you clearly don't have any idea of what is being dealt with here, this is serious stuff, some could say more serious than the rants and wants of politics, as TCP is a fundamental to the internet, there is a lot of trust that all players are reasonable and honest, as time moves on this is clearly not the case, look over your shoulder if you have any public facing networks, this applies to any ISP, corporate or independent.

I suspect most other ISPs are just as vunerable, if not more as they have less resources to stack the odds against attack.

Cyril

27258 posts

Uber Geek
+1 received by user: 6688

Moderator
Trusted
Biddle Corp
Lifetime subscriber

  Reply # 1123658 7-Sep-2014 15:44
Send private message

mikesmac: The DNS DDOS attack that i have seen involves incorrect configured customer modems.  


And it's not just some older modems that respond to lookups, there have been plenty of people on here post issues where they've seen excessive traffic because they've decided to disable their firewall.

There is one thread in particular from a year or two from a guy who posted on here who had purposely disabled his firewall and saw no reason why it should be enabled because he had a firewall enabled on his PC.



133 posts

Master Geek
+1 received by user: 41


  Reply # 1123680 7-Sep-2014 16:17
One person supports this post
Send private message

Talkiet: There's certainly a HUGE amount less of the purely emotional and frankly insulting comments seen on FB and stuff.co.nz comments sections. Many thanks for that.


Sometimes its nice to see there are some appreciative and understanding customers :)


3674 posts

Uber Geek
+1 received by user: 1387

Subscriber

  Reply # 1123792 7-Sep-2014 19:59
One person supports this post
Send private message

Jarsky:
Talkiet: There's certainly a HUGE amount less of the purely emotional and frankly insulting comments seen on FB and stuff.co.nz comments sections. Many thanks for that.


Sometimes its nice to see there are some appreciative and understanding customers :)




That's nice to see. I was getting so angry seeing the comments on Spark's status on facebook. I realise changing your DNS is not simple for some but holy balls some people were just plain rude.

I feel sorry for the engineers, what a cr4p thing to happen on a weekend!!

1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | ... | 21
Filter this topic showing only the reply marked as answer View this topic in a long page with up to 500 replies per page Create new topic



Twitter »

Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.