Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.
Buying anything on Amazon? Please use the Geekzone Amazon aff link.




75 posts

Master Geek


Topic # 111127 24-Oct-2012 22:13 Send private message

This is probably a pretty grey area, and probably something ISPs normally wouldn't have to deal with, other than customers ringing up complaining about why their data usage is through the roof when they haven't been doing anything. Anyhow to my story....

So I've just hit my 100gb cap. Something that normally never happens unless I'm a few days away from the end of the month and I know I've got data to burn to download stuff. Even then I'd be lucky to reach it. I'm currently 20 days into my broadband month, and I've also done no major downloads that I can really think of. If I'm being very generous, I've probably done 50gb.

To my shock and horror I checked my slingshot account yesterday to see I'd done 95% of my cap. Somewhat freaked out thinking what on earth is going on in my network. I've spent most of this afternoon trying to pin point what was going on. I eventually ironed out it wasn't any devices within my network initiating anything so I started to look on my firewall for clues. I did a TCPDump (basically a wireshark capture for those who don't know what that is) and could see a lot of what looks like DNS requests. As I started to analyse it more, I came across some very interesting packets.

Example: 21:32:34.352324 IP 108.162.207.5.http > 10.1.1.254.domain: 14259+ [1au] ANY? isc.org. (36)

Well what the heck is that? At this point I'd like to show my network topology:

INTERNET-------|LinksysAG310|.1----10.0.0.0/24--DMZofEverything------WAN--.254|PFSenseFirewall].254----192.168.0.0/24---internal.

I also looked at some rather interesting graphs on my firewall:
pfsense


As you can see, my traffic IN is at 63GB, and I've uploaded 46GB. That's crazy! but it's correct. Slingshot didn't slow me down for probably 20 hours after I hit my 100gb.


Back to that packet from before. A quick google on "isc.org dns ddos" reveals countless pages on DNS servers geting hit with fake requests exactly like that packet above. One example being this page that goes over it: http://foxpa.ws/2010/07/21/thwarting-the-isc-org-dns-ddos/

For anyone who would like to look, I did a TCPdump on my wan interface on the firewall for about 2 minutes here: http://mattie47.com/Downloads/capture.txt as you can see, the amount of dns and udp traffic is quite large (also throttled obviously right now).


So to my point, what are users suppose to do in this situation? Do ISPs have any role in it? This isn't traffic that I've initiated, and these requests constantly come to my IP as soon as I reconnect the modems PPP connection.

The only solution (for me) that I can think of, is to disconnect my modem over night and hope I get a new dynamic IP. I tried 30 min, but I'm still stuck with the same one.

The alternative is to drop the requests on my firewall as well (which I should, but I'm not 100% sure how), but that's still using data with the attack coming to me. It would only stop upload.

I could be wrong, but the next user to pick up my dynamic IP will encounter the same problem, which is why I'm unsure of what should be done in this position.

I guess you could say I'm posting this here because I'm hoping someone at slingshot will see it (please someone with networking knowledge and understanding like CCNA or CCNP level :/)

Anyone else got thoughts on this?

Thanks,


Matt

View this topic in a long page with up to 500 replies per page Create new topic
 1 | 2
Have plan, send $NZD50m
3475 posts

Uber Geek
+1 received by user: 75

Subscriber

  Reply # 706011 24-Oct-2012 22:40 Send private message

My first thought is way are you accepting traffic on port 53 and not just dropping it, or did I miss understand the attack?

But given that you've for a solid 500kbit's of out going traffic it looks like you're not dropping things you should be, but accepting it and doing something with it.

Why are you double nat'ing and not just terminating the PPP session on your pfs box?

Why are you DMZing everything? Do you actually need the incoming ports?





Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - [email protected]


105 posts

Master Geek


  Reply # 706012 24-Oct-2012 22:46 Send private message

First things first - As said above Double Nat = Puke.

Secondly - Always setup a stateful firewall with default policies to drop with only ports / traffic that is required allowed (Also where possibly specify allowed source addresses or conditions)

Best to just drop it and not reject (as that will return a response that it has been rejected). This way someone probing will simply thing nothing on that port exists there.

Cheers,



Fraser



75 posts

Master Geek


  Reply # 706013 24-Oct-2012 22:47 Send private message

Also I'll point out my slingshot month starts on the 4th of each month. Secondly, slingshot, if you were to look at my account at all, I PM'd you with those details a few weeks back...

I also have an internal DNS server, but port 53 isn't forwarded from the firewall.

@DonGould, yeah I thought someone might pick up on that. I tried setting up my modem as half bridged but never got it working for some reason. Having it DMZ'd to the firewall has worked fine for the last 2 years.

Admittedly my PFSense is a VM, which has never caused me an issue. VM wan is bridged to internal pci nic with tcp/ip and everything else turned off within windows (running server 2008 R2).

I've got a wireshark capture on that external interface here: http://mattie47.com/Downloads/dns%20dos%20wireshark.pcap



75 posts

Master Geek


  Reply # 706017 24-Oct-2012 22:58 Send private message

mattie47: I also have an internal DNS server, but port 53 isn't forwarded from the firewall.



Wow I take that back. I was looking at my port forwarding rules which didn't show 53 anywhere so presumed was closed. I just did a capture on the internal NIC which showed the same traffic. This got me worried. A quick online port check showed 53 as open (what?).


Having a look again around PFSense showed there's a DNS forwarder page I must have skimmed over. Turns out I had "Enable DNS forwarder" ticked. Okay, that's port 53 traffic going to internal now dropped...


On the Double NAT thing above, is it really double Nat'd? Since the modem is set to DMZ everything to the firewall, I would have thought it was only the firewall doing NAT. I did have another modem set up as half bridged but was seeing frequent connection drop outs (before half bridge and after.) I like the linksys as it holds a solid connection and has SNMP, so I just use that to monitor the br0 interface.

Cheers

Have plan, send $NZD50m
3475 posts

Uber Geek
+1 received by user: 75

Subscriber

  Reply # 706018 24-Oct-2012 22:59 Send private message

Yip, what Fraser said... drop the unrelated/established traffic....

oh bring on IpV6 with ICMPv6 and data caps... do I see this ending badly?





Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - [email protected]


Have plan, send $NZD50m
3475 posts

Uber Geek
+1 received by user: 75

Subscriber

  Reply # 706020 24-Oct-2012 23:04 Send private message

is that what DNS Forwarder is? no... (/me wonders off to do some googling, I'm sure that's not what that's about).

You should be dropping all incoming 53/t/udp traffic requests for unrelated established. From what you've said it will just come though as part of the DMZ.

You can use the dns cache on your pfs if you want, but frankly why? dns traffic is so little now days what's the point? Your machines will cache anyway... or are you running 10 classes of 20 computers behind this pup?

yes you are double natting. 10.x is the first layer 192.x is the next layer.

D




Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - [email protected]




75 posts

Master Geek


  Reply # 706022 24-Oct-2012 23:22 Send private message

DonGould: is that what DNS Forwarder is? no... (/me wonders off to do some googling, I'm sure that's not what that's about).

You should be dropping all incoming 53/t/udp traffic requests for unrelated established. From what you've said it will just come though as part of the DMZ.



Doing a wireshark capture on the internal NIC shows the unsolicited DNS requests traffic stop as soon as I disable DNS forwarder. Online port scan shows 53 as closed as soon as I do it also...



You can use the dns cache on your pfs if you want, but frankly why? dns traffic is so little now days what's the point? Your machines will cache anyway... or are you running 10 classes of 20 computers behind this pup?



I use DNS internally for Active Directory and for name resolution of each device. I will add this is of course just my own home network ;)


yes you are double natting. 10.x is the first layer 192.x is the next layer.
 


Of course.... Here's me forgetting the routing being a factor in the NAT. I might have a look at setting up Half bridging again in the weekend.

Cheers for the discussion by the way...



75 posts

Master Geek


  Reply # 706031 24-Oct-2012 23:53 Send private message

DonGould: oh bring on IpV6 with ICMPv6 and data caps... do I see this ending badly?



On that note, does Slingshot give out IPv6 addresses yet? I'm going with no, but I find it interesting that there are a few other NZ ISPs that do give out IPv6.

I admit I find IPv6 quite interesting, and there is a lot of it I really still haven't got my head around, but it is an interesting protocol to play around with. I'm not sure how vast your networking knowledge is, but I thought I'd add I've just finished a couple months doing various work with IPv6 only OSPFv3 networks. Behaves quite similar to OSPFv2, but is still quite interesting :)


Back to the issue, traffic has dropped significantly as you can see here:



blue = down
yellow = up
green = total

So yeah, traffic IS now definitely getting blocked at my firewall, but I am still receiving unsolicited traffic. I'm about to shutdown my modem overnight to receive a new IP, but this hardly seems fair for the next user of that IP to be getting these requests. Or am I missing something here?


Cheers,

Matt

3038 posts

Uber Geek
+1 received by user: 216

Trusted
Subscriber

  Reply # 706035 25-Oct-2012 00:25 Send private message

You sure that the internal DNS forwarder on PFsense isn't the issue? Have you got a deny all rule on the WAN interface? Possibly using you as an open relay.







75 posts

Master Geek


  Reply # 706037 25-Oct-2012 00:35 Send private message

Hmm fair point. I'll check in the morning. It just occurred to me I wasn't seeing the traffic in the firewall logs.

From memory I've got some permit statements for some services on wan followed by deny any any. Again I'll check that later.

Cheers for the thought.



75 posts

Master Geek


  Reply # 706578 25-Oct-2012 22:58 Send private message

Zeon: You sure that the internal DNS forwarder on PFsense isn't the issue? Have you got a deny all rule on the WAN interface? Possibly using you as an open relay.


As above, after turning off the PFsense DNS Forwarder option, I'm not seeing any of this ridiculous amount of traffic. 

To re-iterate what I was saying above, because I originally had that option on, the requests would get forwarded from from pfsense to the DNS servers I had listed in the config. This included my internal DNS server, but I've also got a slight feeling I could have been sending the requests also to Slingshots main two DNS servers. I can't confirm, as I don't think I kept any logs/traces on it. Actually since I'm still getting these DNS requests I might just try something....


Anywho here's my firewall rules. I don't think the new rule I created is doing anything though as there is nothing in the logs. Also ignore the 3389 double up, this is just how the nat portfowarding side of things autoconfigured things. As I have two different external ports I use to RDP back into with....






Also will point out pftop results after 24 hours since turning off DNS forwarder: 

PR D SRC DEST RATE PEAK AVG BYTES STATE P
udp I 108.162.207.5:80 10.1.1.254:53 649 688 650 32M 0:1 *

So unsolicited traffic is only 32MB. Requests I don't want, but still a lot better than things were. Would be interesting to see if dial up speeds are having much of an effect on this though....



75 posts

Master Geek


  Reply # 706582 25-Oct-2012 23:18 Send private message

Oh wow....So I just put the config back to DNS forwarder, and re-added my original DNS config.

My original pfsense setup had my DNS server, and both slingshots in it....Bad move it appears. Just done a tcpdump straight after:

23:06:42.963098 IP (tos 0x0, ttl 64, id 27241, offset 0, flags [+], proto UDP (17), length 1500)
10.1.1.254.domain > 108.162.207.5.http: 8223 q: ANY? isc.org. 25/0/10 isc.org. NS ord.sns-pb.isc.org., isc.org.[|domain]
23:06:42.963161 IP (tos 0x0, ttl 64, id 27241, offset 1480, flags [+], proto UDP (17), length 1500)
10.1.1.254 > 108.162.207.5: udp
23:06:42.963215 IP (tos 0x0, ttl 64, id 27241, offset 2960, flags [none], proto UDP (17), length 454)
10.1.1.254 > 108.162.207.5: udp
23:06:42.964927 IP (tos 0x0, ttl 59, id 23773, offset 0, flags [+], proto UDP (17), length 1500)
nsrv1.tranzpeer.net.domain > 10.1.1.254.62918: 2235 q: ANY? isc.org. 26/5/11 isc.org. RRSIG[|domain]
23:06:42.965171 IP (tos 0x0, ttl 59, id 23773, offset 1480, flags [+], proto UDP (17), length 1500)
nsrv1.tranzpeer.net > 10.1.1.254: udp
23:06:42.965423 IP (tos 0x0, ttl 59, id 23773, offset 2960, flags [none], proto UDP (17), length 1018)
nsrv1.tranzpeer.net > 10.1.1.254: udp
23:06:42.999136 IP (tos 0x0, ttl 242, id 64866, offset 0, flags [none], proto UDP (17), length 64)
108.162.207.5.http > 10.1.1.254.domain: [no cksum] 31369+ [1au] ANY? isc.org. ar: . OPT UDPsize=4096 OK (36)
23:06:42.999682 IP (tos 0x0, ttl 64, id 31560, offset 0, flags [none], proto UDP (17), length 64)
10.1.1.254.56701 > nsrv1.tranzpeer.net.domain: [udp sum ok] 3315+ [1au] ANY? isc.org. ar: . OPT UDPsize=4096 OK (36)
23:06:42.999771 IP (tos 0x0, ttl 64, id 1548, offset 0, flags [none], proto UDP (17), length 64)
10.1.1.254.56701 > nsrv2.tranzpeer.net.domain: [udp sum ok] 3315+ [1au] ANY? isc.org. ar: . OPT UDPsize=4096 OK (36)
23:06:43.001661 IP (tos 0x0, ttl 128, id 12671, offset 0, flags [none], proto UDP (17), length 64)
10.1.1.254.45576 > nsrv1.tranzpeer.net.domain: [udp sum ok] 45046+% [1au] ANY? isc.org. ar: . OPT UDPsize=4000 OK (36)
23:06:43.004253 IP (tos 0x0, ttl 52, id 44368, offset 0, flags [DF], proto TCP (6), length 52)


23:11:33.960309 IP (tos 0x0, ttl 59, id 28300, offset 2960, flags [none], proto UDP (17), length 1160)
nsrv1.tranzpeer.net > 10.1.1.254: udp
23:11:33.960332 IP (tos 0x0, ttl 242, id 28230, offset 0, flags [none], proto UDP (17), length 64)
108.162.207.5.http > 10.1.1.254.domain: [no cksum] 8717+ [1au] ANY? isc.org. ar: . OPT UDPsize=4096 OK (36)
23:11:33.960334 IP (tos 0x0, ttl 59, id 28301, offset 0, flags [+], proto UDP (17), length 1500)
nsrv1.tranzpeer.net.domain > 10.1.1.254.44299: 55900 q: ANY? isc.org. 30/5/13 isc.org. RRSIG[|domain]
23:11:33.960709 IP (tos 0x0, ttl 59, id 28301, offset 1480, flags [+], proto UDP (17), length 1500)
nsrv1.tranzpeer.net > 10.1.1.254: udp
23:11:33.960711 IP (tos 0x0, ttl 59, id 28301, offset 2960, flags [none], proto UDP (17), length 1160)
nsrv1.tranzpeer.net > 10.1.1.254: udp
23:11:33.960879 IP (tos 0x0, ttl 64, id 50719, offset 0, flags [none], proto UDP (17), length 64)
10.1.1.254.vistium-share > nsrv1.tranzpeer.net.domain: [udp sum ok] 6794+ [1au] ANY? isc.org. ar: . OPT UDPsize=4096 OK (36)
23:11:33.960972 IP (tos 0x0, ttl 64, id 52182, offset 0, flags [none], proto UDP (17), length 64)
10.1.1.254.vistium-share > nsrv2.tranzpeer.net.domain: [udp sum ok] 6794+ [1au] ANY? isc.org. ar: . OPT UDPsize=4096 OK (36)
23:11:33.961344 IP (tos 0x0, ttl 128, id 17912, offset 0, flags [none], proto UDP (17), length 64)
10.1.1.254.7375 > nsrv1.tranzpeer.net.domain: [udp sum ok] 26250+% [1au] ANY? isc.org. ar: . OPT UDPsize=4000 OK (36)
23:11:33.963652 IP (tos 0x0, ttl 59, id 28302, offset 0, flags [+], proto UDP (17), length 1500)
nsrv1.tranzpeer.net.domain > 10.1.1.254.29986: 53427 q: ANY? isc.org. 30/5/13 isc.org. RRSIG[|domain]
23:11:33.963902 IP (tos 0x0, ttl 59, id 28302, offset 1480, flags [+], proto UDP (17), length 1500)
nsrv1.tranzpeer.net > 10.1.1.254: udp
23:11:33.963904 IP (tos 0x0, ttl 59, id 28302, offset 2960, flags [none], proto UDP (17), length 1160)
nsrv1.tranzpeer.net > 10.1.1.254: udp
23:11:33.994455 IP (tos 0x0, ttl 52, id 47066, offset 0, flags [DF], proto TCP (6), length 64)



I could be wrong, but it looks like I've also been forwarding the requests on to slingshots DNS servers, then getting a reply with large packets...Opps....Probably the reason why I ended up uploading/down so much data....



75 posts

Master Geek


  Reply # 711352 3-Nov-2012 13:33 Send private message

About time I gave an update to this!

Firstly I would like to acknowledge and thank Slingshot in this situation. I had Bevin@slingshot PM me and provide me with some assistance in which they compensated me some data in which I was completely stoked about! This wasn't expected, and I didn't ask for it, so that made it that much better.

I've still got the same IP, so I'm still getting these requests, but since turning off DNS forwarding, the data used is minimal. I'd hoped to pick up a new IP and I'd left my modem unplugged for several hours at different occasions but still got the same one. Anyhow. Gonna have a go at bridging modem again :)

Matt

20113 posts

Uber Geek
+1 received by user: 1692

Moderator
Trusted
Biddle Corp
Subscriber

  Reply # 711364 3-Nov-2012 13:42 Send private message

Turning your modem off won't necessarily give you a new IP. I don't know about Slingshot specifically but a lot of ISP's use sticky DHCP leases these days.



75 posts

Master Geek


  Reply # 711370 3-Nov-2012 13:53 Send private message

sbiddle: Turning your modem off won't necessarily give you a new IP. I don't know about Slingshot specifically but a lot of ISP's use sticky DHCP leases these days.


Yeah I did ask slingshot to expire my lease or whatever it is they would need to do, but didn't hear anything about it. I just remember the old days where you could reboot your modem and you'd get a new dynamic IP. I've been with slingshot I think for 3-4 years now, and in that time my IP has changed less than 10 times (which had been awesome, since it's basically a static IP), this would probably be the first time I've actually wanted a new IP.

I guess the reason why, is because I've got a HMA VPN set up on my firewall so when I access certain sites, it goes via the VPN. Problem is, when I enable the VPN interface, the DNS traffic floods it again, quite possibly because the VPN runs over port 53. To get around this, I'd probably need to create firewall rules, which I'd tried, but not got working sucessfully. Call me being lazy, but a new IP would make things easier.

 1 | 2
View this topic in a long page with up to 500 replies per page Create new topic




Twitter »
Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





Trending now »

Hot discussions in our forums right now:

My un-consented UFB install
Created by thurthur, last reply by Handle9 on 29-Nov-2014 16:07 (84 replies)
Pages... 4 5 6


This is the end ...
Created by joker97, last reply by joker97 on 29-Nov-2014 13:32 (62 replies)
Pages... 3 4 5


Excessive Vodafone Support call waiting times 75 mins and then i have to wait an hour for a callback
Created by i4n, last reply by dejadeadnz on 29-Nov-2014 16:39 (38 replies)
Pages... 2 3


Gigatown winner town and plans
Created by freitasm, last reply by NonprayingMantis on 29-Nov-2014 07:40 (78 replies)
Pages... 4 5 6


Gull Employment Dispute.
Created by networkn, last reply by richms on 28-Nov-2014 17:57 (153 replies)
Pages... 9 10 11


What the hell MyRepublic!?
Created by gished, last reply by pohutukawa on 28-Nov-2014 17:45 (16 replies)
Pages... 2


The Warehouse pulling R18 games and DVD's
Created by semigeek, last reply by Geektastic on 27-Nov-2014 18:32 (64 replies)
Pages... 3 4 5


A couple of Lightbox updates...
Created by Lightbox, last reply by NZtechfreak on 27-Nov-2014 22:56 (15 replies)


Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.

Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.