ISPs role in unsolicited traffic

Forums › 2degrees (including Slingshot, Orcon, Flip, Stuff Fibre, MyRepublic, 2talk and Vocus) › ISPs role in unsolicited traffic

mattie47

79 posts

Master Geek
+1 received by user: 1

#111127 24-Oct-2012 22:13

This is probably a pretty grey area, and probably something ISPs normally wouldn't have to deal with, other than customers ringing up complaining about why their data usage is through the roof when they haven't been doing anything. Anyhow to my story....

So I've just hit my 100gb cap. Something that normally never happens unless I'm a few days away from the end of the month and I know I've got data to burn to download stuff. Even then I'd be lucky to reach it. I'm currently 20 days into my broadband month, and I've also done no major downloads that I can really think of. If I'm being very generous, I've probably done 50gb.

To my shock and horror I checked my slingshot account yesterday to see I'd done 95% of my cap. Somewhat freaked out thinking what on earth is going on in my network. I've spent most of this afternoon trying to pin point what was going on. I eventually ironed out it wasn't any devices within my network initiating anything so I started to look on my firewall for clues. I did a TCPDump (basically a wireshark capture for those who don't know what that is) and could see a lot of what looks like DNS requests. As I started to analyse it more, I came across some very interesting packets.

Example: 21:32:34.352324 IP 108.162.207.5.http > 10.1.1.254.domain: 14259+ [1au] ANY? isc.org. (36)

Well what the heck is that? At this point I'd like to show my network topology:

INTERNET-------|LinksysAG310|.1----10.0.0.0/24--DMZofEverything------WAN--.254|PFSenseFirewall].254----192.168.0.0/24---internal.

I also looked at some rather interesting graphs on my firewall:
pfsense

As you can see, my traffic IN is at 63GB, and I've uploaded 46GB. That's crazy! but it's correct. Slingshot didn't slow me down for probably 20 hours after I hit my 100gb.

Back to that packet from before. A quick google on "isc.org dns ddos" reveals countless pages on DNS servers geting hit with fake requests exactly like that packet above. One example being this page that goes over it: http://foxpa.ws/2010/07/21/thwarting-the-isc-org-dns-ddos/

For anyone who would like to look, I did a TCPdump on my wan interface on the firewall for about 2 minutes here: http://mattie47.com/Downloads/capture.txt as you can see, the amount of dns and udp traffic is quite large (also throttled obviously right now).

So to my point, what are users suppose to do in this situation? Do ISPs have any role in it? This isn't traffic that I've initiated, and these requests constantly come to my IP as soon as I reconnect the modems PPP connection.

The only solution (for me) that I can think of, is to disconnect my modem over night and hope I get a new dynamic IP. I tried 30 min, but I'm still stuck with the same one.

The alternative is to drop the requests on my firewall as well (which I should, but I'm not 100% sure how), but that's still using data with the attack coming to me. It would only stop upload.

I could be wrong, but the next user to pick up my dynamic IP will encounter the same problem, which is why I'm unsure of what should be done in this position.

I guess you could say I'm posting this here because I'm hoping someone at slingshot will see it (please someone with networking knowledge and understanding like CCNA or CCNP level :/)

Anyone else got thoughts on this?

Thanks,

Matt

Media centre PC - Case Silverstone LC17 case with 430w PSU, MOBOAsus P5VD2-MX, CPU Intel 2.8Ghz Dual Core RAM 2 Gig, HDD 600gb WD Sata II, Tuners Hauppauge HVR-3000, Play TV BT878 UHF/VHF, Video Asus 9400GT Silent, OS Windows 7 Pro
Main PC - Case Raidmax Ninja Black, MOBO Asus P6T, CPU ,Intel i7 930 RAM 6 Gig, HDD X3 Seagate 1tb , Tuner Skystar 2 DVB-S, Video NVIDIA 9600GT OC Edition, OS Windows 7 Pro

1 | 2

3892 posts

Uber Geek
+1 received by user: 164

#706011 24-Oct-2012 22:40

My first thought is way are you accepting traffic on port 53 and not just dropping it, or did I miss understand the attack?

But given that you've for a solid 500kbit's of out going traffic it looks like you're not dropping things you should be, but accepting it and doing something with it.

Why are you double nat'ing and not just terminating the PPP session on your pfs box?

Why are you DMZing everything? Do you actually need the incoming ports?

Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz

frizianz

105 posts

Master Geek
+1 received by user: 1

#706012 24-Oct-2012 22:46

First things first - As said above Double Nat = Puke.

Secondly - Always setup a stateful firewall with default policies to drop with only ports / traffic that is required allowed (Also where possibly specify allowed source addresses or conditions)

Best to just drop it and not reject (as that will return a response that it has been rejected). This way someone probing will simply thing nothing on that port exists there.

Cheers,

Fraser

frizianz.com || twitter.com/frizianz

mattie47

79 posts

Master Geek
+1 received by user: 1

#706013 24-Oct-2012 22:47

Also I'll point out my slingshot month starts on the 4th of each month. Secondly, slingshot, if you were to look at my account at all, I PM'd you with those details a few weeks back...

I also have an internal DNS server, but port 53 isn't forwarded from the firewall.

@DonGould, yeah I thought someone might pick up on that. I tried setting up my modem as half bridged but never got it working for some reason. Having it DMZ'd to the firewall has worked fine for the last 2 years.

Admittedly my PFSense is a VM, which has never caused me an issue. VM wan is bridged to internal pci nic with tcp/ip and everything else turned off within windows (running server 2008 R2).

I've got a wireshark capture on that external interface here: http://mattie47.com/Downloads/dns%20dos%20wireshark.pcap

mattie47

79 posts

Master Geek
+1 received by user: 1

#706017 24-Oct-2012 22:58

mattie47: I also have an internal DNS server, but port 53 isn't forwarded from the firewall.

Wow I take that back. I was looking at my port forwarding rules which didn't show 53 anywhere so presumed was closed. I just did a capture on the internal NIC which showed the same traffic. This got me worried. A quick online port check showed 53 as open (what?).

Having a look again around PFSense showed there's a DNS forwarder page I must have skimmed over. Turns out I had "Enable DNS forwarder" ticked. Okay, that's port 53 traffic going to internal now dropped...

On the Double NAT thing above, is it really double Nat'd? Since the modem is set to DMZ everything to the firewall, I would have thought it was only the firewall doing NAT. I did have another modem set up as half bridged but was seeing frequent connection drop outs (before half bridge and after.) I like the linksys as it holds a solid connection and has SNMP, so I just use that to monitor the br0 interface.

Cheers

DonGould

3892 posts

Uber Geek
+1 received by user: 164

#706018 24-Oct-2012 22:59

Yip, what Fraser said... drop the unrelated/established traffic....

oh bring on IpV6 with ICMPv6 and data caps... do I see this ending badly?

Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz

DonGould

3892 posts

Uber Geek
+1 received by user: 164

#706020 24-Oct-2012 23:04

is that what DNS Forwarder is? no... (/me wonders off to do some googling, I'm sure that's not what that's about).

You should be dropping all incoming 53/t/udp traffic requests for unrelated established. From what you've said it will just come though as part of the DMZ.

You can use the dns cache on your pfs if you want, but frankly why? dns traffic is so little now days what's the point? Your machines will cache anyway... or are you running 10 classes of 20 computers behind this pup?

yes you are double natting. 10.x is the first layer 192.x is the next layer.

D

Promote New Zealand - Get yourself a .kiwi.nz domain name!!!

Check out mine - i.am.a.can.do.kiwi.nz - don@i.am.a.can.do.kiwi.nz

Samsung

Shop now on Samsung phones, tablets, TVs and more (affiliate link).

mattie47

79 posts

Master Geek
+1 received by user: 1

#706022 24-Oct-2012 23:22

DonGould: is that what DNS Forwarder is? no... (/me wonders off to do some googling, I'm sure that's not what that's about).

You should be dropping all incoming 53/t/udp traffic requests for unrelated established. From what you've said it will just come though as part of the DMZ.

Doing a wireshark capture on the internal NIC shows the unsolicited DNS requests traffic stop as soon as I disable DNS forwarder. Online port scan shows 53 as closed as soon as I do it also...

You can use the dns cache on your pfs if you want, but frankly why? dns traffic is so little now days what's the point? Your machines will cache anyway... or are you running 10 classes of 20 computers behind this pup?

I use DNS internally for Active Directory and for name resolution of each device. I will add this is of course just my own home network ;)

yes you are double natting. 10.x is the first layer 192.x is the next layer.

Of course.... Here's me forgetting the routing being a factor in the NAT. I might have a look at setting up Half bridging again in the weekend.

Cheers for the discussion by the way...

mattie47

79 posts

Master Geek
+1 received by user: 1

#706031 24-Oct-2012 23:53

DonGould: oh bring on IpV6 with ICMPv6 and data caps... do I see this ending badly?

On that note, does Slingshot give out IPv6 addresses yet? I'm going with no, but I find it interesting that there are a few other NZ ISPs that do give out IPv6.

I admit I find IPv6 quite interesting, and there is a lot of it I really still haven't got my head around, but it is an interesting protocol to play around with. I'm not sure how vast your networking knowledge is, but I thought I'd add I've just finished a couple months doing various work with IPv6 only OSPFv3 networks. Behaves quite similar to OSPFv2, but is still quite interesting :)

Back to the issue, traffic has dropped significantly as you can see here:

blue = down
yellow = up
green = total

So yeah, traffic IS now definitely getting blocked at my firewall, but I am still receiving unsolicited traffic. I'm about to shutdown my modem overnight to receive a new IP, but this hardly seems fair for the next user of that IP to be getting these requests. Or am I missing something here?

Cheers,

Matt

Zeon

3926 posts

Uber Geek
+1 received by user: 759

Trusted

#706035 25-Oct-2012 00:25

You sure that the internal DNS forwarder on PFsense isn't the issue? Have you got a deny all rule on the WAN interface? Possibly using you as an open relay.

Speedtest 2019-10-14

mattie47

79 posts

Master Geek
+1 received by user: 1

#706037 25-Oct-2012 00:35

Hmm fair point. I'll check in the morning. It just occurred to me I wasn't seeing the traffic in the firewall logs.

From memory I've got some permit statements for some services on wan followed by deny any any. Again I'll check that later.

Cheers for the thought.

mattie47

79 posts

Master Geek
+1 received by user: 1

#706578 25-Oct-2012 22:58

Zeon: You sure that the internal DNS forwarder on PFsense isn't the issue? Have you got a deny all rule on the WAN interface? Possibly using you as an open relay.

As above, after turning off the PFsense DNS Forwarder option, I'm not seeing any of this ridiculous amount of traffic.

To re-iterate what I was saying above, because I originally had that option on, the requests would get forwarded from from pfsense to the DNS servers I had listed in the config. This included my internal DNS server, but I've also got a slight feeling I could have been sending the requests also to Slingshots main two DNS servers. I can't confirm, as I don't think I kept any logs/traces on it. Actually since I'm still getting these DNS requests I might just try something....

Anywho here's my firewall rules. I don't think the new rule I created is doing anything though as there is nothing in the logs. Also ignore the 3389 double up, this is just how the nat portfowarding side of things autoconfigured things. As I have two different external ports I use to RDP back into with....

Also will point out pftop results after 24 hours since turning off DNS forwarder:

PR D SRC DEST RATE PEAK AVG BYTES STATE P
udp I 108.162.207.5:80 10.1.1.254:53 649 688 650 32M 0:1 *

So unsolicited traffic is only 32MB. Requests I don't want, but still a lot better than things were. Would be interesting to see if dial up speeds are having much of an effect on this though....

Quic Broadband

Move to New Zealand's best fibre broadband service (affiliate link). Free setup code: R587125ERQ6VE. Note that to use Quic Broadband you must be comfortable with configuring your own router.

mattie47

79 posts

Master Geek
+1 received by user: 1

#706582 25-Oct-2012 23:18

Oh wow....So I just put the config back to DNS forwarder, and re-added my original DNS config.

My original pfsense setup had my DNS server, and both slingshots in it....Bad move it appears. Just done a tcpdump straight after:

23:06:42.963098 IP (tos 0x0, ttl 64, id 27241, offset 0, flags [+], proto UDP (17), length 1500)
10.1.1.254.domain > 108.162.207.5.http: 8223 q: ANY? isc.org. 25/0/10 isc.org. NS ord.sns-pb.isc.org., isc.org.[|domain]
23:06:42.963161 IP (tos 0x0, ttl 64, id 27241, offset 1480, flags [+], proto UDP (17), length 1500)
10.1.1.254 > 108.162.207.5: udp
23:06:42.963215 IP (tos 0x0, ttl 64, id 27241, offset 2960, flags [none], proto UDP (17), length 454)
10.1.1.254 > 108.162.207.5: udp
23:06:42.964927 IP (tos 0x0, ttl 59, id 23773, offset 0, flags [+], proto UDP (17), length 1500)
nsrv1.tranzpeer.net.domain > 10.1.1.254.62918: 2235 q: ANY? isc.org. 26/5/11 isc.org. RRSIG[|domain]
23:06:42.965171 IP (tos 0x0, ttl 59, id 23773, offset 1480, flags [+], proto UDP (17), length 1500)
nsrv1.tranzpeer.net > 10.1.1.254: udp
23:06:42.965423 IP (tos 0x0, ttl 59, id 23773, offset 2960, flags [none], proto UDP (17), length 1018)
nsrv1.tranzpeer.net > 10.1.1.254: udp
23:06:42.999136 IP (tos 0x0, ttl 242, id 64866, offset 0, flags [none], proto UDP (17), length 64)
108.162.207.5.http > 10.1.1.254.domain: [no cksum] 31369+ [1au] ANY? isc.org. ar: . OPT UDPsize=4096 OK (36)
23:06:42.999682 IP (tos 0x0, ttl 64, id 31560, offset 0, flags [none], proto UDP (17), length 64)
10.1.1.254.56701 > nsrv1.tranzpeer.net.domain: [udp sum ok] 3315+ [1au] ANY? isc.org. ar: . OPT UDPsize=4096 OK (36)
23:06:42.999771 IP (tos 0x0, ttl 64, id 1548, offset 0, flags [none], proto UDP (17), length 64)
10.1.1.254.56701 > nsrv2.tranzpeer.net.domain: [udp sum ok] 3315+ [1au] ANY? isc.org. ar: . OPT UDPsize=4096 OK (36)
23:06:43.001661 IP (tos 0x0, ttl 128, id 12671, offset 0, flags [none], proto UDP (17), length 64)
10.1.1.254.45576 > nsrv1.tranzpeer.net.domain: [udp sum ok] 45046+% [1au] ANY? isc.org. ar: . OPT UDPsize=4000 OK (36)
23:06:43.004253 IP (tos 0x0, ttl 52, id 44368, offset 0, flags [DF], proto TCP (6), length 52)

23:11:33.960309 IP (tos 0x0, ttl 59, id 28300, offset 2960, flags [none], proto UDP (17), length 1160)
nsrv1.tranzpeer.net > 10.1.1.254: udp
23:11:33.960332 IP (tos 0x0, ttl 242, id 28230, offset 0, flags [none], proto UDP (17), length 64)
108.162.207.5.http > 10.1.1.254.domain: [no cksum] 8717+ [1au] ANY? isc.org. ar: . OPT UDPsize=4096 OK (36)
23:11:33.960334 IP (tos 0x0, ttl 59, id 28301, offset 0, flags [+], proto UDP (17), length 1500)
nsrv1.tranzpeer.net.domain > 10.1.1.254.44299: 55900 q: ANY? isc.org. 30/5/13 isc.org. RRSIG[|domain]
23:11:33.960709 IP (tos 0x0, ttl 59, id 28301, offset 1480, flags [+], proto UDP (17), length 1500)
nsrv1.tranzpeer.net > 10.1.1.254: udp
23:11:33.960711 IP (tos 0x0, ttl 59, id 28301, offset 2960, flags [none], proto UDP (17), length 1160)
nsrv1.tranzpeer.net > 10.1.1.254: udp
23:11:33.960879 IP (tos 0x0, ttl 64, id 50719, offset 0, flags [none], proto UDP (17), length 64)
10.1.1.254.vistium-share > nsrv1.tranzpeer.net.domain: [udp sum ok] 6794+ [1au] ANY? isc.org. ar: . OPT UDPsize=4096 OK (36)
23:11:33.960972 IP (tos 0x0, ttl 64, id 52182, offset 0, flags [none], proto UDP (17), length 64)
10.1.1.254.vistium-share > nsrv2.tranzpeer.net.domain: [udp sum ok] 6794+ [1au] ANY? isc.org. ar: . OPT UDPsize=4096 OK (36)
23:11:33.961344 IP (tos 0x0, ttl 128, id 17912, offset 0, flags [none], proto UDP (17), length 64)
10.1.1.254.7375 > nsrv1.tranzpeer.net.domain: [udp sum ok] 26250+% [1au] ANY? isc.org. ar: . OPT UDPsize=4000 OK (36)
23:11:33.963652 IP (tos 0x0, ttl 59, id 28302, offset 0, flags [+], proto UDP (17), length 1500)
nsrv1.tranzpeer.net.domain > 10.1.1.254.29986: 53427 q: ANY? isc.org. 30/5/13 isc.org. RRSIG[|domain]
23:11:33.963902 IP (tos 0x0, ttl 59, id 28302, offset 1480, flags [+], proto UDP (17), length 1500)
nsrv1.tranzpeer.net > 10.1.1.254: udp
23:11:33.963904 IP (tos 0x0, ttl 59, id 28302, offset 2960, flags [none], proto UDP (17), length 1160)
nsrv1.tranzpeer.net > 10.1.1.254: udp
23:11:33.994455 IP (tos 0x0, ttl 52, id 47066, offset 0, flags [DF], proto TCP (6), length 64)

I could be wrong, but it looks like I've also been forwarding the requests on to slingshots DNS servers, then getting a reply with large packets...Opps....Probably the reason why I ended up uploading/down so much data....

mattie47

79 posts

Master Geek
+1 received by user: 1

#711352 3-Nov-2012 13:33

About time I gave an update to this!

Firstly I would like to acknowledge and thank Slingshot in this situation. I had Bevin@slingshot PM me and provide me with some assistance in which they compensated me some data in which I was completely stoked about! This wasn't expected, and I didn't ask for it, so that made it that much better.

I've still got the same IP, so I'm still getting these requests, but since turning off DNS forwarding, the data used is minimal. I'd hoped to pick up a new IP and I'd left my modem unplugged for several hours at different occasions but still got the same one. Anyhow. Gonna have a go at bridging modem again :)

Matt

sbiddle

30853 posts

Uber Geek
+1 received by user: 9996

Retired Mod

Trusted

Biddle Corp

Lifetime subscriber

#711364 3-Nov-2012 13:42

Turning your modem off won't necessarily give you a new IP. I don't know about Slingshot specifically but a lot of ISP's use sticky DHCP leases these days.

mattie47

79 posts

Master Geek
+1 received by user: 1

#711370 3-Nov-2012 13:53

sbiddle: Turning your modem off won't necessarily give you a new IP. I don't know about Slingshot specifically but a lot of ISP's use sticky DHCP leases these days.

Yeah I did ask slingshot to expire my lease or whatever it is they would need to do, but didn't hear anything about it. I just remember the old days where you could reboot your modem and you'd get a new dynamic IP. I've been with slingshot I think for 3-4 years now, and in that time my IP has changed less than 10 times (which had been awesome, since it's basically a static IP), this would probably be the first time I've actually wanted a new IP.

I guess the reason why, is because I've got a HMA VPN set up on my firewall so when I access certain sites, it goes via the VPN. Problem is, when I enable the VPN interface, the DNS traffic floods it again, quite possibly because the VPN runs over port 53. To get around this, I'd probably need to create firewall rules, which I'd tried, but not got working sucessfully. Call me being lazy, but a new IP would make things easier.

1 | 2