DDoS Protection

Forums › IT Pro and developers › DDoS Protection

eXDee

4032 posts

Uber Geek

Trusted

#61412 16-May-2010 21:52

Straight to the point:
Situation is an Ubuntu 9.10 VPS (OpenVZ) running Apache is being hit by 1,500+ bots from a botnet. Method of attack is HTTP based, so basically reloading pages over and over, causing excessive server load (consumes all 2.5gb of ram and so everything freezes) and totally saturating the 100mbit pipe.
The VPS Host has some Cisco Guard thing, but because the attack looks like normal traffic they cant stop it.

Short of blocking them all individually (which is where the figure of 1,500+ comes from, thats how many are manually blocked with iptables, yet the attack still continues), what do i do?

1 | 2

729 posts

Ultimate Geek

#330748 16-May-2010 22:20

If it's a DDoS then blocking hosts on iptables will probably have little effect, as all of the requests will still be hitting your VPS. Especially so if the attack is filling the entire 100mbit pipe.

Only real solution is to talk to your host and get them to block it upstream, which they should be able to do.

marpada

475 posts

Ultimate Geek

#330753 16-May-2010 22:46

mod_evasive is useful to prevent this kind of attacks. You can combine it with fail2ban to automatically create deny iptables rule to block the attacking IPs.

insane

3239 posts

Uber Geek

ID Verified

Trusted

#330765 17-May-2010 00:03

Kraven: If it's a DDoS then blocking hosts on iptables will probably have little effect, as all of the requests will still be hitting your VPS. Especially so if the attack is filling the entire 100mbit pipe.

Only real solution is to talk to your host and get them to block it upstream, which they should be able to do.

This is really what you have to do if your connectivity is being saturated. I'm suprised your provider hasn't told you to leave as if you're getting hit with a contsant 100mbps of html requests then you can be sure they are getting more upstream.

What site or kind of site are you hosting, and who have you pissed off lately for this to happen to you? These are targeted attacks so presumably you've done somehting to anger a bunch of kiddies.

PenultimateHop

637 posts

Ultimate Geek

Trusted

#330785 17-May-2010 07:15

insane: What site or kind of site are you hosting, and who have you pissed off lately for this to happen to you? These are targeted attacks so presumably you've done somehting to anger a bunch of kiddies.

This is pretty much it. "Someone said something about someone else's (real|perceived) girlfriend on the Internet" -> attack.

I am surprised if the ISP has Cisco Guard and cannot provide a scrubbing function - this is precisely what they are supposed to do: receive inbound traffic, identify what's anomolous, remove it, pass on legit traffic. Very handy.

vborcan

2 posts

Wannabe Geek
Inactive user

#330795 17-May-2010 08:27

There are some much cheaper, Linux-based alternatives to Cisco Guard that dynamically use iptables. Andrisoft WANGuard for example.

Kyanar

4089 posts

Uber Geek

ID Verified

Trusted

#330957 17-May-2010 16:33

vborcan: There are some much cheaper, Linux-based alternatives to Cisco Guard that dynamically use iptables. Andrisoft WANGuard for example.

That's still burning your bandwidth, and still saturating your pipe. Effectively worthless.

PenultimateHop

637 posts

Ultimate Geek

Trusted

#330968 17-May-2010 16:55

Kyanar:
vborcan: There are some much cheaper, Linux-based alternatives to Cisco Guard that dynamically use iptables. Andrisoft WANGuard for example.

That's still burning your bandwidth, and still saturating your pipe. Effectively worthless.

I don't agree. The point of these tools is to protect your servers from exactly this scenario: your server or application is being overwhelmed by malicious traffic but needs to continue serving legitimate transactions. Assuming your ISP has one of these tools (Cisco Guard or the amusingly named WANGuard), they can scrub out the anomolous/malicious traffic and hand you the clean traffic.

There is quite a lot of value in that - look at all of the ISPs that offer "CleanPipe" type services based around Cisco Guard.

Equinox IT

Cloud spending continues to surge globally, but most organisations haven’t made the changes necessary to maximise the value and cost-efficiency benefits of their cloud investments. Download the whitepaper From Overspend to Advantage now.

Ragnor

8219 posts

Uber Geek

Trusted

#330993 17-May-2010 18:33

I think Kyanar was referring to the linux based alternatives that use iptables and run on the servers.

PenultimateHop

637 posts

Ultimate Geek

Trusted

#331023 17-May-2010 19:37

Ragnor: I think Kyanar was referring to the linux based alternatives that use iptables and run on the servers.

The product/solution he referred to is intended to be used in exactly the same way Cisco Guard is - at the service provider edge. It just happens to run on a Linux based server.

oldmaknz

536 posts

Ultimate Geek

#331032 17-May-2010 19:56

You should talk to your provider. There's very little a server admin can do against a large attack. If it was smaller using fail2ban/iptables rules can help.

Ragnor

8219 posts

Uber Geek

Trusted

#331358 18-May-2010 13:40

PenultimateHop:
Ragnor: I think Kyanar was referring to the linux based alternatives that use iptables and run on the servers.

The product/solution he referred to is intended to be used in exactly the same way Cisco Guard is - at the service provider edge. It just happens to run on a Linux based server.

Hmm that's interesting there aren't any throughput limitations with large iptables rulesets?

PenultimateHop

637 posts

Ultimate Geek

Trusted

#331385 18-May-2010 14:23

Ragnor: Hmm that's interesting there aren't any throughput limitations with large iptables rulesets?

It's been a long time since I've had to deal with it; but in the early mid 2000s I had no problems with ~25K rules and 150Mbps of sustained traffic (peaks to 200-300Mbps) in iptables on modest Xeon hardware. I was looking at nf-hipac though, which promised to be much faster. I have no idea what WANGuard is using under the hood.

vborcan

2 posts

Wannabe Geek
Inactive user

#331703 19-May-2010 00:52

PenultimateHop:
Ragnor: I think Kyanar was referring to the linux based alternatives that use iptables and run on the servers.

The product/solution he referred to is intended to be used in exactly the same way Cisco Guard is - at the service provider edge. It just happens to run on a Linux based server.

PenultimateHop: The usual deployment is indeed at the service provider edge, and thats how I use it. But being a relatively cheap software solution distributed as RPMs & DEBs, I thought that you might just deploy it on the server itself, or in front of the server. Obviously it will not protect from attacks that will exhaust the internet link, but it has a way to dynamically alter iptables rules to drop malicious traffic while allowing valid traffic to pass.

eXDee

4032 posts

Uber Geek

Trusted

#333452 23-May-2010 00:43

Wall of text incoming. This is probably going to put people off - sigh. But i do try and give as much information as possible.

marpada: mod_evasive is useful to prevent this kind of attacks.

No, it isnt. Mod_evasive only counts across a single thread/process in apache, which is totally useless unless you run apache as a single thread/process.
See here

insane:
Kraven: If it's a DDoS then blocking hosts on iptables will probably have little effect, as all of the requests will still be hitting your VPS. Especially so if the attack is filling the entire 100mbit pipe.

Only real solution is to talk to your host and get them to block it upstream, which they should be able to do.

This is really what you have to do if your connectivity is being saturated. I'm suprised your provider hasn't told you to leave as if you're getting hit with a contsant 100mbps of html requests then you can be sure they are getting more upstream.

What site or kind of site are you hosting, and who have you pissed off lately for this to happen to you? These are targeted attacks so presumably you've done somehting to anger a bunch of kiddies.

Its a gamesserver related website. So yes, i imagine its immature kiddies. The odd thing is when i googled some of the IP's when i noticed a bunch of similar ones from from singapore, there have been posts from pre 2008 about them attacking websites. These are in the 220.255.7.XXX (220.255.0.0/17) range. By that i mean .131-.139, .181-.199, .211-.229. Thats ~50 sequential IP's, which would appear to be dedicated boxes rather than just zombified computers?

Its only saturating the pipe IF the requests get through, because if they do get through they are requesting heaps of images which is what causes the large bandwidth.

I got up to manually blocking 1500+ IP's in iptables which basically blocked out the attack (until more bots came online). I've now swapped to blocking international subnets to make this more effective, and then blocking individual IP's for NZ and Australia (which i then emailed their ISP's about). This is not a problem, as the users of the site are all from australia/NZ, so other international visitors arent needed at all.

Yet this requires too much babysitting. Each day i have to check to see to see if fail2ban has picked anything up, and frequently recreate blank log files, since it cant handle such a vast amount of text. Once fail2ban gets overloaded, and more bots come online with dynamic IP's, apache (and therefore mysql, even with quite agressive caching) quickly begins to get overwhelmed (meaning more logs for fail2ban to process) and then snowballs until all 2.5gb of ram on the server is gone (so everything halts), and it requires a hard reset.

The host (burst.net) did some 'analysis'. Heres what they said:
"I am not sure what kind of attack they are running as we can't see much on the guards and even some of the packets we see we can't see the source. The main problem is that we can't see any high packets to be able to set the riles on the guards to intercept those.

We tried it a few times and seems to be the same every time."

This is understandable to me as the host looks like genuine traffic - spoofed useragent, and simply requesting pages over and over from multiple IP's. In fact, this is probably what traffic would look like on a very popular website (obviously with much more powerful equipment)

vborcan: There are some much cheaper, Linux-based alternatives to Cisco Guard that dynamically use iptables. Andrisoft WANGuard for example.

This is the hosts decision to use cisco guard, not ours and i can't change this.

When the attack is in full force and no blocks at all are in place, these programs which block IP's based on too many requests per second don't work at all, because you'd be lucky to see the same IP make a successful request twice in 10s because there's so much traffic.

What i can't work out, is that its actually so hard to fine information about blocking a DDoS attack, even of only ~1000-2000 unique hosts, which isnt exactly a large botnet by any standard. I even swapped the OS to ubuntu in the hope that someone in the large community could help me, but yet no one actually gave me a working solution, only names like 'fail2ban' and 'mod_evasive' get thrown at you. I don't expect a solution on a plate, however i cant even seem to get a guide in the right direction.

I must say, i did consider a career in networking and/or server administration, but this has pretty much put me off entirely because its been so difficult to even get information, yet alone do the required tasks.

Speaking of such programs, fail2ban has a terrible bug where it trys to throw too many rules at iptables at once and they dont go through (i even tried a hacked workaround where you make it wait a random amount of time, but no go). This means if you have too many jails, it fails to create entire chains and then the rules just generate heaps of errors, which just means more log spam, more hard disk activity, and more server lag.

PenultimateHop

637 posts

Ultimate Geek

Trusted

#333752 23-May-2010 21:20

eXDee: The odd thing is when i googled some of the IP's when i noticed a bunch of similar ones from from singapore, there have been posts from pre 2008 about them attacking websites. These are in the 220.255.7.XXX (220.255.0.0/17) range. By that i mean .131-.139, .181-.199, .211-.229. Thats ~50 sequential IP's, which would appear to be dedicated boxes rather than just zombified computers?

Those are SingTel's proxies. Almost all HTTP traffic in Singapore is transparently proxied for compliance with MDA content ban rules. So those IP addresses aren't attacking you (or anyone else) per se - the devices behind them are.

eXDee: Its only saturating the pipe IF the requests get through, because if they do get through they are requesting heaps of images which is what causes the large bandwidth.

Out of curiosity are there any Referrer: headers? Has someone been hotlinking your images into another site somewhere?

eXdee: This is understandable to me as the host looks like genuine traffic - spoofed useragent, and simply requesting pages over and over from multiple IP's. In fact, this is probably what traffic would look like on a very popular website (obviously with much more powerful equipment)

Well, the theory behind the Cisco Guard system is that it does detect what is anomolous behavior and should be able to determine the difference between a legitimate user and a 'fake' one. I've always wondered how well it would handle a low-PPS attack which yours sounds like - I guess I know the answer now.

Some more information about the attack (log examples - from apache, iptables, and if you can - packet captures) would make analysis a little easier.

1 | 2