Web server outages and DDOS attacks

Forums › IT Pro and developers › Web server outages and DDOS attacks - how common are they?

1 | 2 | 3 | 4

20581 posts

Uber Geek

Trusted

Lifetime subscriber

#1462352 6-Jan-2016 14:38

Some suggestions for web hosts, based on practical experience. There are plenty of web hosting review places online too.

RobinmNZ

24 posts

Geek

#1462355 6-Jan-2016 14:40

marpada: You didn't mention the nature of the attack, but one common problem with big sites is bots aggressively crawling the content, both bona-fide or rogue bots (which don't honour robots.txt). In all cases I was able to parry them by blocking dodgy User-Agents or through webserver modules to limit the request rate (http://nginx.org/en/docs/http/ngx_http_limit_req_module.html). If using Apache you want to check whether is has been tuned properly according to the server specs.

Using Cloudfare is usually also an easy win.

Thanks, Marpada, yes, I believe bots crawling has also been a problem ... we seem to attract all that is bad on the web and not the good!!

timmmay

20581 posts

Uber Geek

Trusted

Lifetime subscriber

#1462371 6-Jan-2016 14:54

Rate limiting is pretty easy on nginx, probably easy enough on Apache. You could also try updating your robots.txt, if they honor it they may stop indexing altogether.

jnimmo

1097 posts

Uber Geek

#1462373 6-Jan-2016 14:56

It could also be to do with the stack you're running, as to how well it will cope with DoS or surges in traffic.
For example, running Wordpress behind Nginx or Varnish will make a huge difference. (i.e. majority of pages served from cache, if PHP stops running under attack or something pages are served from cache).

I'd be surprised if a dedicated server was required, you could get better reliability of managed hosting or high quality VPS. It sounds like the more management you can outsource the better..

A couple of suggestions:

1. The folks at rtCamp are amazing at configuring high performance sites. I highly recommend looking at their Managed hosting or if you were hoping to reduce costs further get them to configure a Virtual private server for you with a different provider. They could do the migration for you. Their EasyEngine product is fantastic and my VPSs run like a dream (with tiny resources)
https://easyengine.io/services/managed-hosting/

2. LightningBase provide specialist Wordpress hosting, and they can automatically push out security updates to Wordpress etc as required. They handle Varnish caching for you, see below quote for details.

"Our CDN caches all the static files other than the html page itself. What this means is that each time someone visits your site, all the server has to do is serve up a single small file, while the CDN handles everything else. On top of all this sits Varnish, speeding up the delivery of these pages with in-memory caching. With a relatively simple page, even our smallest account can handle over 1,000 pageviews per second, which is an unheard of number (we’re talking over 86 million visitors in a single day at that rate). If your site is really large and complex, this might be lowered somewhat, but will still easily be hundreds per second."

I believe Chris can handle the migrations there too if you asked. They have servers in Chicago, Melbourne or Netherlands depending on where your target audience is, with a global CDN to speed things up.

3. If your audience is NZ based, Sitehost provide some fantastic hosting. Grab a VPS (or other) from them and get the rtCamp guys to setup EasyEngine stack (Nginx, fail2ban, etc) on it for you and do the migration. 250k visitors per year is easy as long as you have the right configuration.

MadEngineer

4291 posts

Uber Geek

Trusted

#1462400 6-Jan-2016 15:13

I've had great experience with hosting by inspire.net.nz - always quick to help and always knowledgeable.

Don't know if their plans or location suit your visitors.

You're not on Atlantis anymore, Duncan Idaho.

mattwnz

20164 posts

Uber Geek

#1462432 6-Jan-2016 15:23

timmmay:
RobinmNZ:
timmmay: Amazon m4.xlarge is US$1200 per year on a one year term, double that to double the specs. It's not dedicated but it's 4 core 16GB RAM. No support as such, if it breaks you fix it. May be some extra charges like traffic, backups, etc.

Thanks - the pricing seems OK compared with what I have now, but does that mean if there is a hacker or DDOS attack I have to fix it? I am afraid that would be beyond me ...

It means there's no-one to call and say "hey my website is down" or "my site has been hacked". You have full control, you fix it yourself. However Amazon will deal with DDOS type stuff.

If you need that you need to use a provider for support. You'd just need a better provider than the one you have now.

You highlight a very important fact. It is the support that actually costs the money, not necessarily the hosting service itself. This is why the cheap web hosts often have terrible support, and it often gets worse overtime especially if they get taken over by another host.

With Amazon, what about hardware problems, which you obviously can't fix yourself? Is there no support whatsoever? Or do they charge for a ticket. I have seen some companies now charging for support tickets.

marpada

476 posts

Ultimate Geek

#1462440 6-Jan-2016 15:37

With Amazon, what about hardware problems, which you obviously can't fix yourself? Is there no support whatsoever? Or do they charge for a ticket. I have seen some companies now charging for support tickets.

By default AWS accounts come with no support (you can only add tickets to raise some default limits, like the number of Elastic IP Addresses). If you need help from AWS you can try creating a forum threads and pray for some AWS engineers to pick up. There's a Developer support plan for a fixed monthly fee (around $50?) and the Business plan that costs 10% of your monthly bill. Even if you spend thousand of dollars support is just ok and scoped to only AWS infrastructure or services. They're never going to ssh to your instances for troubleshooting.

Quic Broadband

Move to New Zealand's best fibre broadband service (affiliate link). Free setup code: R587125ERQ6VE. Note that to use Quic Broadband you must be comfortable with configuring your own router.

kingjj

1728 posts

Uber Geek

ID Verified

Trusted

#1462478 6-Jan-2016 16:29

RobinmNZ:
mattwnz: Get a new provider. It is not normal. Even on shared hosting it isn't normal, unless you are using cheap ones.

I don't think what we have is a cheap one: here are the specs - that is the per month price, it was $260 but there was a reducing 2 months ago.

Intel Xeon 5430 Single Processor, Quad Core - (01/26/2016 - 02/25/2016) $231.00 USD
cPanel/WHM: Yes
Additional drive: Yes
Additional RAM: No
Additional RAM: Yes
Backup service (requires backup drive above): Yes
Enhanced monitoring package: Yes
Fantastico: No
------------------------------------------------------
Sub Total: $231.00 USD

$231 USD a month is pretty damned high for those specs IMHO. Does this come with DDoS protection? Cpanel itself is usually around $35 USD for a month. Assuming this comes with a management contract? Apache is great but not so great when it comes to large swarms of traffic. If you have IT people ask them to look into something like Nginx (free open-source) or LiteSpeed (commercial) or anyone of the many event driven servers. The advantage of Ngnix is that you could run it along side Cpanel easily enough (there are several plugins) and use it soley as a reverse proxy or as your HTTP server.

Can you quote us some of the comments that the IT fellas say are the causes? There are 101 things that could cause issues and I'd hate for you to move your setup only to find it was x-plugin in WP or a nightly server backup run that was locking up the IO.

Edit: oh and don't jump into the AWS deep end if you/your IT aren't completely up with it. If you use their pay-as-you-go model you can quickly rack up the bills.

jnimmo

1097 posts

Uber Geek

#1462494 6-Jan-2016 16:57

Just as an example (seeing it won't let me edit my post above), one WP site I setup using the EasyEngine stack, (Nginx, W3TC) has served up 103K pageviews (Google Analytics) over the last year without any issues on the Sitehost $30/month VPS plan. 1 core, 1.5GB RAM. If I was expecting a higher volume of traffic I'd look to upgrade to the next plan up for an extra core and add CDN

RobinmNZ

24 posts

Geek

#1462499 6-Jan-2016 17:05

kingjj:
Can you quote us some of the comments that the IT fellas say are the causes? There are 101 things that could cause issues and I'd hate for you to move your setup only to find it was x-plugin in WP or a nightly server backup run that was locking up the IO.

Edit: oh and don't jump into the AWS deep end if you/your IT aren't completely up with it. If you use their pay-as-you-go model you can quickly rack up the bills.

Thanks KingJJ ... appreciate your comments.

Here are some of the responses I've had - grab a coffee, it's a long read: (I'd have thought that with all the blocking going on over the past year or so that things would be tight as a drum, but obviously not). I've removed any site names and put eeee to protect the sites' privacy.

they were pounding away at the xmlrpc file under your site, and claiming to be a google bot while doing it.

mySQL is running, but it looks like your site has some kind of strange lock hang going on. The process list output is filled with stuff like this:

| 5614779 | eeee_wp49 | localhost | horse_wp49 | Query | 1017 | Locked | ALTER TABLE wp_popularpostssummary ENGINE=INNODB |
| 5614780 | eeee_wp49 | localhost | horse_wp49 | Query | 982 | Locked | ALTER TABLE wp_popularpostssummary ENGINE=INNODB |

It just chimed in with a mySQL error (too many connections) and a load alert (which is caused by all those mySQL connections). Looking at the connections, there doesn't seem to be any sort of attack going on. There is a lot of bot/spider activity from yahoo and msn.

We killed off some IPs making a bunch of bogus requests by tossing them into the firewall awhile ago. It's been fine since then.

/usr/local/apache/domlogs/eeee.co.nz:209.43.112.2 - - [21/Oct/2015:14:25:47 +1300] "GET /8.php?450699=1&php4=1&root=1&upl=1&wphp4=1&abdullkarem=1&wp=1&module=1&php=1&php5=1&wphp5=1 HTTP/1.0" 404 16775 "-" "-"
/usr/local/apache/domlogs/eeee.co.nz:209.43.112.2 - - [21/Oct/2015:14:25:32 +1300] "GET /wso.php?450699=1&php4=1&root=1&upl=1&wphp4=1&abdullkarem=1&wp=1&module=1&php=1&php5=1&wphp5=1 HTTP/1.0" 404 16775 "-" "-"
/usr/local/apache/domlogs/eeee.co.nz:209.43.112.2 - - [21/Oct/2015:14:26:20 +1300] "GET /sql.php?450699=1&php4=1&root=1&upl=1&wphp4=1&abdullkarem=1&wp=1&module=1&php=1&php5=1&wphp5=1 HTTP/1.0" 404 16775 "-" "-"

I pulled a report of connecting IPs, and saw a few that looked like scrapers, so I booted those into the firewall and restarted the firewall and the web server service to get rid of them. The load is still within reasonable range and the mysql process output list is not erroring in any way.

The load is a bit high, but the server is dealing with it without any intervention. [after another outage where the site/server was down]

Already handled. There's nothing being targeted. You simply have visitors running up a ton of connections for whatever it is they're doing, which wordpress is terrible about if there's no caching plugin, and if they're going to old posts trying to do any commenting, it just makes it worse:

21 68.180.230.31
47 204.237.81.85
59 108.92.170.62
62 70.198.192.241

That last one according to the logs was trying to POST something, probably a spammy comment. You should consider closing comments on posts older than x days, to cut down not just on the number of junk things to wade through, but to stop wordpress from generating a bunch of mySQL queries related to that activity.

RobinmNZ: {I stopped comments on articles older than 2 weeks after this}

We've got stuff hitting three different servers. In your case, it's a bunch of stuff from amazon's web service itself, pulling the feed for some reason, but in groups of like 15 at a time. It makes no sense at all.
We're going to try some mysql tweaks to try and cache some of the more used queries the site kicks off. It takes more memory, but it uses less CPU and doesn't take up a mysql slot unnecessarily.
Right now I'm trying to get rid of these amazon aws IPs, because they're the ones creating the problem right now.

These things are not pulls from your site to the outside, they are direct requests from whatever is running on those IPs pulling from the site, and not for any good reason that I can see.
The server does have a resource limiter on it that can be enabled, but what that does it limit or throttle access to your site - it's as unavailable as it is when the number of mysql processes run away with things, but the limit/throttle shuts off access to the site until the request rate goes back to normal, leaving all the other sites going. There is also the option of splitting the database functions to another server, to offload that processing to entirely different hardware, or upgrading to an even heavier server, spec-wise. Those options would of course involve additional expenditures.
I've got the amazon aws IPs tossed in the firewall, and we're looking over the mysql config to see what can be tweaked.

we blocked off the entirety of SAS France (a hosting provider) earlier. Various IPs of their allocation were snooping around practically all over the network throughout the weekend, but with anywhere from 1-10 connections at a time, instead of a hundred or more, which would look more suspicious. This morning we found their IPs crawling all over the box, so I tossed them in firewall jail on horse, too, and even went ahead and added them to the master/global blocking list for the network here, because all we've seen from that pit is abuse. We aren't missing anything from the blocking, as far as I can tell.

Just more bots and nonsense. We knocked another entire hosting datacenter into the firewall from your-server.de, in Germany.

Bots attack anything. The only real issue is that most of them come in the overnight server time, versus overnight in our time, and that's the same time that everything else is running, too, like backups, cpanel updates, legit search engine crawlers (yahoo consistently shows up as the top one with the most requests, in fact, whenever we have to get in and calm things down), and maintenance things. It isn't any sort of coordinated attack. It's nuisance stuff that when combined with the usual overnight stuff is creating issues that don't normally come up during the day because by that time all the other things are finished. If a larger or second server is not really feasible, it might be helpful to look into a CDN, like amazon or some others, to see about offloading things like images and stuff, because that would serve up the content at a node closest to wherever the requestor happens to be, lessening the load on the server from having to send out responses to what are essentially bogus requests. Short of that, some kind of caching plugin that doesn't create a billion requests of its own would help.

RobinmNZ: I have tried a caching plugin in the past but it paralysed the entire site.

There isn't much that can be done with things in this current state without taking some kind of proactive measure to deal with the ever-increasing traffic to the site, whether that's getting a more powerful server, splitting mySQL processing off to another server to separate that processing, implementing a CDN of some sort, and so on. Or implement some kind of caching plugin that in and of itself is not going to bring the server to its knees. Otherwise, all we can do is continually block large ranges of IPs, when they show up in groups like this from the same CIDR ranges:
Most of those are from Romania, and some are from some hosting outfit in the UK. We dumped all three larger IP allocations into the firewall to deny them and restarted the services and everything is back to the usual.

RobinmNZ: In early Dec the server was upgraded to mySQL 5.6 and PHP 5.6

RobinmNZ: "Another 3-4am outage - WP tells me it was an hour neat."
Response: I don't have any notices at all from the monitoring server last night, and the backups on the server completed around 11PM EST.

RobinmNZ: 22 Dec monitoring info: Good news — your site is back up!
Your total downtime was 23 mins, but your site was up again as of Wednesday, December 23, 7:02 am.
Your total downtime was 38 mins, but your site was up again as of Wednesday, December 23, 2:22 am.

response: We saw it and took care of it. We bounced off into the firewall some company called Vultr, which apparently leases servers from Choopa. We threw an entire /24 into the firewall, actually, in the event they have other IPs allocated to them from Choopa versus just the one IP we saw.

And finally the latest outage on 5 Jan, which was the second of the day:
response: Yes, we dealt with it. We tossed Bulgaria into the firewall, and Romania is going to follow when we find an accurate list of their IPs.

RobinmNZ: "Thanks, but it was out for a whole hour.... It can't surely take that long to restart all services? Then out for another 43 minutes at the usual 4am time."

response: The backups run in the middle of the night. Combine that with bots or whatever, and it's going to be a problem, especially given the size and scope of at least the {name removed} site. Get another swarm, with the usual search engines crawlers, and yes, it's going to be a problem when we're trying to gather data to block things instead of just kicking it for a reboot, which does nothing except interrupt them for a couple of minutes before they start all over again. Sorting out what is legit (yahoo, google) and what is not (Bulgaria, et al) is not an instant process. Toss in the activity for other sites, and it's likely things have simply grown to the point where the existing server does not have enough horsepower (so to speak) to get through these things without bogging down.

marpada

476 posts

Ultimate Geek

#1462516 6-Jan-2016 17:40

The User Agent is empty on those requests. There's no good reason for that, so start by adding a RewriteRule to reject those requests

http://johannburkard.de/blog/www/spam/block-empty-user-agent.html

If they figure out what's going on the bot might change its User Agent to something less conspicous, or might just as well to move along.

michaelmurfy

meow
13254 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#1462571 6-Jan-2016 18:55

I host a site that generates thousands of hits per hour (is a popular YouTuber) and the site is hosted off a Linode 1gb along with several other sites and is Wordpress based.

How I do it is simple - I block off every IP range except Cloudflare on HTTP/HTTPS and force sites to go via Cloudflare to use the server resources. This way I have DDOS protection. I have had it once (before Cloudflare) the server was getting hammered by a brute-force attack causing the whole server to go down (inbound traffic was in excess of 1gbit) and I ended up (as a patch) scripting something to block excess traffic via iptables.

Now sites run really quick - and the server doesn't see much traffic since most traffic just gets served via Cloudflare. This isn't ideal in most cases but having Linode was very handy since they're designed to handle a heap of traffic, their virtual machines are cheap and also if there is a sudden influx of traffic it only takes a few minutes to upgrade the machine to handle it.

Michael Murphy | https://murfy.nz
Referral Links: Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.

jnimmo

1097 posts

Uber Geek

#1462578 6-Jan-2016 19:14

Great - PHP 5.6 has Zend Opcache built in, you should enable this
Another step might be to try object caching with memcache in W3 Total Cache

Are your users logging in or just guests?

https://easyengine.io/tutorials/wordpress/performance-optimization/

freitasm

BDFL - Memuneh
79289 posts

Uber Geek

Administrator

ID Verified

Trusted

Geekzone

Lifetime subscriber

#1462597 6-Jan-2016 19:43

Placing the domain behind Cloudflare is a good idea. The only problem is if the DDoS is also affecting other domains in the same shared IP, which won't be protected, overloading the box anyway.

But in any case, put on Cloudflare.

timmmay

20581 posts

Uber Geek

Trusted

Lifetime subscriber

#1462601 6-Jan-2016 19:52

mattwnz: You highlight a very important fact. It is the support that actually costs the money, not necessarily the hosting service itself. This is why the cheap web hosts often have terrible support, and it often gets worse overtime especially if they get taken over by another host.

With Amazon, what about hardware problems, which you obviously can't fix yourself? Is there no support whatsoever? Or do they charge for a ticket. I have seen some companies now charging for support tickets.

Hardware is virtualised. If something goes wrong you will either be migrated automatically to another host, or you may have to stop and start your virtual server. There's a great range of monitoring tools, and you can architect for failure. Netflix run on AWS, their "chaos monkey" randomly turns things off to make sure it's robust. They have another that I forget the name of, it randomly switches entire data centers off to make sure they're really, really well architected for failure.

That might be a bit more for a small company with one server, but it's possible. Also scaling is also quite easy, so if your server gets busy you can spin up another (or another 500 servers) for as as is needed.

1 | 2 | 3 | 4