URGENT Linux help needed, server under attack

Forums › Linux › URGENT Linux help needed, server under attack

1 | 2 | 3

meow
13579 posts

Uber Geek
+1 received by user: 10910

Moderator

ID Verified

Trusted

Lifetime subscriber

#1694709 24-Dec-2016 21:16

Please don't attempt to keep using this server.

If they've had access to remove contents inside /var/log they've had root access meaning they could have done anything. From this point you can't trust the server so recreate it and start fresh. Unless if you were using ZFS (and are able to restore a ZFS snapshot) which is highly unlikely given you're using CentOS there is no way to recover the files.

The only setup I'd trust is one I've done myself. Given your hosting provider set this server up ask them for support. Yes, there will be downtime and yes, it'll be (likely) costly.

Michael Murphy | https://murfy.nz
Referral Links: Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.

ScuL

491 posts

Ultimate Geek
+1 received by user: 118

Trusted

#1694717 24-Dec-2016 21:27

The problem is that setting up a new server is not an option due to time restriction and immediate requirements.

We have 300,000+ user accounts, and currently up to tens of thousands unique visits daily.
We've just released a new game update for Christmas and there are/will be thousands of keen punters looking to get the latest update.

Combine that with our speedy pay-to-download service the loss of income is considerable in what is normally the busiest time of the year.

I have precisely 48 hours to get something working again or I won't be able to work on this for another month due to the holidays which means many angry gamers around the world, a disappointed community and huge loss of income for our team. Even if I could do it in 48 hours it wouldn't be enough to set up all the features of the site from scratch. :(

Really bummed out on this one

Haere taka mua, taka muri; kaua e wha.

MadEngineer

4591 posts

Uber Geek
+1 received by user: 2570

Trusted

#1694723 24-Dec-2016 21:48

ScuL:

ratsun81:

ssh should never be open to public it is asking for trouble. If you are going to have ssh open it should ONLY be to your IP address or to your main sysadmins IP.

It's on a non-standard port, and I access it frequently from different locations. If I were to lock it to my home address I wouldn't be able to access my server when travelling

the world should not have access to anything other than 443/80. only one IP address should have the ability to connect out of this range. if you need roaming or shared access then employ a vpn from that one trusted IP. It's also trivial to employ sftp, sandpitted at that.

You're not on Atlantis anymore, Duncan Idaho.

networkn

Networkn
32862 posts

Uber Geek
+1 received by user: 15453

ID Verified

Trusted

Lifetime subscriber

#1694746 24-Dec-2016 22:59

I can't offer much in the way of specifics on Linux as it's not my area of expertise, but the advise already given is pretty spot on from what I can see.

The concern I have with a quick fix, is you aren't doing yourself or your users any favours with this approach as it will almost certainly mean more downtime later, and potentially explaining to them that you ignored best practices and didn't wipe the server and now the x consequences are affecting them.

I'm with Michael, as painful as it would be, you simply can't trust this server now.

hio77

'That VDSL Cat'
13036 posts

Uber Geek
+1 received by user: 3896

ID Verified

Trusted

Lizard Networks

Subscriber

#1694747 24-Dec-2016 23:01

ScuL:

The problem is that setting up a new server is not an option due to time restriction and immediate requirements.

We have 300,000+ user accounts, and currently up to tens of thousands unique visits daily.
We've just released a new game update for Christmas and there are/will be thousands of keen punters looking to get the latest update.

Combine that with our speedy pay-to-download service the loss of income is considerable in what is normally the busiest time of the year.

I have precisely 48 hours to get something working again or I won't be able to work on this for another month due to the holidays which means many angry gamers around the world, a disappointed community and huge loss of income for our team. Even if I could do it in 48 hours it wouldn't be enough to set up all the features of the site from scratch. :(

Really bummed out on this one

to be honest, with those numbers... You need to redo your whole system design.

You have put all your eggs in the one basket, and as a result one simple attack has caused a hell of a lot of havoc.

#include <std_disclaimer>

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.

michaelmurfy

meow
13579 posts

Uber Geek
+1 received by user: 10910

Moderator

ID Verified

Trusted

Lifetime subscriber

#1694778 25-Dec-2016 01:57

I've been helping @ScuL tonight to migrate to a new server I'm setting up.

The server was compromised via dirty cow - dodgy .php script by the looks. They disabled all logging on the server (including bash history) and hidden all evidence of what has been done. In ScuL's defense he was following best practices with most things but had way too much sitting on one server and whilst it was patched for most things (including being SELinux enforced) it wasn't patched against Dirty Cow with its impressive uptime of 326 days. I've been going through dumping everything off and migrating them to a new Ubuntu VM sitting on a VM host I have space on for now.

Without going into too many details this site handles quite a few thousand hits per day (seeing on average ~200/sec hits to the new server).

Quite a big job to do but I can't have somebody lose a server on Christmas day to some script kiddie especially since they just did a release of what they're working on. It sucks as it is losing a production server especially hosting a busy site but the timing on this really blows too. Christmas eve, after a release (which seems many people are excited about) and dealing with many GB's of data, multiple MySQL databases and a forum with other peoples hard work.

Anyway with most of the hard work out of the way I am sure ScuL can sleep easier now and enjoy Christmas.

Geekzone support

Support Geekzone with one-off or recurring donations Donate via PressPatron.

RunningMan

9185 posts

Uber Geek
+1 received by user: 4838

#1694815 25-Dec-2016 08:06

michaelmurfy:

I've been helping @ScuL tonight to migrate to a new server I'm setting up.

@michaelmurfy Good on ya for helping out @ScuL, especially at this time! High Fives to you.

ScuL

491 posts

Ultimate Geek
+1 received by user: 118

Trusted

#1694850 25-Dec-2016 09:55

Michael thanks so much for helping me out, this is true Christmas spirit. We both stayed up until the early hours of the morning on Christmas Day to arrange backups and a temporary VM hosted by Michael.

I've also just set up an Amazon EC2 instance to move some of the smaller scripts too so I can get them going independently from the community server.

I have two family functions today (which means more downtime) but am hoping to have restored 80% of the functionality by this evening.

Then after making a dump of the server (which I hope will contain some more evidence of what the culprits have done) I will fully wipe the server and reinstall it ..

Like Michael said I felt the server was pretty reasonably patched up but it's easy to make a slip up.

Haere taka mua, taka muri; kaua e wha.

timmmay

20858 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#1694864 25-Dec-2016 11:29

I'm pretty good with AWS, certified etc, I can help with securing it, cost optimisation, backups, etc. Probably not today though.

michaelmurfy

meow
13579 posts

Uber Geek
+1 received by user: 10910

Moderator

ID Verified

Trusted

Lifetime subscriber

#1694975 26-Dec-2016 00:13

Managed to get most of the sites up and running today nicely (very nicely in-fact) complete with HTTPS (via letsencrypt) and HTTP2 globally. Blame the fact the house was full of girls who liked the scorching summer heat whilst I hid away in my room with the fan running and curtains closed trying to not pass out from heat stroke.

Hopefully (fingers crossed) we don't see a reinfect. At-least this time it is sandboxed... Currently seeing around 300 hits/s with room to grow.

bigalow

568 posts

Ultimate Geek
+1 received by user: 112

#1694979 26-Dec-2016 00:42

michaelmurfy:

I've been helping @ScuL tonight to migrate to a new server I'm setting up.

The server was compromised via dirty cow - dodgy .php script by the looks. They disabled all logging on the server (including bash history) and hidden all evidence of what has been done. In ScuL's defense he was following best practices with most things but had way too much sitting on one server and whilst it was patched for most things (including being SELinux enforced) it wasn't patched against Dirty Cow with its impressive uptime of 326 days. I've been going through dumping everything off and migrating them to a new Ubuntu VM sitting on a VM host I have space on for now.

Without going into too many details this site handles quite a few thousand hits per day (seeing on average ~200/sec hits to the new server).

Quite a big job to do but I can't have somebody lose a server on Christmas day to some script kiddie especially since they just did a release of what they're working on. It sucks as it is losing a production server especially hosting a busy site but the timing on this really blows too. Christmas eve, after a release (which seems many people are excited about) and dealing with many GB's of data, multiple MySQL databases and a forum with other peoples hard work.

Anyway with most of the hard work out of the way I am sure ScuL can sleep easier now and enjoy Christmas.

good to hear

i wonder how many servers out there are not been patched from dirtycow since it been around since 2007

here a tutorial if any one want to see of there servers are patched or not

https://www.digitalocean.com/community/tutorials/how-to-protect-your-server-against-the-dirty-cow-linux-vulnerability

Quic Broadband

Move to New Zealand's best fibre broadband service (affiliate link). Free setup code: R587125ERQ6VE. Note that to use Quic Broadband you must be comfortable with configuring your own router.

ScuL

491 posts

Ultimate Geek
+1 received by user: 118

Trusted

#1695068 26-Dec-2016 11:44

They're really out to get us, they're currently running a DDoS attack to get Michael's new server offline.

He's increased the bandwidth and I'm currently setting up Cloudflare

Haere taka mua, taka muri; kaua e wha.

timmmay

20858 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#1695071 26-Dec-2016 11:52

Once you set up CloudFlare you'll need to change IPs. Probably best to go the business plan, once you pay anything at all their willingness to absorb a DDOS and provide service rises.

timmmay

20858 posts

Uber Geek
+1 received by user: 5350

Trusted

Lifetime subscriber

#1695091 26-Dec-2016 12:20

Also for CloudFlare, make sure your IP isn't in any DNS record that leaves CloudFlare - including MX records, subdomains, etc. That gives them another way in. If you're hosting in AWS set up both network ACLs and security groups that allow traffic only from CloudFlare (IPs here) and your home/work IPs, not everywhere.

hio77

'That VDSL Cat'
13036 posts

Uber Geek
+1 received by user: 3896

ID Verified

Trusted

Lizard Networks

Subscriber

#1695144 26-Dec-2016 14:03

as @timmmay has mentioned, cloudflare is not the end all for ddos attacks.

Free accounts will get temporarily disabled if an attack is deemed too large, Speaking from experience with a previous community.. If they wish to hit you offline and you actually do a good job at keeping your IP hidden, they will just storm the gates till cloudflare gives up.

Do be aware if your trying to keep your IP safe, things like email headers are easy places to find your obscured IP, If that fails a remote image upload feature is also very easily abused (where you give the weblink to the image rather than uploading it and the website downloads the image itself)

Lastly, attacks are normal for communities, It is a sad truth.. take head of this attack as a warning to stay ontop of your security.

#include <std_disclaimer>

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.

1 | 2 | 3