URGENT Linux help needed, server under attack

Forums › Linux › URGENT Linux help needed, server under attack

ScuL

455 posts

Ultimate Geek

Trusted

# 207432 24-Dec-2016 15:14

Hi guys,

To my utter frustration there are some people who seem to be in a mood to destroy my Christmas spirit by hacking our gaming server.

I am a "moderate" level Linux user but the things they seem to have messed up are above my level of expertise.

Any avid Linux guru who is willing to have a look please drop a line..

Cheers

Gigabit

1 | 2 | 3

145 posts

Master Geek

# 1694600 24-Dec-2016 15:43

One person supports this post

Take your server offline, then get your ISP to change your IP address. Once thats done, i'd follow these instructions

Close all unused ports, use shields up! https://www.grc.com/shieldsup to check and see what other people can see from your connection.
Some routers have a DDOS prevention/mitigation option, turn that on obviously.
Turn off "respond to WAN ping"

A DDoS attack attempts to push you off the internet by literally flooding you with data. A more powerful system can cope with the load better.

if you do come under a serious DDoS attack, start logging the requests coming in. If it goes on for long enough you can get your ISP involved and have them ban certain IP ranges.

TL;DR Batten down the hatches and keep your helmet on, you'll be fine.

PSlover14

145 posts

Master Geek

# 1694601 24-Dec-2016 15:45

If its not a DDoS, try and take the server offline anyway and attempt to restore any settings that have been messed with to original settings, Cloud hosting services usually have some sort of backup thing if your server is in the cloud on a VM

If its on a physical computer in your house, disconnect network access temporarily and attempt to repair any issues. Messed up game files and usually be repaired easily through the installer for the dedicated server eg steamcmd for steam dedicated servers like Garry's Mod

ScuL

455 posts

Ultimate Geek

Trusted

# 1694612 24-Dec-2016 16:37

The problem with taking it offline is that you can't restore anything.

It seems to be a targeted attack. I run a very large community hosting a game modification, changing the IP would not make a difference because I would have to re-point the main domain and therefore I would be making the new IP public again. This server is also a webserver and not just a gaming server, it also hosts a 30GB subversion repository.

I've logged into SSH and have disabled all ports except for the SSH port.

Judging by what has happened they haven't been able to obtain shell access, however they seem to have removed all the contents of /var/log

No DDoS attack has been issued

Some webpages have been defaced and they could be easily restored. The index.php files were renamed to index.pwned

Gigabit

timmmay

15235 posts

Uber Geek

Trusted

Subscriber

# 1694618 24-Dec-2016 16:51

Are you running any kind of a firewall? Hardware, software, or service? If not that'd be a good start.

kingjj

Baby Get Shaky!
1648 posts

Uber Geek

Trusted

Subscriber

# 1694620 24-Dec-2016 16:53

What public facing scripts are you running on port 80? Anything like Wordpress that may be out of date and could have been used for a php breach or sql injection?

Any backup that you can restore of the webserver content and than do a full update on all installed software? Can you install rkhunter and/or cxs and see if they find anything?

If your server has been pawned to the extent of clearing /var/log you may need to start fresh with a backup... If you need some temp space to store a backup or stage let me know and I can give ya a vps for a while. I'm probably at the same level of sys admin experience as you so I probably wouldn't be much help to diagnose and clean up.

ScuL

455 posts

Ultimate Geek

Trusted

# 1694626 24-Dec-2016 17:03

timmmay:

Are you running any kind of a firewall? Hardware, software, or service? If not that'd be a good start.

All ports are blocked except for the public facing ones (80, 21, SSH, and a secure HTTPS channel for Plesk/PSA)

kingjj:

What public facing scripts are you running on port 80? Anything like Wordpress that may be out of date and could have been used for a php breach or sql injection?

Any backup that you can restore of the webserver content and than do a full update on all installed software? Can you install rkhunter and/or cxs and see if they find anything?

If your server has been pawned to the extent of clearing /var/log you may need to start fresh with a backup... If you need some temp space to store a backup or stage let me know and I can give ya a vps for a while. I'm probably at the same level of sys admin experience as you so I probably wouldn't be much help to diagnose and clean up.

Apache, PHP, MySQL.

Wordpress is installed on 2 sites however it's on auto-update via Plesk.

All other server software is also set to auto-update via YUM on a daily basis.

I have daily remote backups set up to my homeserver which backups all www/vhosts contents as well as all SQL databases, but the SVN repository is not backed up.

If I restore my server to default and configure everything again it will take me at least a month to get things back to the original configuration and I do not have that time because I'm about to leave on a holiday in a few days

I suspect the culprit managed to run shell commands through a backdoor in Apache and/or PHP and has been able to run "rm -rf /var/log" like that.

I've done a lot of research in the shell now (recently modified files etc) and it doesn't look like anything has changed other than the contents of some vhosts folders.

Gigabit

hio77

'That VDSL Cat'
11033 posts

Uber Geek

Trusted

Spark

Subscriber

# 1694637 24-Dec-2016 17:48

Sounds indeed like a backdoor has been used.

Willing to bet one of your PHP scripts will be the offending targetpoint.

Hearing that your var/log has been fully wiped is concerning, are you running PHP/Apache as root?

As someone who used to run large communities, be glad you have not been attacked with a ddos, That can be far worse due to the affect it will have on your datacenter and such..

word of advise for if that issue should crop up for you though, putting all your eggs in one basket is often dangerous, though the cheapest option.

#include <std_disclaimer>

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.

ratsun81

273 posts

Ultimate Geek

# 1694645 24-Dec-2016 18:29

2 people support this post

ScuL:

All ports are blocked except for the public facing ones (80, 21, SSH, and a secure HTTPS channel for Plesk/PSA)

ssh should never be open to public it is asking for trouble. If you are going to have ssh open it should ONLY be to your IP address or to your main sysadmins IP.

biggal

386 posts

Ultimate Geek

# 1694646 24-Dec-2016 18:30

One person supports this post

change login pass

install Fail2Ban

install this script

https://github.com/trick77/ipset-blacklist

if your using wordpress use zbblock

http://www.spambotsecurity.com/zbblock.php

block all ports and accept port 80 443 ssh turn off ftp and only turn it on when neaded

and change ssh port from port 22 to something else like port 2345 etc

hio77

'That VDSL Cat'
11033 posts

Uber Geek

Trusted

Spark

Subscriber

# 1694649 24-Dec-2016 18:37

One person supports this post

biggal:

and change ssh port from port 22 to something else like port 2345 etc

in all honesty, if someone wishes to attack your ssh no port change is going to help at all. A simple portscan fixes that.

#include <std_disclaimer>

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.

michaelmurfy

Mr Snotty
8876 posts

Uber Geek

Moderator

Trusted

Lifetime subscriber

# 1694660 24-Dec-2016 19:36

5 people support this post

Right - if /var/log is wiped then do not trust this server. It may have well have a rootkit on it. I'm afraid you need to wipe / restore this server.

1) If Plesk is used follow the Plesk security best practices Here. Ensure SSH is firewalled off to a trusted host only. I'd recommend either using Debian or CentOS (with SELinux enabled and configured).
2) For web apps that run outside of the web server configure nginx (included with Plesk) to proxy the web app instead of allowing direct access.
3) Ensure that updates are run - this could well have been an attack on an insecure website coupled with the Dirty Cow exploit allowing full root access. Debian can install security updates automatically with cron-apt however you'll need to maintain kernel upgrades via a reboot.
4) If the sites are yours then consider if you really need Plesk.
5) Put SVN/GIT on another server or host it outside of local management on something like BitBucket - always consider a server with a control panel (like Plesk) insecure as these are frequently targeted.

Also ensure you check each site before restoring as you may be restoring a compromised site and get set back to step 1. Do not take shortcuts when it comes with security and ensure all non-essential ports are firewalled off.

There are some excellent tips for securing CentOS Here otherwise with Debian / Ubuntu there are many guides out there too. With Plesk I would recommend CentOS + SELinux set to enforcing in the future.

rphenix

894 posts

Ultimate Geek

Subscriber

# 1694671 24-Dec-2016 20:05

One person supports this post

Like Michael said - if they wiped /var/log/ completely - not just apache logs then they have had some form of root access. You cant trust your shell anymore. If you use any passwords on that server elsewhere - immediately change them.

The provider of your server may have IT that for a fee can assist you.

If it were me - I would start setting up a new server and migrate services off this system to the new one. Wordpress belongs on a seperate system even if its just a small virtual on linode etc.. that way its low value and easy to setup again.

Personally I wouldnt use Plesk on the new system I would try and work without it or anythign similar (cpanel etc..).

Ideally long term:

Use Ansible or any other configuration management system so you can recreate your server config without any guess work - put that under source control so if a server is owned you simply run a command to setup a new server - the configuration isnt special or hard to replicate. Your repositories need to be somewhere else - be it another system or bitbucket, github.

Snapshot the current server so later when you have more time you can identify the scope of what they did.

timmmay

15235 posts

Uber Geek

Trusted

Subscriber

# 1694672 24-Dec-2016 20:07

Take a backup before you roll back to a backup.

Immutable computing (Ansible etc) is super useful, occasionally, or for systems that change regularly. On AWS I would use CloudFormation with static resources stored or backed up on S3.

ScuL

455 posts

Ultimate Geek

Trusted

# 1694700 24-Dec-2016 21:00

hio77:

Sounds indeed like a backdoor has been used.

Willing to bet one of your PHP scripts will be the offending targetpoint.

Hearing that your var/log has been fully wiped is concerning, are you running PHP/Apache as root?

I think so too. No, PHP/Apache are definitely not running as root. They are under a different user.

As someone who used to run large communities, be glad you have not been attacked with a ddos, That can be far worse due to the affect it will have on your datacenter and such..

word of advise for if that issue should crop up for you though, putting all your eggs in one basket is often dangerous, though the cheapest option.

I have been DDoS-ed several times before. I've been running large phpBB sites since 2004 and have regularly been a victim of such attacks. Our host (Leaseweb) is running anti-DDoS mechanisms so it typically isn't that much of a problem.

ratsun81:

ssh should never be open to public it is asking for trouble. If you are going to have ssh open it should ONLY be to your IP address or to your main sysadmins IP.

It's on a non-standard port, and I access it frequently from different locations. If I were to lock it to my home address I wouldn't be able to access my server when travelling

biggal:

change login pass

install Fail2Ban

install this script

https://github.com/trick77/ipset-blacklist

if your using wordpress use zbblock

http://www.spambotsecurity.com/zbblock.php

block all ports and accept port 80 443 ssh turn off ftp and only turn it on when neaded

and change ssh port from port 22 to something else like port 2345 etc

Thanks for your tips, Fail2Ban has been installed and is active and up-to-date. SSH is already on a non-standard port.

michaelmurfy:

Right - if /var/log is wiped then do not trust this server. It may have well have a rootkit on it. I'm afraid you need to wipe / restore this server.

1) If Plesk is used follow the Plesk security best practices Here. Ensure SSH is firewalled off to a trusted host only. I'd recommend either using Debian or CentOS (with SELinux enabled and configured).
2) For web apps that run outside of the web server configure nginx (included with Plesk) to proxy the web app instead of allowing direct access.
3) Ensure that updates are run - this could well have been an attack on an insecure website coupled with the Dirty Cow exploit allowing full root access. Debian can install security updates automatically with cron-apt however you'll need to maintain kernel upgrades via a reboot.
4) If the sites are yours then consider if you really need Plesk.
5) Put SVN/GIT on another server or host it outside of local management on something like BitBucket - always consider a server with a control panel (like Plesk) insecure as these are frequently targeted.

Also ensure you check each site before restoring as you may be restoring a compromised site and get set back to step 1. Do not take shortcuts when it comes with security and ensure all non-essential ports are firewalled off.

There are some excellent tips for securing CentOS Here otherwise with Debian / Ubuntu there are many guides out there too. With Plesk I would recommend CentOS + SELinux set to enforcing in the future.

My server has been installed with a CentOS distribution by our hosting provider that was totally locked down to begin with. It contains CentOS 6.8 (64), kernel 2.6.32 and SELinux is enforced.
Plesk licenses are included and are installed by default, I have been running on this platform for close to 6 years.

I will attempt to follow the rest of your guidelines, thanks.

rphenix:

Like Michael said - if they wiped /var/log/ completely - not just apache logs then they have had some form of root access. You cant trust your shell anymore. If you use any passwords on that server elsewhere - immediately change them.

Apache logs are still in tact, just /var/log that has been targeted.

The provider of your server may have IT that for a fee can assist you.

They do but it will be a nightmare to reinstall everything, and very costly.

If it were me - I would start setting up a new server and migrate services off this system to the new one. Wordpress belongs on a seperate system even if its just a small virtual on linode etc.. that way its low value and easy to setup again.

Right, I will consider using different sites for the Wordpress instances, they are indeed not relevant for the main community website.

Ideally long term:

Use Ansible or any other configuration management system so you can recreate your server config without any guess work - put that under source control so if a server is owned you simply run a command to setup a new server - the configuration isnt special or hard to replicate. Your repositories need to be somewhere else - be it another system or bitbucket, github.

Snapshot the current server so later when you have more time you can identify the scope of what they did.

I have never heard of this platform. Can you suggest what I can use to snapshot the current server?

Gigabit

ScuL

455 posts

Ultimate Geek

Trusted

# 1694701 24-Dec-2016 21:03

Oh by the way, I read up on "undeleting" files on Linux.

Would it be possible to restore the contents of the /var/log folders?

I just looked into the bash_history files of all accounts, they weren't removed, modified or showed any suspicious activity. So it looks like he was able to execute some kind of shell commands from whatever backdoor that was used.

Gigabit

1 | 2 | 3