Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.




455 posts

Ultimate Geek

Trusted

# 207432 24-Dec-2016 15:14
Send private message

Hi guys,

 

 

 

To my utter frustration there are some people who seem to be in a mood to destroy my Christmas spirit by hacking our gaming server.

 

I am a "moderate" level Linux user but the things they seem to have messed up are above my level of expertise.

 

Any avid Linux guru who is willing to have a look please drop a line..

 

 

 

Cheers

 

 





Gigabit


Filter this topic showing only the reply marked as answer View this topic in a long page with up to 500 replies per page Create new topic
 1 | 2 | 3
145 posts

Master Geek


  # 1694600 24-Dec-2016 15:43
One person supports this post
Send private message

Take your server offline, then get your ISP to change your IP address. Once thats done, i'd follow these instructions

 

Close all unused ports, use shields up! https://www.grc.com/shieldsup to check and see what other people can see from your connection.
Some routers have a DDOS prevention/mitigation option, turn that on obviously.
Turn off "respond to WAN ping"

A DDoS attack attempts to push you off the internet by literally flooding you with data. A more powerful system can cope with the load better.

 

if you do come under a serious DDoS attack, start logging the requests coming in. If it goes on for long enough you can get your ISP involved and have them ban certain IP ranges.

TL;DR Batten down the hatches and keep your helmet on, you'll be fine.


145 posts

Master Geek


  # 1694601 24-Dec-2016 15:45
Send private message

If its not a DDoS, try and take the server offline anyway and attempt to restore any settings that have been messed with to original settings, Cloud hosting services usually have some sort of backup thing if your server is in the cloud on a VM

 

 

 

If its on a physical computer in your house, disconnect network access temporarily and attempt to repair any issues. Messed up game files and usually be repaired easily through the installer for the dedicated server eg steamcmd for steam dedicated servers like Garry's Mod


 
 
 
 




455 posts

Ultimate Geek

Trusted

  # 1694612 24-Dec-2016 16:37
Send private message

The problem with taking it offline is that you can't restore anything.

 

It seems to be a targeted attack. I run a very large community hosting a game modification, changing the IP would not make a difference because I would have to re-point the main domain and therefore I would be making the new IP public again. This server is also a webserver and not just a gaming server, it also hosts a 30GB subversion repository.

 

I've logged into SSH and have disabled all ports except for the SSH port.

 

Judging by what has happened they haven't been able to obtain shell access, however they seem to have removed all the contents of /var/log

 

No DDoS attack has been issued

 

Some webpages have been defaced and they could be easily restored. The index.php files were renamed to index.pwned

 

 





Gigabit


15235 posts

Uber Geek

Trusted
Subscriber

  # 1694618 24-Dec-2016 16:51
Send private message

Are you running any kind of a firewall? Hardware, software, or service? If not that'd be a good start.


Baby Get Shaky!
1648 posts

Uber Geek

Trusted
Subscriber

  # 1694620 24-Dec-2016 16:53
Send private message

What public facing scripts are you running on port 80? Anything like Wordpress that may be out of date and could have been used for a php breach or sql injection? 

 

Any backup that you can restore of the webserver content and than do a full update on all installed software? Can you install rkhunter and/or cxs and see if they find anything?

 

If your server has been pawned to the extent of clearing /var/log you may need to start fresh with a backup... If you need some temp space to store a backup or stage let me know and I can give ya a vps for a while. I'm probably at the same level of sys admin experience as you so I probably wouldn't be much help to diagnose and clean up.




455 posts

Ultimate Geek

Trusted

  # 1694626 24-Dec-2016 17:03
Send private message

timmmay:

 

Are you running any kind of a firewall? Hardware, software, or service? If not that'd be a good start.

 

 

 

 

All ports are blocked except for the public facing ones (80, 21, SSH, and a secure HTTPS channel for Plesk/PSA)

 

kingjj:

 

What public facing scripts are you running on port 80? Anything like Wordpress that may be out of date and could have been used for a php breach or sql injection? 

 

Any backup that you can restore of the webserver content and than do a full update on all installed software? Can you install rkhunter and/or cxs and see if they find anything?

 

If your server has been pawned to the extent of clearing /var/log you may need to start fresh with a backup... If you need some temp space to store a backup or stage let me know and I can give ya a vps for a while. I'm probably at the same level of sys admin experience as you so I probably wouldn't be much help to diagnose and clean up.

 

 

Apache, PHP, MySQL.

 

Wordpress is installed on 2 sites however it's on auto-update via Plesk.

 

All other server software is also set to auto-update via YUM on a daily basis.

 

 

 

I have daily remote backups set up to my homeserver which backups all www/vhosts contents as well as all SQL databases, but the SVN repository is not backed up.

 

If I restore my server to default and configure everything again it will take me at least a month to get things back to the original configuration and I do not have that time because I'm about to leave on a holiday in a few days

 

I suspect the culprit managed to run shell commands through a backdoor in Apache and/or PHP and has been able to run "rm -rf /var/log" like that.

 

 I've done a lot of research in the shell now (recently modified files etc) and it doesn't look like anything has changed other than the contents of some vhosts folders.





Gigabit


'That VDSL Cat'
11033 posts

Uber Geek

Trusted
Spark
Subscriber

  # 1694637 24-Dec-2016 17:48
Send private message

Sounds indeed like a backdoor has been used.

 

 

 

Willing to bet one of your PHP scripts will be the offending targetpoint.

 

Hearing that your var/log has been fully wiped is concerning, are you running PHP/Apache as root?

 

 

 

 

 

As someone who used to run large communities, be glad you have not been attacked with a ddos, That can be far worse due to the affect it will have on your datacenter and such..

 

word of advise for if that issue should crop up for you though, putting all your eggs in one basket is often dangerous, though the cheapest option. 





#include <std_disclaimer>

 

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.


 
 
 
 


273 posts

Ultimate Geek


  # 1694645 24-Dec-2016 18:29
2 people support this post
Send private message

ScuL:

 

 

 

 

 

All ports are blocked except for the public facing ones (80, 21, SSH, and a secure HTTPS channel for Plesk/PSA)

 

 

 

 

 

 

ssh should never be open to public it is asking for trouble. If you are going to have ssh open it should ONLY be to your IP address or to your main sysadmins IP. 

 

 


386 posts

Ultimate Geek


  # 1694646 24-Dec-2016 18:30
One person supports this post
Send private message

change login pass

 

install Fail2Ban 

 

install this script

 

https://github.com/trick77/ipset-blacklist

 

if your using wordpress use zbblock 

 

http://www.spambotsecurity.com/zbblock.php

 

block all ports and accept port 80 443 ssh  turn off ftp and only turn it on when neaded

 

and change ssh port from port 22 to something else like port 2345 etc

 

 

 

 





 

 

 


'That VDSL Cat'
11033 posts

Uber Geek

Trusted
Spark
Subscriber

  # 1694649 24-Dec-2016 18:37
One person supports this post
Send private message

biggal:

 

 

 

and change ssh port from port 22 to something else like port 2345 etc

 

 

 

 

 

in all honesty, if someone wishes to attack your ssh no port change is going to help at all. A simple portscan fixes that.





#include <std_disclaimer>

 

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.


Mr Snotty
8876 posts

Uber Geek

Moderator
Trusted
Lifetime subscriber

  # 1694660 24-Dec-2016 19:36
5 people support this post
Send private message

Right - if /var/log is wiped then do not trust this server. It may have well have a rootkit on it. I'm afraid you need to wipe / restore this server.

 

1) If Plesk is used follow the Plesk security best practices Here. Ensure SSH is firewalled off to a trusted host only. I'd recommend either using Debian or CentOS (with SELinux enabled and configured).
2) For web apps that run outside of the web server configure nginx (included with Plesk) to proxy the web app instead of allowing direct access.
3) Ensure that updates are run - this could well have been an attack on an insecure website coupled with the Dirty Cow exploit allowing full root access. Debian can install security updates automatically with cron-apt however you'll need to maintain kernel upgrades via a reboot.
4) If the sites are yours then consider if you really need Plesk.
5) Put SVN/GIT on another server or host it outside of local management on something like BitBucket - always consider a server with a control panel (like Plesk) insecure as these are frequently targeted.

 

Also ensure you check each site before restoring as you may be restoring a compromised site and get set back to step 1. Do not take shortcuts when it comes with security and ensure all non-essential ports are firewalled off.

 

There are some excellent tips for securing CentOS Here otherwise with Debian / Ubuntu there are many guides out there too. With Plesk I would recommend CentOS + SELinux set to enforcing in the future.





894 posts

Ultimate Geek

Subscriber

  # 1694671 24-Dec-2016 20:05
One person supports this post
Send private message

Like Michael said - if they wiped /var/log/ completely - not just apache logs then they have had some form of root access.  You cant trust your shell anymore.  If you use any passwords on that server elsewhere - immediately change them.

 

The provider of your server may have IT that for a fee can assist you.

 

If it were me - I would start setting up a new server and migrate services off this system to the new one. Wordpress belongs on a seperate system even if its just a small virtual on linode etc.. that way its low value and easy to setup again.

 

Personally I wouldnt use Plesk on the new system I would try and work without it or anythign similar (cpanel etc..).

 

Ideally long term:

 

Use Ansible or any other configuration management system so you can recreate your server config without any guess work - put that under source control so if a server is owned you simply run a command to setup a new server - the configuration isnt special or hard to replicate.  Your repositories need to be somewhere else - be it another system or bitbucket, github.

 

Snapshot the current server so later when you have more time you can identify the scope of what they did.


15235 posts

Uber Geek

Trusted
Subscriber

  # 1694672 24-Dec-2016 20:07
Send private message

Take a backup before you roll back to a backup.

 

Immutable computing (Ansible etc) is super useful, occasionally, or for systems that change regularly. On AWS I would use CloudFormation with static resources stored or backed up on S3.




455 posts

Ultimate Geek

Trusted

  # 1694700 24-Dec-2016 21:00
Send private message

hio77:

 

Sounds indeed like a backdoor has been used.

 

Willing to bet one of your PHP scripts will be the offending targetpoint.

 

Hearing that your var/log has been fully wiped is concerning, are you running PHP/Apache as root?

 

 

 

 I think so too. No, PHP/Apache are definitely not running as root. They are under a different user.

 

 

 

 

As someone who used to run large communities, be glad you have not been attacked with a ddos, That can be far worse due to the affect it will have on your datacenter and such..

 

word of advise for if that issue should crop up for you though, putting all your eggs in one basket is often dangerous, though the cheapest option. 

 

 

I have been DDoS-ed several times before. I've been running large phpBB sites since 2004 and have regularly been a victim of such attacks. Our host (Leaseweb) is running anti-DDoS mechanisms so it typically isn't that much of a problem.

 

ratsun81:

 

 ssh should never be open to public it is asking for trouble. If you are going to have ssh open it should ONLY be to your IP address or to your main sysadmins IP. 

 

 

 

 

 

 

It's on a non-standard port, and I access it frequently from different locations. If I were to lock it to my home address I wouldn't be able to access my server when travelling

 

 

 

biggal:

 

change login pass

 

install Fail2Ban 

 

install this script

 

https://github.com/trick77/ipset-blacklist

 

if your using wordpress use zbblock 

 

http://www.spambotsecurity.com/zbblock.php

 

block all ports and accept port 80 443 ssh  turn off ftp and only turn it on when neaded

 

and change ssh port from port 22 to something else like port 2345 etc

 

 

 

 

 

Thanks for your tips, Fail2Ban has been installed and is active and up-to-date. SSH is already on a non-standard port.

 

 

 

michaelmurfy:

 

Right - if /var/log is wiped then do not trust this server. It may have well have a rootkit on it. I'm afraid you need to wipe / restore this server.

 

1) If Plesk is used follow the Plesk security best practices Here. Ensure SSH is firewalled off to a trusted host only. I'd recommend either using Debian or CentOS (with SELinux enabled and configured).
2) For web apps that run outside of the web server configure nginx (included with Plesk) to proxy the web app instead of allowing direct access.
3) Ensure that updates are run - this could well have been an attack on an insecure website coupled with the Dirty Cow exploit allowing full root access. Debian can install security updates automatically with cron-apt however you'll need to maintain kernel upgrades via a reboot.
4) If the sites are yours then consider if you really need Plesk.
5) Put SVN/GIT on another server or host it outside of local management on something like BitBucket - always consider a server with a control panel (like Plesk) insecure as these are frequently targeted.

 

Also ensure you check each site before restoring as you may be restoring a compromised site and get set back to step 1. Do not take shortcuts when it comes with security and ensure all non-essential ports are firewalled off.

 

There are some excellent tips for securing CentOS Here otherwise with Debian / Ubuntu there are many guides out there too. With Plesk I would recommend CentOS + SELinux set to enforcing in the future.

 

 

My server has been installed with a CentOS distribution by our hosting provider that was totally locked down to begin with. It contains CentOS 6.8 (64), kernel 2.6.32 and SELinux is enforced.
Plesk licenses are included and are installed by default, I have been running on this platform for close to 6 years.

 

I will attempt to follow the rest of your guidelines, thanks.

 

 

 

rphenix:

 

Like Michael said - if they wiped /var/log/ completely - not just apache logs then they have had some form of root access.  You cant trust your shell anymore.  If you use any passwords on that server elsewhere - immediately change them.

 

 

Apache logs are still in tact, just /var/log that has been targeted.

 

 

 

 

The provider of your server may have IT that for a fee can assist you.

 

They do but it will be a nightmare to reinstall everything, and very costly.

 

 

If it were me - I would start setting up a new server and migrate services off this system to the new one. Wordpress belongs on a seperate system even if its just a small virtual on linode etc.. that way its low value and easy to setup again.

 

Right, I will consider using different sites for the Wordpress instances, they are indeed not relevant for the main community website.

 

 

 

 

Ideally long term:

 

Use Ansible or any other configuration management system so you can recreate your server config without any guess work - put that under source control so if a server is owned you simply run a command to setup a new server - the configuration isnt special or hard to replicate.  Your repositories need to be somewhere else - be it another system or bitbucket, github.

 

Snapshot the current server so later when you have more time you can identify the scope of what they did.

 

 

 

 

I have never heard of this platform. Can you suggest what I can use to snapshot the current server?

 

 

 

 





Gigabit




455 posts

Ultimate Geek

Trusted

  # 1694701 24-Dec-2016 21:03
Send private message

Oh by the way, I read up on "undeleting" files on Linux.

 

 

 

Would it be possible to restore the contents of the /var/log folders?

 

 

 

I just looked into the bash_history files of all accounts, they weren't removed, modified or showed any suspicious activity. So it looks like he was able to execute some kind of shell commands from whatever backdoor that was used.

 

 





Gigabit


 1 | 2 | 3
Filter this topic showing only the reply marked as answer View this topic in a long page with up to 500 replies per page Create new topic



Twitter and LinkedIn »



Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





News »

Arlo unveils its first video doorbell
Posted 21-Oct-2019 08:27


New Zealand students shortlisted for James Dyson Award
Posted 21-Oct-2019 08:18


Norton LifeLock Launches Norton 360
Posted 21-Oct-2019 08:11


Microsoft New Zealand Partner Awards results
Posted 18-Oct-2019 10:18


Logitech introduces new Made for Google keyboard and mouse devices
Posted 16-Oct-2019 13:36


MATTR launches to accelerate decentralised identity
Posted 16-Oct-2019 10:28


Vodafone X-Squad powers up for customers
Posted 16-Oct-2019 08:15


D Link ANZ launches EXO Smart Mesh Wi Fi Routers with McAfee protection
Posted 15-Oct-2019 11:31


Major Japanese retailer partners with smart New Zealand technology IMAGR
Posted 14-Oct-2019 10:29


Ola pioneers one-time passcode feature to fight rideshare fraud
Posted 14-Oct-2019 10:24


Spark Sport new home of NZC matches from 2020
Posted 10-Oct-2019 09:59


Meet Nola, Noel Leeming's new digital employee
Posted 4-Oct-2019 08:07


Registrations for Sprout Accelerator open for 2020 season
Posted 4-Oct-2019 08:02


Teletrac Navman welcomes AI tech leader Jens Meggers as new President
Posted 4-Oct-2019 07:41


Vodafone makes voice of 4G (VoLTE) official
Posted 4-Oct-2019 07:36



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.


Support Geekzone »

Our community of supporters help make Geekzone possible. Click the button below to join them.

Support Geezone on PressPatron



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.