Geekzone: technology news, blogs, forums
Guest
Welcome Guest.
You haven't logged in yet. If you don't have an account you can register now.


Filter this topic showing only the reply marked as answer View this topic in a long page with up to 500 replies per page Create new topic
1 | 2 | 3
Mr Snotty
8865 posts

Uber Geek

Moderator
Trusted
Lifetime subscriber

  # 1694709 24-Dec-2016 21:16
6 people support this post
Send private message

Please don't attempt to keep using this server.

 

If they've had access to remove contents inside /var/log they've had root access meaning they could have done anything. From this point you can't trust the server so recreate it and start fresh. Unless if you were using ZFS (and are able to restore a ZFS snapshot) which is highly unlikely given you're using CentOS there is no way to recover the files.

 

The only setup I'd trust is one I've done myself. Given your hosting provider set this server up ask them for support. Yes, there will be downtime and yes, it'll be (likely) costly.







455 posts

Ultimate Geek

Trusted

  # 1694717 24-Dec-2016 21:27
Send private message

The problem is that setting up a new server is not an option due to time restriction and immediate requirements.

 

We have 300,000+ user accounts, and currently up to tens of thousands unique visits daily.
We've just released a new game update for Christmas and there are/will be thousands of keen punters looking to get the latest update.

 

Combine that with our speedy pay-to-download service the loss of income is considerable in what is normally the busiest time of the year.

 

I have precisely 48 hours to get something working again or I won't be able to work on this for another month due to the holidays which means many angry gamers around the world, a disappointed community and huge loss of income for our team. Even if I could do it in 48 hours it wouldn't be enough to set up all the features of the site from scratch. :(

 

Really bummed out on this one





Gigabit


 
 
 
 


1903 posts

Uber Geek


  # 1694723 24-Dec-2016 21:48
Send private message

ScuL:

 

 

 

ratsun81:

 

 ssh should never be open to public it is asking for trouble. If you are going to have ssh open it should ONLY be to your IP address or to your main sysadmins IP. 

 

 

 

 

 

 

It's on a non-standard port, and I access it frequently from different locations. If I were to lock it to my home address I wouldn't be able to access my server when travelling

 

 

the world should not have access to anything other than 443/80. only one IP address should have the ability to connect out of this range.  if you need roaming or shared access then employ a vpn from that one trusted IP.  It's also trivial to employ sftp, sandpitted at that.


21263 posts

Uber Geek

Trusted
Lifetime subscriber

  # 1694746 24-Dec-2016 22:59
4 people support this post
Send private message

I can't offer much in the way of specifics on Linux as it's not my area of expertise, but the advise already given is pretty spot on from what I can see. 

 

The concern I have with a quick fix, is you aren't doing yourself or your users any favours with this approach as it will almost certainly mean more downtime later, and potentially explaining to them that you ignored best practices and didn't wipe the server and now the x consequences are affecting them.

 

I'm with Michael, as painful as it would be, you simply can't trust this server now.


'That VDSL Cat'
10998 posts

Uber Geek

Trusted
Spark
Subscriber

  # 1694747 24-Dec-2016 23:01
2 people support this post
Send private message

ScuL:

 

The problem is that setting up a new server is not an option due to time restriction and immediate requirements.

 

We have 300,000+ user accounts, and currently up to tens of thousands unique visits daily.
We've just released a new game update for Christmas and there are/will be thousands of keen punters looking to get the latest update.

 

Combine that with our speedy pay-to-download service the loss of income is considerable in what is normally the busiest time of the year.

 

I have precisely 48 hours to get something working again or I won't be able to work on this for another month due to the holidays which means many angry gamers around the world, a disappointed community and huge loss of income for our team. Even if I could do it in 48 hours it wouldn't be enough to set up all the features of the site from scratch. :(

 

Really bummed out on this one

 

 

 

 

to be honest, with those numbers... You need to redo your whole system design. 

 

 

 

You have put all your eggs in the one basket, and as a result one simple attack has caused a hell of a lot of havoc.

 

 





#include <std_disclaimer>

 

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.


Mr Snotty
8865 posts

Uber Geek

Moderator
Trusted
Lifetime subscriber

  # 1694778 25-Dec-2016 01:57
25 people support this post
Send private message

I've been helping @ScuL tonight to migrate to a new server I'm setting up.

The server was compromised via dirty cow - dodgy .php script by the looks. They disabled all logging on the server (including bash history) and hidden all evidence of what has been done. In ScuL's defense he was following best practices with most things but had way too much sitting on one server and whilst it was patched for most things (including being SELinux enforced) it wasn't patched against Dirty Cow with its impressive uptime of 326 days. I've been going through dumping everything off and migrating them to a new Ubuntu VM sitting on a VM host I have space on for now.

 

Without going into too many details this site handles quite a few thousand hits per day (seeing on average ~200/sec hits to the new server).

 

Quite a big job to do but I can't have somebody lose a server on Christmas day to some script kiddie especially since they just did a release of what they're working on. It sucks as it is losing a production server especially hosting a busy site but the timing on this really blows too. Christmas eve, after a release (which seems many people are excited about) and dealing with many GB's of data, multiple MySQL databases and a forum with other peoples hard work.

 

Anyway with most of the hard work out of the way I am sure ScuL can sleep easier now and enjoy Christmas.





5572 posts

Uber Geek


  # 1694815 25-Dec-2016 08:06
Send private message

michaelmurfy:

 

I've been helping @ScuL tonight to migrate to a new server I'm setting up.

 

 

@michaelmurfy Good on ya for helping out @ScuL, especially at this time! High Fives to you.


 
 
 
 




455 posts

Ultimate Geek

Trusted

  # 1694850 25-Dec-2016 09:55
One person supports this post
Send private message

Michael thanks so much for helping me out, this is true Christmas spirit. We both stayed up until the early hours of the morning on Christmas Day to arrange backups and a temporary VM hosted by Michael.

 

I've also just set up an Amazon EC2 instance to move some of the smaller scripts too so I can get them going independently from the community server.

 

I have two family functions today (which means more downtime) but am hoping to have restored 80% of the functionality by this evening.

 

Then after making a dump of the server (which I hope will contain some more evidence of what the culprits have done) I will fully wipe the server and reinstall it ..

 

Like Michael said I felt the server was pretty reasonably patched up but it's easy to make a slip up.

 

 

 

 





Gigabit


15206 posts

Uber Geek

Trusted
Subscriber

  # 1694864 25-Dec-2016 11:29
Send private message

I'm pretty good with AWS, certified etc, I can help with securing it, cost optimisation, backups, etc. Probably not today though.


Mr Snotty
8865 posts

Uber Geek

Moderator
Trusted
Lifetime subscriber

  # 1694975 26-Dec-2016 00:13
Send private message

Managed to get most of the sites up and running today nicely (very nicely in-fact) complete with HTTPS (via letsencrypt) and HTTP2 globally. Blame the fact the house was full of girls who liked the scorching summer heat whilst I hid away in my room with the fan running and curtains closed trying to not pass out from heat stroke.

 

Hopefully (fingers crossed) we don't see a reinfect. At-least this time it is sandboxed... Currently seeing around 300 hits/s with room to grow.





385 posts

Ultimate Geek


  # 1694979 26-Dec-2016 00:42
One person supports this post
Send private message

michaelmurfy:

 

I've been helping @ScuL tonight to migrate to a new server I'm setting up.

The server was compromised via dirty cow - dodgy .php script by the looks. They disabled all logging on the server (including bash history) and hidden all evidence of what has been done. In ScuL's defense he was following best practices with most things but had way too much sitting on one server and whilst it was patched for most things (including being SELinux enforced) it wasn't patched against Dirty Cow with its impressive uptime of 326 days. I've been going through dumping everything off and migrating them to a new Ubuntu VM sitting on a VM host I have space on for now.

 

Without going into too many details this site handles quite a few thousand hits per day (seeing on average ~200/sec hits to the new server).

 

Quite a big job to do but I can't have somebody lose a server on Christmas day to some script kiddie especially since they just did a release of what they're working on. It sucks as it is losing a production server especially hosting a busy site but the timing on this really blows too. Christmas eve, after a release (which seems many people are excited about) and dealing with many GB's of data, multiple MySQL databases and a forum with other peoples hard work.

 

Anyway with most of the hard work out of the way I am sure ScuL can sleep easier now and enjoy Christmas.

 

 

 

 

good to hear

 

i wonder how many servers out there are not been patched from dirtycow since it been around since 2007

 

here a tutorial if any one want to see of there servers are patched or not

 

https://www.digitalocean.com/community/tutorials/how-to-protect-your-server-against-the-dirty-cow-linux-vulnerability





 

 

 




455 posts

Ultimate Geek

Trusted

  # 1695068 26-Dec-2016 11:44
Send private message

They're really out to get us, they're currently running a DDoS attack to get Michael's new server offline.

 

He's increased the bandwidth and I'm currently setting up Cloudflare





Gigabit


15206 posts

Uber Geek

Trusted
Subscriber

  # 1695071 26-Dec-2016 11:52
One person supports this post
Send private message

Once you set up CloudFlare you'll need to change IPs. Probably best to go the business plan, once you pay anything at all their willingness to absorb a DDOS and provide service rises.


15206 posts

Uber Geek

Trusted
Subscriber

  # 1695091 26-Dec-2016 12:20
Send private message

Also for CloudFlare, make sure your IP isn't in any DNS record that leaves CloudFlare - including MX records, subdomains, etc. That gives them another way in. If you're hosting in AWS set up both network ACLs and security groups that allow traffic only from CloudFlare (IPs here) and your home/work IPs, not everywhere.


'That VDSL Cat'
10998 posts

Uber Geek

Trusted
Spark
Subscriber

  # 1695144 26-Dec-2016 14:03
Send private message

as @timmmay has mentioned, cloudflare is not the end all for ddos attacks.

 

 

 

Free accounts will get temporarily disabled if an attack is deemed too large, Speaking from experience with a previous community.. If they wish to hit you offline and you actually do a good job at keeping your IP hidden, they will just storm the gates till cloudflare gives up.

 

 

 

Do be aware if your trying to keep your IP safe, things like email headers are easy places to find your obscured IP, If that fails a remote image upload feature is also very easily abused (where you give the weblink to the image rather than uploading it and the website downloads the image itself)

 

 

 

 

 

Lastly, attacks are normal for communities, It is a sad truth.. take head of this attack as a warning to stay ontop of your security.





#include <std_disclaimer>

 

Any comments made are personal opinion and do not reflect directly on the position my current or past employers may have.


1 | 2 | 3
Filter this topic showing only the reply marked as answer View this topic in a long page with up to 500 replies per page Create new topic



Twitter and LinkedIn »



Follow us to receive Twitter updates when new discussions are posted in our forums:



Follow us to receive Twitter updates when news items and blogs are posted in our frontpage:



Follow us to receive Twitter updates when tech item prices are listed in our price comparison site:





News »

Major Japanese retailer partners with smart New Zealand technology IMAGR
Posted 14-Oct-2019 10:29


Ola pioneers one-time passcode feature to fight rideshare fraud
Posted 14-Oct-2019 10:24


Spark Sport new home of NZC matches from 2020
Posted 10-Oct-2019 09:59


Meet Nola, Noel Leeming's new digital employee
Posted 4-Oct-2019 08:07


Registrations for Sprout Accelerator open for 2020 season
Posted 4-Oct-2019 08:02


Teletrac Navman welcomes AI tech leader Jens Meggers as new President
Posted 4-Oct-2019 07:41


Vodafone makes voice of 4G (VoLTE) official
Posted 4-Oct-2019 07:36


2degrees Reaches Milestone of 100,000 Broadband Customers
Posted 1-Oct-2019 09:17


Nokia 1 Plus available in New Zealand from 2nd October
Posted 30-Sep-2019 17:46


Ola integrates Apple Pay as payment method in New Zealand
Posted 25-Sep-2019 09:51


Facebook Portal to land in New Zealand
Posted 19-Sep-2019 18:35


Amazon Studios announces New Zealand as location for its upcoming series based on The Lord of the Rings
Posted 18-Sep-2019 17:24


The Warehouse chooses Elasticsearch service
Posted 18-Sep-2019 13:55


Voyager upgrades core network to 100Gbit
Posted 18-Sep-2019 13:52


Streaming service Acorn TV launches in New Zealand with selection with British shows
Posted 18-Sep-2019 08:55



Geekzone Live »

Try automatic live updates from Geekzone directly in your browser, without refreshing the page, with Geekzone Live now.


Support Geekzone »

Our community of supporters help make Geekzone possible. Click the button below to join them.

Support Geezone on PressPatron



Are you subscribed to our RSS feed? You can download the latest headlines and summaries from our stories directly to your computer or smartphone by using a feed reader.

Alternatively, you can receive a daily email with Geekzone updates.