URGENT Linux help needed, server under attack

Forums › Linux › URGENT Linux help needed, server under attack

1 | 2 | 3

michaelmurfy

meow
13260 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#1695176 26-Dec-2016 16:24

My provider mitigated around 5Gbit of the attack - there was still some traffic reaching the server which took everything down for a brief period. Just some script kiddies idea of fun by firing through an attack using a booter (pay to DDOS service). Once Cloudflare was implemented I locked down the server even more so it doesn't respond to ICMP etc, the attack was over as quickly as it started really.

Sure you could get the IP address from the Mail Relay and fire traffic to it but my provider is pretty good at mitigating blind attacks (since the IP doesn't respond to anything).

Michael Murphy | https://murfy.nz
Referral Links: Quic Broadband (use R122101E7CV7Q for free setup)

Are you happy with what you get from Geekzone? Please consider supporting us by subscribing.
Opinions are my own and not the views of my employer.

gzt

17149 posts

Uber Geek

Lifetime subscriber

#1696015 28-Dec-2016 23:46

There is a lot of phpmailer news today. If the server operates a list...

ScuL

487 posts

Ultimate Geek

Trusted

#1696017 28-Dec-2016 23:50

Not phpmailer I used sendy for mailing lists

Haere taka mua, taka muri; kaua e wha.

muppet

2570 posts

Uber Geek

Trusted

#1696039 29-Dec-2016 08:40

ratsun81:

ssh should never be open to public it is asking for trouble. If you are going to have ssh open it should ONLY be to your IP address or to your main sysadmins IP.

What trouble is it asking for? I've had all my SSH ports open to the public for the last ~16 years. It hasn't bitten me yet, nor anyone else I know that does the same.

Decreasing attack surface certainly isn't bad advice, but I think there's a lot of other things you should be doing before this sort of thing which provides minimal surface reduction. Things like mod_security, fail2ban, a custom kernel with modules removed (makes installing a rootkit much harder) or patching your kernel with grsecurity.

The biggest worry I see with this thread is what appears to be a lack of backups! If your server IS compromised you should hopefully have some backups so you can just restore with a fresh patched box and push all your data back over.

Good work though MM, though I did have chuckle at the melodramatic "but I can't have somebody lose a server on Christmas day to some script kiddie especially since they just did a release of what they're working on." What would have happened if you DID? :)

MadEngineer

4292 posts

Uber Geek

Trusted

#1696276 29-Dec-2016 19:12

SSH access hasn't been infallible. Blocking access to it adds another layer of security.

You're not on Atlantis anymore, Duncan Idaho.

jarledb

Webhead
3257 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#1696282 29-Dec-2016 19:17

SSH in itself is not the problem. As with everything else, you have to keep things updated and have sufficiently secure passwords and keys.

Mind you, there have been quite a few vulnerabilities in SSH, so just setting things up and thinking you can forget about them would be a big mistake.

Jarle Dahl Bergersen | Referral Links: Want $50 off when you join Octopus Energy? Use this referral code
Are you happy with what you get from Geekzone? Please consider supporting us by making a donation or subscribing.

JimsonWeed

126 posts

Master Geek
Inactive user

#1696287 29-Dec-2016 19:27

muppet:

ratsun81:

ssh should never be open to public it is asking for trouble. If you are going to have ssh open it should ONLY be to your IP address or to your main sysadmins IP.

What trouble is it asking for? I've had all my SSH ports open to the public for the last ~16 years. It hasn't bitten me yet, nor anyone else I know that does the same.

Decreasing attack surface certainly isn't bad advice, but I think there's a lot of other things you should be doing before this sort of thing which provides minimal surface reduction. Things like mod_security, fail2ban, a custom kernel with modules removed (makes installing a rootkit much harder) or patching your kernel with grsecurity.

The biggest worry I see with this thread is what appears to be a lack of backups! If your server IS compromised you should hopefully have some backups so you can just restore with a fresh patched box and push all your data back over.

Good work though MM, though I did have chuckle at the melodramatic "but I can't have somebody lose a server on Christmas day to some script kiddie especially since they just did a release of what they're working on." What would have happened if you DID? :)

Aye, you're right but, a lot of folks don't know to remove Protocol 1 in favour of Protocol 2 out of the configuration. You can configure SSH listen on any port you like and a simple NMAP of your network will cough up where it's running if it's exposed to the outside world :) I tend to trust be overly trusting of SSH also so, you're in good company. Where I see people getting into trouble is running things like WordPress. It's a kewl system and all that but, it's been ridden like Sea Biscuit for years now. I've got a hosted web presence and the hosting company won't let you shell in. They force you to use a web interface :) So yeah nah... I don't offer Jackola for services on my site - come read and go away. Pffft... running PHP that I have no control over... I might as well be running CGI scripts with huge GET/POST text fields.

Quic Broadband

Move to New Zealand's best fibre broadband service (affiliate link). Free setup code: R587125ERQ6VE. Note that to use Quic Broadband you must be comfortable with configuring your own router.

muppet

2570 posts

Uber Geek

Trusted

#1696301 29-Dec-2016 19:57

jarledb:

SSH in itself is not the problem. As with everything else, you have to keep things updated and have sufficiently secure passwords and keys.

Mind you, there have been quite a few vulnerabilities in SSH, so just setting things up and thinking you can forget about them would be a big mistake.

There's 1 remote hole (exploit) I can see there. Yeah, there's been some bugs in SSH but only 1 there that'll give a remote attacker a shell.

Vs PHP/Apache that's got random holes everywhere.

I'm not suggesting you shouldn't keep it updated - I'm suggesting that disabling from being public facing is an over-reaction.

Audiophiles are such twits! They buy such pointless stuff: Gold plated cables, $2000 power cords. Idiots.

OOOHHHH HYPERFIBRE!

muppet

2570 posts

Uber Geek

Trusted

#1696302 29-Dec-2016 20:01

JimsonWeed:

Aye, you're right but, a lot of folks don't know to remove Protocol 1 in favour of Protocol 2 out of the configuration. You can configure SSH listen on any port you like and a simple NMAP of your network will cough up where it's running if it's exposed to the outside world :) I tend to trust be overly trusting of SSH also so, you're in good company. Where I see people getting into trouble is running things like WordPress. It's a kewl system and all that but, it's been ridden like Sea Biscuit for years now. I've got a hosted web presence and the hosting company won't let you shell in. They force you to use a web interface :) So yeah nah... I don't offer Jackola for services on my site - come read and go away. Pffft... running PHP that I have no control over... I might as well be running CGI scripts with huge GET/POST text fields.

Show me an SSH install these days that still has SSH1 enabled and I'll show you Jesus :) - I mean a Linux Distro etc that enables it by default, forcing the user to disable it. I'd say it's been ~7-8 years?

But yes, agree with all your other points. You have to keep all your Wordpress/Plugins updated. I'm lucky in that I've always been able to have a number of boxes on the net where I fully manage them. If you don't have shell access but have to manage a PHP install I feel for you. That'd be a nightmare.

Audiophiles are such twits! They buy such pointless stuff: Gold plated cables, $2000 power cords. Idiots.

OOOHHHH HYPERFIBRE!

michaelmurfy

meow
13260 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#1696352 29-Dec-2016 21:45

@muppet Can I see Jesus now?

@JimsonWeed by default on most Linux distributions SHA1 is disabled. Yes, there are plenty of SSH servers out there likely with SHA1 or insecure OpenSSH configurations but you likely won't be able to find any out there on the internet.

@jarledb Again, in unpatched systems. I have a SSH server running here open to the internet however it does have fail2ban and also 2FA enabled on the only user account with the ability to login and I trust it won't get pwned (however you're correct if there was a OpenSSH exploit it /can/ get pwned). Since I maintain my systems (and this is the only SSH server of mine open to the rest of the internet) I'd like to think I am pretty safe.

All my systems have SSH closed off to a few trusted hosts of mine.

muppet

2570 posts

Uber Geek

Trusted

#1696415 30-Dec-2016 07:41

michaelmurfy:

@muppet Can I see Jesus now?

No.

You

a) Don't tell SSH to use version 1 (that's the ssh -1 flag)

2) you append the diffie-hellman-group1-sha1 to the list of algorithms supported. Appending it means that SSH will still use the default list, with this one added. If your server supported a better algo it'd be used. If you wanted to force the use of only diffie-hellman-group1-sha1 you should have removed the + sign. Even so, insecure key algo's isn't what we were talking about, we were talking about SSH versions.

As I said in my original post, I'm talking about a recent distro - the comment I was referring to was about people forgetting to disable it. I'm sure there's plenty of very old boxes out there with it enabled still, probably some not even supporting v2.

Audiophiles are such twits! They buy such pointless stuff: Gold plated cables, $2000 power cords. Idiots.

OOOHHHH HYPERFIBRE!

michaelmurfy

meow
13260 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#1696431 30-Dec-2016 09:46

muppet:

No.

You

a) Don't tell SSH to use version 1 (that's the ssh -1 flag)

2) you append the diffie-hellman-group1-sha1 to the list of algorithms supported. Appending it means that SSH will still use the default list, with this one added. If your server supported a better algo it'd be used. If you wanted to force the use of only diffie-hellman-group1-sha1 you should have removed the + sign. Even so, insecure key algo's isn't what we were talking about, we were talking about SSH versions.

As I said in my original post, I'm talking about a recent distro - the comment I was referring to was about people forgetting to disable it. I'm sure there's plenty of very old boxes out there with it enabled still, probably some not even supporting v2.

This server (dropbear running on a Squeezebox radio) only uses SHA1 and is recent (2014) - the dropbear instance (Dropbear sshd v0.49) also has several exploits. I'm just using this as an example of an insecure SSH installation.

mmurphy@pikachu:~$ ssh root@192.168.2.162
Unable to negotiate with 192.168.2.162 port 22: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1

However you're right about SSHv1 out in the wild - this has been disabled by default for years now.

MadEngineer

4292 posts

Uber Geek

Trusted

#1696471 30-Dec-2016 11:59

The other risk is internal staff/contractors that may have ssh access (probably not an issue in this case). Forcing them to connect via their workplace/client's VPN rather than allowing them to connect from anywhere has many benefits from additional logging to the fact that the VPN access is likely AD controlled and when staff leave this is always the first point of control that is disabled.

You're not on Atlantis anymore, Duncan Idaho.

Abo

78 posts

Master Geek

#1696909 31-Dec-2016 20:57

Just want to point out a few things:

Just because it's inconvenient to you and your income is no excuse to try re use an already compromised server, it's a danger to your users/customers. The only response needs to be to bring up a new server, freeze the compromised one for later evaluation/forensic eval.

I hope you have notified any users with accounts or were using that service that you were compromised (and/or reset all user passwords). If they used dirty cow to gain root access then they could have added files (malware etc) that users have downloaded or taken database information.

Unless you have the ubuntu live kernel patching or ksplice you will need to reboot the server if you do a kernel update. (and also things like glibc you should reboot for - technically just need to restart services)

Might be worth signing up to your OS security mailing list for any important notifications from them.

personally I prefer centos to ubuntu (selinux is so good) but each to their own

michaelmurfy

meow
13260 posts

Uber Geek

Moderator

ID Verified

Trusted

Lifetime subscriber

#1696965 1-Jan-2017 00:38

@Abo yes I fully understand where you're coming from however there are Linux users out there under the impression that Linux never needs rebooting (or as much as Windows). With me, I watch the security mailing lists and if anything comes up I use Puppet to update my servers and reboot if needed. Furthermore I use automated patching (using cron-apt) across everything.

Ubuntu is a very good operating system and features AppArmor which when set up is very comparable to SELinux. Personally I use Debian Jessie with most of my servers however Ubuntu 16.04 brings many new improvements that I do really like. I've deployed CentOS and Amazon Linux to places but to be perfectly honest think it is overrated and quite hard to manage especially to those new to Linux, for example Apache administration where Ubuntu / Debian has commands like "a2enmod" and "a2ensite" and the configuration in logical locations where CentOS takes the approach "if it aint broke, just leave it as-is" - remember, your Linux distro of choice is personal preference, it is only as secure as the person configuring it.

Anyway, I think I'll be assisting @ScuL for the foreseeable future with administration of his server fleet. We've moved what was once hosted on one server over to multiple servers and I'm about to do the fun things like MariaDB database replication and load balancing once I get some spare time. We had some teething problems at the start however these are mostly resolved and I've heard some pretty good things from their community.

1 | 2 | 3