It's The Muppet Show!

Linux System Hardening

, posted: 14-Oct-2011 20:30

Anyone who knows me fairly well knows that I'm a big Linux fan. I'm not so crazy as to think Linux is the answer for anything (my laptop happily runs XP), but I'm pretty sure as a server, it can't be beat.

In this article I'm going to talk about some of the methods you can use to harden a running linux system in a server environment. Some of the methods are also applicable to a desktop system, but given the nature where people usually want to tinker and fiddle, hardening isn't usually a great idea, as you're often having to edit rules to allow things. You get to the point where an “allow all” would be easier, defeating the purpose of the rules in the first place.
This article is written as a guide to some of the methods available to you to protect your systems.  It doesn't attempt to explain detailed implementation steps, there's many guides and howto's out there if you're after these things. I'm happy to help if you ask as well.

Finally, before I start it's important to remember that even if you add every single thing I talk about here to your system, it's a fallacy to sit back and think “Now I'm unhackable!”. If your sensitive data is stored in a web directory you leave exposed to world, no amount of fancy pants rules/filtering and memory hardening is going to help you. It always pays to monitor your logs, check your system and sometimes approach your own system as if you're an attacker without the passwords and see how far you can get.

Starting from Scratch
There's aready a number of Linux distributions that are built from the ground up with hardening as part of the tool chain. Examples include Openwall and Hardened Gentoo.
If you're starting from scratch, I'd suggest looking at one of those systems.
This article is written to give some pointers and ideas to people who already have a working system but would like to take some steps to strengthen it.

SpamHaus Drop List
The first and most simple way to help protect your machine against automated attacks is to add the SpamHaus DROP list to your iptables rules. The drop list is a list of IP addresses that haven't officially been allocated by IP registers yet.  It means if you see traffic coming from them, you can be pretty sure it's traffic from a spammer.  There's a number of scripts out there that make applying DROP a simple step, this is the one that I currently use.

Active Response - Fail2Ban
The next way to protect your system against attack is to monitor the system logs and act when you see suspicious activity. The tool I use for this job is called fail2ban and will, after seeing a custom number of errors/denies appear in a log file, take a certain action. That action could be to email you, to send a tweet, or to add an iptables rule that filters the IP triggering the log from access the machine.

I have fail2ban setup so that when it sees more than 3 failed SSH attempts from the same IP address, it adds that IP Address to an iptables rule which drops all traffic to/from that address. I also have it setup to blacklist people trying to spam me (if they're listed in a Real-Time Blackhole List they get blacklisted). Addresses that tries to brute force the IMAP server are added to IPTables as are addresses that triggers the Web Server firewall rules (more on this in a minute)

By adding a simple system such as fail2ban, you protect your server against the automated SSH/IMAP scans that are common, as well as anyone else without much clue that tries to bruteforce your password. This attack won't stop a resourceful attacker who has an army of zombies with many thousands of IP addresses from trying though.

Apache Webserver with PHP

Suhosin PHP Patch + Module
The next step I take is to harden the webserver. I, like many people, use the Apache server with a lot of PHP Scripts that I just have to trust are security-hole free. Sadly I don't really have a strong enough understanding to read every file and verify it's secure. Given that I run some fairly commonly exploited software (phpMyAdmin being the biggest) I feel it's important to make it as harder to exploit my server than normal, in the hope of thwarting a dumb script kiddie.
The two tools I use for this are: The Suhosin PHP patch and mod_security. The Suhosin PHP patch is the easiest for me, as Debian user the PHP binary itself is already patched with the “simple” version. Suhosin is best descirbed by this quote from their website:

Suhosin comes in two independent parts, that can be used separately or in combination. The first part is a small patch against the PHP core, that implements a few low-level protections against bufferoverflows or format string vulnerabilities and the second part is a powerful PHP extension that implements all the other protections."

It's also quite simple, just a single Debian command to add the Sushion PHP module as well, thus completing the protection offered by Suhosin.
I then go through and tweak the various options it offers, such as enabling session and cookie encryption. You can configure Sushion so that cookies sent via the standard PHP Method (note: Not all PHP apps use this!) are valid only if they're used from the /24, /16 or /8 netblock. Which means that if someone in Mexico steals your cookie, they can't use it from their IP Address.

Sushion is easy to install, simple to configure and hasn't in my experience broken any of the PHP Applications I use. I consider it a fairly low-key but effective protection.

Apache Module - mod_security

mod_security is the next level of protection I add to a Apache webserver. mod_security by itself does nothing, it's not until you give it a list of rules that it becomes useful. Thankfully it comes with what's known as the Core Ruleset, a set of rules that apply broadly (but not perfectly) to most webapps. Using this ruleset, mod_security can detect and act upon a very wide range of commonly used web exploits. Things such as SQL Injection, Javascript Injection etc can all be protected against. It will trigger when it sees the webserver echoing PHP instead of text, things like that. mod_security is a very powerful Apache module but it comes at a cost.
The first is that the more rules you throw at it, the slower it is to respond to each web request. The second cost is that the Core Ruleset is not easy to configure. It's best to install it in a monitoring only mode and then use your website(s) as you and your users normally would. Then you can examine the logs that mod_security generates, then modify the rules you have to protect against the false positives it has no doubt detected. It's a bit of trial and error for a while until you get the hang of the way the rules work, how they can be modified and tweaked.

Once you've got mod_security's rules to a level you're happy with, you can move mod_security into active mode. This means it will drop/block/deny requests (you can configure which) that match one of it's rules. I use this in conjuction with fail2ban above. If an IP Address triggers a rule a few too many times (and spambots, zombie and exploit scanners do) they too are added to the iptables rules. This effectively stops most people scanning my websites and attempting 100+ different exploits on it.

Another feature of mod_security is the ability to change the string Apache uses to announce itself.  I change mine to not say Apache. This stops a number of the scanners that have in the past scanned for the Apache string before attempting an exploit.

Hardening the Kernel Itself - grsecurity
The final level of protection is the most complex, but one you get the most benefit from. Anyone that knows me well knows I'm always blabbing on about it, so apologies in advance if you're one of those people. Maybe stop reading now. The final system hardening method I use is the grsecurity patch for the Linux kernel. Grsecurity adds many hardening features to the kernel all which help to prevent exploits. The features it add are so numerous that really only reading the features page will do it justice.
I'll simplify and list a few here:

  • Memory Protection - Enforces non-executable pages
  • Kernel Stack Protection
  • Proper Address Space Randomization
  • Prevents various kernel structures
  • Role Based Access Control - Very powerful (A bit like SELinux)

The amazing thing is, adding all these features does not under most circumstances make the system any less usable. In it's default state (not using the RBAC system) you don't really notice that it's running. It will however catch many different memory and buffer overflows that the standard kernel would happily let happen.

The problem with grsecurity is that it's a patch you have to apply to the vanilla Linux kernel. This means you're in the territory of compiling your own Linux kernel, though there are some sites out there that offer precompiled gresecurity kernels for download.
For this reason it's probably out of the reach of many people, which is a shame. It'd be great if oneday the features that grsecurity are added into the kernel, but I don't think that'll happen anytime soon. A few features have already crept in on a case by case basis, such as address space randomisation.

The second part of the grsecurity is the Role Based Access Control. What this does is add another level of security to the system. With proper rule tuning, you effectively render the root login just as powerful as a normal user, that is not very!
Root won't be able to kill important processes, won't be allowed to replace important system binaries etc. The only way to do this is to authenticate to the RBAC system in an admin role, when full system privledges are once again granted. This makes an attackers job that much harder. They might fight for a couple of hours to get a root shell on your system, only to find that root shell is worthless to them.

grsecurity - a couple of stories
've used grsecurity for many years and there's been once it saved me when a webapp was exploited.  A bad person uploaded a binary that attempted a buffer overflow to root.  However grsecurity realized it was a buffer overflow, crashed the app and logged it.  It's also stopped some of the other root exploits that have been released, when I tested them on a grsecurity kernel, they were caught.  The system was unstable and crashed, but I'd much rather have a crashing system than a still working one + an unknown attacker with root.
If you're sharing your system with other people, another powerful feature is the ability to stop users running their own binaries.  They can only run the ones in /usr/bin, /bin etc.  Not their own ones in /home/user.  This shuts down another attack vector.

I hope this article gets you interested in a few of the measures you can take to help secure your system.    There's a lot of nasty people out there and anything you can do to make your system that bit more secure is a good thing.  Hopefully someone will realise your server isn't worth the effort and move onto someone elses!
If you have any questions or would like to know anything more, please post a comment below.  Thanks!

- muppet

Like this post? Don't forget to follow me on Twitter!

More information

Comment by sbiddle, on 15-Oct-2011 10:40

fail2ban is a great app. It's a mandatory install for anybody running an Asterisk PBX - and if more people installed it they wouldn't get h8x0red.

Comment by lefty.crupps, on 16-Oct-2011 00:11

Good info on securing the OS! The more we know about security and administration as a community, the better off we are.

Comment by shawnh, on 18-Oct-2011 08:49

What a great post!

You do all the security things so we don't have to!

(I have every intention of reading it real soon!) 

Comment by Michael, on 21-Oct-2013 20:05

Tim, You made a great start in hardening a Linux system!Due to my interest in Linux hardening, I was wondering if you know my tool Lynis as well?

muppet's profile

New Zealand

My name's Tim and I'm a Network Engineer! Well these days I'm actually a Network Architect. I'm one of those weird people that really love their job. I work with a great bunch of people in an interesting industry that's always changing. Sometimes slower than I'd like.

In my spare time I work on the LiCe Script for the EPIC5 IRC Client. Try it out, it's like Irssi but better.