Hardening an AWS EC2 Instance

One thing that AWS does is lull us into a sense of false security. You feel secure because your system is on one of the most robust infrastructures available, and it has a nice UI with security groups that you proudly followed Amazon's instructions to set up. But now what? You aren't done by a long shot. We should add an extra layer of security to your instance.

You need to harden your server. What does this term mean? Well, it means that you need to secure it from outside attacks. Typically, the vector of attack on an EC2 instance is a brute force SSH attack or an attack through some other open port on unsecured HTTP interface or compromised user account etc.

To sum it up, at one point or another, your server will come under attack. Typically, the attack is a bot trying to brute force its way into your machine. I will also provide some common commands that you can implement to attempt to mitigate people doing bad things on your machines. Unfortunately, I do need to stress that these are suggestions based on my experience and have helped me in the past. We would be foolish to think that the below are they are the only steps that you would need to implement to keep secure. 

For This Article I am Going to Make a Few Assumptions:

• You have an AWS account.

• You have an Ubuntu EC2 instance with LAMP stack.

• You have an established SSH.

• You have access to a root account.

• You need to provide access to multiple services and ports to diverse individuals.

• That you are running a public facing web interface.

• Your instance is not behind any sort of firewall.

For this article, we are going to be focused on the User Interface of AWS combined with the command line.

Part I: Getting Ready

OK, now that we have had our preamble, let's get to work on this.

User Groups/Security:

Step 1: Login to your AWS account and move to your EC2 instance

Step 2: On the left side of the navigation click on Security Groups

Step 3: Click on the inbound tab and you can do the following:

*Please note that these are only suggestions. You may need to set-up the instance and security to meet your specific needs.

If you need to have HTTP/HTTPS open to anybody (so, that would be 80/443) then select the HTTP/HTTPS option. AWS will fill out everything you need. For the IP section, this up to you, but if it is a public facing site, then say any IP.

If you need to manage MySQL, then select that option and select My IP (this is an article about security so we don't want anybody to be able to try and access your database).

You most likely need port 22 open. This is the default SSH port and a way to securely tunnel into your instance to maintain it by command line. Or if you are using FileZilla or WinSCP, port 22 serves to move files around if you don't use command line from your instance.

Additional Information About the SSH Port:

Now, you can do some fancy things like move port 22, but that involves frankly too much work. When I mean too much work, I mean it is a few files and some services. In all due candor, I have done it only once years ago with help and hated it then. With that in mind, I do everything in my power to avoid that. The reality is that I have a job to do and I need to do it fast.

After we do the basic security steps on the AWS interface, we need to use Linux's internal firewall referenced by the IP tables service. You can manage pretty much everything from this service. You can program who/what can interact with your instance. For most people, it is also a general pain in the neck to use, and, as usual, the Internet is replete with overly complex examples of how to configure it. If you search on the net there, is an insane number of articles written about it, and like anything else on the net, it has some useful information, but of course, a lot of detailed info you probably don't need at this stage.

Part II: Starting Using IP Tables via SSH

Now that we are done with the preamble, we can dive into the command line and turn on your additional protections. As with all my articles, there are always a few simple things that you need to keep in mind:

Step 1: Login to your instance

Step 2: Sudo su - or whatever your super user is.

Step 3: Now we need to know if anything is going on with your IPtables.

Type:

Iptables --list

It probably will show you some things:

Chain INPUT (policy ACCEPT)

target prot opt source destination

Chain FORWARD (policy ACCEPT)

target prot opt source destination

Chain OUTPUT (policy ACCEPT)

target prot opt source destination

So right now, we have a vanilla set-up that will pretty much allow anything.

The first thing we need to do is stop the standard brute force SSH attacks. This is by far one of the most common forms of attack. The idea is that a bot or other automated program will keep attacking your SSH until it gets in.

Type:

iptables -I INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --set
iptables -I INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 -j DROP

Basically, what we have done with these two commands is create a bottleneck that will slow down an SSH brute force attack so that it doesn't impact your machine and eventually the attack stops. The syntax is a bit funny, but basically, we are telling the system to accept connections on port 22 only if it is a new connection. It will also drop connections if the hit count (failed attempts) hits four and it slows it to 60 seconds.

Now, let's manage our port 80 and 443 connections by typing:

iptables -A INPUT -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p tcp --sport 80 -m state --state ESTABLISHED -j ACCEPT

iptables -A INPUT -p tcp --dport 443 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p tcp --sport 443 -m state --state ESTABLISHED -j ACCEPT

At this point, your AWS interface will allow HTTP port 80/443 and 22 into your system. We also want to block pretty much anything other than a few other items.

This basically allows already connected to keep going using conntrack:

iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

This allows your server(s) to talk to each other. (I.e. one of your services needs to talk to another service on the server):

iptables -A INPUT -i lo -j ACCEPT

Another traditional attack vector is scanning and trying to log into available ports. Now by default, most of your ports are closed on an initial AWS set-up. The only ones that would be open or should be are the ones you set-up earlier via the AWS console.

However, let's be prudent and just make sure.

Use the below command to see what ports are open (they should match the rules you already set-up).

Type:

netstat -ntlp | grep LISTEN

There may come a time where you need to open a specific port and don't want to do it via the web-interface for AWS.

The command to open a port is below. Feel free to change the --dport to any number you wish.

Type:

iptables -A INPUT -p tcp --dport 25 -j ACCEPT

To close an open port:

Iptables -A INPUT -p tcp --dport 25 -j DROP

If you are anything like me and have screwed this up in the past, you need to dump these rules:

Iptables -f (or flush) this will remove everything and get you back to start again.

One of the other things that you need to recognize with your instance is that any rules that you create will only be good until you reboot your server. It will not be persistent (i.e. it won't last if you reboot). There are ways to handle that, and I will explain later.

So really, at this point, we have set up AWS to accept HTTP/HTTPS calls on port 80 and 443. We have allowed port 3306 to be open as well and, most importantly, port 22 for SSH. We have set up some basic rules that should discourage brute force SSH attacks and generally speaking, via AWS, most ports are closed but for the ones we have opened.

If this is a default setup then, for the most part, you should be OK with this. Basically, we have simply enabled the basic IP tables defenses for your instance. You can do much more with IP tables, such as allowing MySQL to only talk to a specific IP address or range of computers via 3306.

You actually may be asking why do I need this? Amazon sets up the control points through its UI. The problem with that type of thinking is if something gets compromised either through your user accounts, your instance setup, or an AWS firewall hiccup, you will want an extra layer of security.

Part III: Saving Your IP Tables Configuration via Command Line

Now that we have set up our IP tables, we have to take care of that persistent matter. There are two common ways of dealing with this. Because we are using an Ubuntu setup, then a command like iptables save will not work.

This is where things get sticky (and they shouldn't in my mind) but we will dive into this a bit. If you are using a different distro than Ubuntu then you can type:

Iptables save

Or to reload a saved configuration:

Iptables reload

Pretty easy right? But for Ubuntu, you need to do all sorts of gyrations and manipulate a lot of files. Who has the time? There is a utility available now called iptables-persistent, which we will discuss later.

The traditional Ubuntu commands to save are:

iptables-save > /etc/iptables.up.rules

The below will restore them into memory on a reboot — there are ways to add it to a reboot script, but that's not my focus:

iptables-restore < /etc/iptables.up.rules

Now for third-party iptables-persistent, which many suggest is an easier approach:

Sudo apt-get update
Sudo apt-get install iptables-persistent

Now, to be candid I was a bit leary of this as I was used to doing things the other way, but decided to use it.

Type:

sudo service iptables-persistent save

To reload a saved iptables:

sudo service iptables-persistent reload

To clear out all rules:

sudo Service iptables-persistent flush

Part IV: Third-Party Solutions to Assist

The Usage of IP tables should be able to handle most of your needs when combined with AWS' built-in security. However, for a complete security suite, you may want to consider the following programs.

For security needs, I will suggest two programs that will help IP tables by monitoring your services that perform log entries. These programs are fail2ban and DenyHost. They monitor the log files such as Apache, MySQL, SSH, and others.

Whereas IP tables tries to stop the attack before it hits (as in the SSH attack) these programs will scan the log files and looks for patterns. At that point, they can stop them.

Fail2ban scans most log files, whereas DenyHost works in the same way but is more centered on SSH connection monitoring.

What if I Think or Have Been Compromised:

If you feel that you may have been compromised, there are a number of things you can do. However, since that is not the focus of this article I will mention two common tools that can help you determine if you have a rootkit installed on your machine.

RKHunter

Chrootkit

Part V: More Practical Usages:

To block the echo request a common request for DDoS attacks:

iptables -A OUTPUT -p ICMP --ICMP-type echo-request -j DROP

To drop certain IP addresses of known offenders.

iptables -A INPUT -s xxx.xxx.xxx.xxx -j DROP

Part VI: Summary and Conclusion

Security should always be your biggest concern in your cloud or local deployments. Although you can't stop a determined attacker, you can mitigate your chances by following some prudent steps as described above. The tools that I have listed above are designed to help you get a better understanding of what you may need to do to help harden your instance. 

This article is by no means definitive, but for the developer who suddenly finds himself in a position to harden an instance, or a consultant on a project, this should give you the basic understandings of what you need to do. 

If you feel that your instance(s) would be a target, then you should consult with security consultants. Also, make sure that your web application security has been checked as well as your internal access procedures.