Intrusion Prevention

The Problem

During the week of 1/25/2016 I noticed both the production and the development systems were under a brute force password attack. This was detected with "tail -f /var/log/auth.log" and watching the constant flow of failed login attempts:

Jan 26 14:53:48 sshd[7887]: last message repeated 5 times

Jan 26 14:53:48 ip-10-73-128-167 sshd[7887]: Disconnecting: Too many authentication failures for root [preauth] Jan 26 14:53:48 ip-10-73-128-167 sshd[7887]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=59.63.188.53 user=root Jan 26 14:53:48 ip-10-73-128-167 sshd[7887]: PAM service(sshd) ignoring max retries; 6 > 3 Jan 26 14:53:51 ip-10-73-128-167 sshd[7889]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=59.63.188.53 user=root Jan 26 14:53:53 ip-10-73-128-167 sshd[7889]: Failed password for root from 59.63.188.53 port 10066 ssh2

On prod, it seems all the attacks were to the account with username root, over a thousand attempts a day (each probably with 3 different passwords). On dev, other usernames are being guessed. The IP address of the attack changes at least daily.

##Susceptibility For production, the root account does not accept logins (root is only available via sudo, but I need to double check). Also the ubuntu account requires a ssh key and does not use passwords. So, I do not think production is really susceptible. This is not the case for the dev server. The server does use username and passwords to authenticate so this attack could succeed.

##Prevention A quick review of the possible solutions points to Fail2ban. It is the most popular option, has been recently updated, has useful documentation, it is discussed on stackoverflow and has a good lightning talk video. Like similar tools, fail2ban scans for attacks by reading standard log files. For example, from /var/log/auth.log it can see repeated failed login attempts from a specific IP address and add that address to a blacklist. Other kinds of attacks are also detected.

Both prod and dev have logcheck installed but it isn't running. It does not appear to be widely used.

##Configuring Fail2ban and Logwatch Fail2ban is installed with:

sudo apt-get install fail2ban

It was not previously installed on prod or dev.

Fail2ban detects attacks by searching with regular expressions in specific log files. The regular expressions and related details are specified in filter files (such as /etc/fail2ban/filters.d/sshd.conf). The actions to take in response to an attack are defined in action files (such as /etc/fail2ban/action.d/iptables.conf). Finally, jail files link filters to use with the actions to take when they are detected (/etc/fail2ban/jail.com). None of the built-in config files should be changed (to permit seamless upgrades). Instead, use a separate file with a .local extension.

The installation included three surprises. First, a bunch of libraries are loaded:

Reading state information... Done

The following extra packages will be installed:

gamin libdate-manip-perl libgamin0 libyaml-syck-perl postfix python-central python-gamin python-support whois

Suggested packages:

mailx fortune-mod procmail postfix-mysql postfix-pgsql postfix-ldap postfix-pcre sasl2-bin dovecot-common

postfix-cdb postfix-doc

The following NEW packages will be installed:

fail2ban gamin libdate-manip-perl libgamin0 libyaml-syck-perl logwatch postfix python-central python-gamin

python-support whois

Second, it asks some questions about configuring a mail server. Third, is seems to start running fail2ban with the default configuration automatically.

To examine the status of fail2ban use:

% sudo fail2ban-client status

Status |- Number of jail: 1 `- Jail list: ssh

This says it is running the "ssh jail" which monitors log files for ssh activity and temporarily blacklists offending IP addresses.

By the default configuration, when an IP address is blacklisted it will be ignored for 10 minutes. If login fails 3 times in a row in quick succession, you may be blacklisted. Since each login attempt allows you to enter the password 3 different times, you actually get 9 opportunities to enter a password before blacklisting can occur.

The config file /etc/fail2ban/jail.local holds local settings. It must start with a directive like [DEFAULT]. After it is changed, you must:

% sudo fail2ban-client reload

fail2ban, probably via logwatch, can send daily emails that recap activity. I'm not sure of those details.

##Results The fail2ban log file at /var/log/fail2ban.log contains:

2016-01-26 23:54:37,579 fail2ban.actions: WARNING [ssh] Ban 183.3.202.112

2016-01-27 00:04:38,221 fail2ban.actions: WARNING [ssh] Unban 183.3.202.112

2016-01-27 00:55:02,011 fail2ban.actions: WARNING [ssh] Ban 59.45.79.39

2016-01-27 01:05:02,709 fail2ban.actions: WARNING [ssh] Unban 59.45.79.39