Once You're Hacked, You Never Go Back

August 2017 ยท 2 minute read

A few days ago, I had my dedicated scraper instance on Vultr shutdown for sending too much outbound traffic which ressembled a DoS attack.

I assumed that they were detecting my scraper so I spent my time arguing instead of diagnosing. I ended up convincing management that what I was doing was okay - it was not: My instance was doing a SYN Denial of Service attack against a multitude of other servers.

Resistance was futile

At first, I operated on the assumption that the highest CPU consumer was the virus. I was correct. It cloaked itself as if it were a gnome terminal. Odd, I didn’t install anything gnome-y on servers. I later traced it to randomly named binaries. I tried analysing one but it deleted itself. I backed up the other one before deleting it.

As an extra precaution, I started dropping the malicious traffic by outgoing ip. This fixed everything - or so I thought.

Hours later it resumed: new binaries, new IPs… My nload screen went crazy. This was bad, really bad.

I realized what I was faced with; a game of cat and mouse, and one that I wasn’t going to win.

Destroy Server? Get outta here!

The server was stripped down to essentials. I may never figure out how it got hacked in the first place but the stripping down took a few hours of my life, wasn’t going to redo it. Instead, I hardened the server by dropping all the traffic except a few IPs using iptables.

Example:

 # allow google dns
iptables -I OUTPUT -d 8.8.8.8 -j ACCEPT 
iptables -I INPUT -s 8.8.8.8 -j ACCEPT
 # maintain ssh access on port 22
iptables -I OUTPUT --dport 22 -j ACCEPT 
iptables -I INPUT --sport 22 -j ACCEPT
 # install fail2ban because I'm not insane
apt install fail2ban -y
 # drop all the rest
iptables -I INPUT DROP 
iptables -I OUTPUT DROP

By the last command, everything became peaceful - bandwidth dropped, CPU usage stabilized and I relaxed. All became good: the botnet had lost control.