A few days ago, I had my dedicated scraper instance on Vultr shutdown for sending too much outbound traffic which ressembled a DoS attack.
I assumed that they were detecting my scraper so I spent my time arguing instead of diagnosing. I ended up convincing management that what I was doing was okay - it was not: My instance was doing a SYN Denial of Service attack against a multitude of other servers.
Resistance was futile
At first, I operated on the assumption that the highest CPU consumer was the virus. I was correct. It cloaked itself as if it were a gnome terminal. Odd, I didn’t install anything gnome-y on servers. I later traced it to randomly named binaries. I tried analysing one but it deleted itself. I backed up the other one before deleting it.
As an extra precaution, I started dropping the malicious traffic by outgoing ip. This fixed everything - or so I thought.
Hours later it resumed: new binaries, new IPs… My
nload screen went crazy.
This was bad, really bad.
I realized what I was faced with; a game of cat and mouse, and one that I wasn’t going to win.
Destroy Server? Get outta here!
The server was stripped down to essentials. I may never figure out how it got hacked in the first place but the stripping down took a few hours of my life, wasn’t going to redo it. Instead, I hardened the server by dropping all the traffic except a few IPs using iptables.
# allow google dns iptables -I OUTPUT -d 18.104.22.168 -j ACCEPT iptables -I INPUT -s 22.214.171.124 -j ACCEPT # maintain ssh access on port 22 iptables -I OUTPUT --dport 22 -j ACCEPT iptables -I INPUT --sport 22 -j ACCEPT # install fail2ban because I'm not insane apt install fail2ban -y # drop all the rest iptables -I INPUT DROP iptables -I OUTPUT DROP
By the last command, everything became peaceful - bandwidth dropped, CPU usage stabilized and I relaxed. All became good: the botnet had lost control.