Getting cracked

Introduction

The following is an account of how the server I run, and the one that runs the Admin Mod forums, got cracked. Now, you may ask why I say "cracked" rather than "hacked". Well, its a long story which you can read here.

One fine morning

It all started on the morning of the 14th of August. I was merrily checking my email and enjoying my morning coffee before I set off on my bike ride to work. I had a couple emails from the rest of the Admin Mod team questioning the performance of the web server. We had been having problems due to only having a dual T1 for all the traffic that our servers attracted. However, a couple days earlier the link had been upgraded to an OC-3 fibre connection, so I was curious as to what the problem could be.

I ssh'd (secure shell, an encrypted command prompt of sorts) into the server (as usual) and checked what the load was. As the logged in user list scrolled past my window I noticed a user that I didn't remember adding. Our servers have very few shell users, which are only used to maintain the servers. So I know everyone who has login access, and the person on there I didn't know! Next I checked what processes were running. Ahh, "BitchX" will up and running. BitchX is an IRC client for unix machines, and it seems my wayward user was surfing the chat "highway".

So, I had a user that wasn't meant to be there, what did I do? I went into panic mode is what I did! My first move was to kill the ssh session of the logged in user, which effectively logged them out. Thinking back, this wasn't the smartest of moves, it alerted the cracker as to my knowledge of his intrusion. If they had been agressive or knowledgable my server could have been trashed in seconds. Luckily, the crackers were neither agressive or knowledgable. All they did was keep trying to log back in. So I just kept killing off their ssh sessions.

At the same time I was busily shutting down every service on the machine trying to stop them getting back in. At this stage I didn't know what the entry method could be, so I was busily killing off services as fast as I could . This didn't help the problem however, they kept getting back in. I then took a look at the /etc/passwd file and noticed some new accounts as well as some (poor) attempts at hiding backdoor accounts. I had only one option left to protect the server (remember that I didn't know who I was dealing with at the time), to shut it down. Shutting down the server would mean all the websites would go down, my email would stop and DNS would fail. But the only other choice was to let these guys keep getting back in, until one time they get grumpy and trash the server. So it was off for the machine.

The Aftermath

Little was I to know how long it would take for the machine to come back online. CFGN (the group who runs the server, which I am a part of) has its server telehoused in Texas, America. I live in Canberra, Australia. See any problems yet? The Internet in Australia is in a sorry state with charges based on bytes transfered rather than the speed of the pipe. On top of that we pay through the nose down here for access, so it was uneconomical to run the server with an Australian host. So our fine friends at clanshack came to our rescue and offered us hosting over in Texas. However the gentlemen with physical access to the server live over an hour away from it, so its not trivial to have them reboot it.

There I am with a shutdown server and with no idea of what had been done to the data on the hard disk. The crackers had undoubtably installed backdoors on the machine so they could get back in, so the only option to getting the machine back online was to reinstall the operating system from scratch. That required a new hard disk which meant another delay.

A week later the machine finally came back online, with the help of the great guys at clanshack (Zep in particular). With the machine online I began the task of rebuidling the web servers data and the email system. Luckily the new operating system had been installed on a new hard disk so the old (compromised) disks could be left in their pristine state for later analysis.

A couple of hours later the servers were back up so it was time to sit down and work out what happened.

Forensics

Whoever had got into our machine had stomped around making a mighty "digital" racket. They first enter the machine on the night of Sunday August 12th. They got in via a telnetd buffer overflow expliot, one that we had failed to patch. From there they got sloppy. I found a copy of the source code to the telnetd exploit on the machine, they were obviously try to crack into more machines from ours. They also installed BitchX onto the machine and were logged into some IRC servers. One of them was in the process of installing "screen" onto the machine, probably so they could "detach" the BitchX clients from their login sessions.

Apart from running BitchX and installing a couple of trivial backdoors they thankfully didn't do anything else. One of them attempted to install a rootkit designed to Linux, but this was doomed to failure as our servers run FreeBSD (I have no idea how they could make this mistake...)

The one thing they did do was leave their ugly fingerprints all over the server. I have an extensive log of each users activity and the locations they logged in from. By doing a quicky bit of detective work I have even found photographs of the people who broke in. Not only are they careless, they are dumb enough to write about themselves on websites.

Conclusion

I have informed the Universities and another couple of ISP's which they have compromised and hopefully there parents will tell them its wrong to break into other peoples property, whether its physical or digital. An interesting side note to what they did is in the telnetd buffer overflow exploit code they used. Here is an excerpt of the license contained in the code:

 *
 * This is unpublished proprietary source code of TESO Security.
 *
 * The contents of these coded instructions, statements and computer
 * programs may not be disclosed to third parties, copied or duplicated in
 * any form, in whole or in part, without the prior written permission of
 * TESO Security. 

It also seems that one semi-compotent cracker made it into the box, and they then gave out accounts to try hard friends. So choose your friends carefully, they may become your downfall.