Possible Security Issue - Advice Welcome

[DistributedBen]

For a few weeks I have been trying to track down the reasons for our primary file server running low on storage at an unusually high rate. Many times it has gone below 1GB of free space. At first I figured it was due to large project files being saved to the server.

However, this week I'm thinking it is something else. While I was gone on Tuesday the 6th - the server parition with the files filled up. This also caused other services running on that parition - mainly Exchange '03 to shut down.

When I left work the day before we had almost 1.5GB free. I had twice that free the previous Friday. I asked around to see if anyone had dumped files on the server - nope.

After tracking the storage growth for the home directories over a few days, they never grew by more than a hundred or so MBs during this week. But I was still running out of space - down to less than 400MB yesterday at lunch.

There are other folders on the partition but they are non shared or only I have access. I saw no change in their sizes. The Exchange folder (with the db) was holding steady at 8GB which is normal.

After everyone went home yesterday I decided to take a closer look. Much to my suprise I now had 23GB free. How the %$@ did that happen? In less than 5 hours and during the workday 23GB of space had been freed up.

I paniced, thinking the worse that if it was a security issue, they may have deleted a company data. After all, I knew we were low on space, but I didn't know how big a gap there was between our files and what was taking up all that room. Luckily, I checked and the home directories don't apear to have been touched and Exchange is running as usual. In fact, I have no idea where the free space came from anymore that I knew what was taking up all the space.

I figure that an outside source was using the server to route junk mail or possible a file server for files. But I haven't (yet) found any services that shouldn't be running. There never seemed to be a processor or memory drain on the server, just disappearing space. I hadn't had time to change anything either, it just all came back.

So today, I'm scanning the logs/server, etc to see if there is anything out of the ordinary. One thing that has aleady caught my eye are repititive and successful login attempts from a remote location to one of our employee accounts during the early morning hours.

Unfortunatly this server does everything (AD/DNS, Exchange w/ WebAccess, File/Print). I need to get Exchange and OWA off this box.

Any ideas and suggestions on how to go about this? The confusing part is that it just stopped. I ran filemon and nothing out of the ordinary came up last night. I will compare it to the previous nights capture.

 

[xphil3]

How is this box being protected, is it behind a firewall? Have you checked your firewall logs to correlate the remote successful login attempt? What kind of login type is it? Which services are you setting rules for in your firewall?

Unfortuently I don't know much about exchange(im a sendmail guy), but have you checked any of your mail logs? Do you have authentication for relay configured(this is a common problem for most mail servers these days, and then the admins wonder why they eventually get BL'd).

I seriously hope that you have some sort of backup procedures if your fileserver houses that many services, especially if your company needs to meet SoX com plianc.

 

[Jay_2]

you seriously need to watch this server. I had a bad security issue and the only way I sorted it was sitting with the system for 24 hours and waiting.

Keep a watch on the space in real time... look for bandwidth usage at stage times, check all users with VPN access and change the passwords (wait for the user to call you to have it changed to a new password), look at your firewall for gaps if all else fails get yourself a serious firewall, cisco PIX or even a Linux IP Cop style firewall.

things like that are normally users with wrong access rights downloading crap, 90% of network problems / security issues are from people on the inside. Its strange that as soon as you ask around the space jumps back up. Did some one realise your on to them and remove their crap?

and wow... all those services on one box!!! my god how do you sleep at night???

 

[Boscoh]

Rule #1 in dealing with compromised systems:

You can no longer assume the system is secure, even after you think you've locked out the intruder and patched up the hole. There are so many ways to install backdoors into a system.

You need to get the data off this server that you want to save, and wipe the hard drive, and reinstall everything again. It sucks, but this is the only way to ensure that a system is no longer compromised.

You need to investigate how this person got in remotely. I'm guessing probably an exploit via whatever remote access method you have opened or a weak password. I would force every user in your company to change passwords. I believe in Active Directory you can enforce strong passwords and force passwords to be reset every x number of days. I'd implement the strong password policy now (consisting of numbers, upper and lower characters, and one special character) and force them to change every 90 days. You have to assume that whoever got access to your system both compromised every other account password and installed a backdoor into the system.

Three things can explain the sudden stop in what you're seeing: 1) the person doing it went on vacation, 2) it was a hit and run, it's common for people to find easily exploitable systems and use them as an FTP server for warez or kiddie porn, 3) the person knows you've found them and has either left, or is playing it cool and waiting for you to think they're gone.

 

[nessus]

What backup software are you using? I've seen the advanced file open option on Veritas backup exec cause space issues like you ar reporting.

Also, do some reading on alternate data streams on NTFS. It's possible to attach data on to files so that you don't see the increase in file size in Explorer, but the the free space is reported correctly. If you run a command line chkdsk in read-only mode and it reports a different data amount than Explorer, then the data is being stored in an alternate stream.

In my case, the backup program was using an alternate data stream on a file to work with its processing of backups in a locked status and the program got stuck in a loop. We were losing disk space at 1.2 gigs per hour until we found it. It had apparently been having issues for about 12 hours before we got a low disk space flag that sent us investigating. Stopped the backup service and 20 gigs of disk space reappeared instantly. Freaky stuff.

http://msdn2.microsoft.com/en-us/library/ms810604.aspx#ntfs5_topic3