What do you do if your server has been hacked?

[KapsZ28]

What should I look for that may have been compromised? Whoever accessed my server, installed the following applications on 8/15.

Safari
Apple Software Update
Bonjour
SeaMonkey 2.12
Opera 12.01
Netscape Navigator
Mozilla Mainenance Service
Mozilla Firefox 15.0
Maxthon 3
K-Meleon 1.5.4
Flock
Comodo Dragon

I've always left port 3389 open so I can easily remote desktop to my server. I'll be changing the port. The other mistake I made was leaving the Administrator account enabled. Normally I rename the account to something much harder to guess. Unfortunately I rebuilt the server recently and left the admin account enable. When I did a search for files modified on the 15th, I found that the person was using the administrator account to download and install all these applications.

I have a SonicWALL router, but the log files only show todays date.

Anyone ever see a hack where they install tons of Internet browsers?

 

[mayhem87]

no but have you checked to see if the logs get tar'ed and zipped once they reach a certain limit? i know some firewalls will keep them as backups until its gets replaced. I know this wont help you on the figuring out what happened on your server but at least you might be able to find out where it was coming from.

 

[Brak710]

I would assume your server was acting as a bot to somehow game on advertising systems. Multiple browsers would make it seem Iike more than one unique user.

Reformat and reinstall. A hacked box is a forever questionable box.

 

[tangoseal]

I would assume your server was acting as a bot to somehow game on advertising systems. Multiple browsers would make it seem Iike more than one unique user.

Reformat and reinstall. A hacked box is a forever questionable box.

Agree.... take backups of your data and scan the living shit out of those files.

I would reinstall OS and reconfigure. That is why baremetal backups are absolutely REQUIRED in IT. You could have easily loaded the hour or day before you got compromised and then fixed the issue and would not have to go through all the labor of re installation line item by line item.

 

[Langly]

I would assume your server was acting as a bot to somehow game on advertising systems. Multiple browsers would make it seem Iike more than one unique user.

Reformat and reinstall. A hacked box is a forever questionable box.

Yep, reformat and re-install.

 

[MrGuvernment]

changing your RDC port to something else does nothing, really does nothing, you should run RDC over a VPN instead.

changing ports for services does next to nothing as anyone who wants to get in will scan every port to find what services are hosted on it.

 

[Red Squirrel]

Never leave RDP opened to the public. If you really really have to, change the port at least, but if someone really wants to get in they'll just scan and find it.

When I had to run windows for an online app I had it in a VM on a Linux host with NAT configuration. Only way to connect to RDP was from within the host, so I'd use a SSH tunnel. Much more secure especially with fail2ban installed. Really, any public facing server should have Fail2ban.

As for what to do, it's probably best to rebuild it. The time you will spend checking every possible place to ensure there are no traces will be more than the time it will take to reinstall and restore the data from backups.

 

[YeOldeStonecat]

Up to date with Microsoft updates....and just ensure your Administrator account password is good and strong. Or have the Admin account disabled and have unique additional user accounts with strong passwords.

Only successful RDP hacks I've seen have ground through weak Administrator passwords, and the hacking tool also ground against common user names such as "scanner", "sales", "info", etc.

 

[XOR != OR]

RDP itself is perfectly safe, or rather it is not inherently riskier than ssh. Just as with ssh: a few points to keep in mind:

1) Make sure common account names are precreated and disabled. Change the administrator account to something non-trivial to figure out. If a common account name must be used, then give it a strong password that you rotate every 30 days.

2) If you are fortunate enough to be running a linux firewall, rate throttle RDP connection attempts.

Additionally

3) Secure the channel with SSL, with a cert from a trusted root authority. If you can swing it, use the TS Gateway. Not really required, but does make administrative tasks a touch easier.

 

[YeOldeStonecat]

2) If you are fortunate enough to be running a linux firewall, rate throttle RDP connection attempts.

Can actually do that within Windows too.....right in local policies....XXX failed connection attempts and RDP host will stop answering for XXX minutes. You fill in the XXX.

Someone runs a grinding attempt on you...now even a 4 letter password will take years to figure out.

 

[gangolfus]

Can actually do that within Windows too.....right in local policies....XXX failed connection attempts and RDP host will stop answering for XXX minutes. You fill in the XXX.

Is it possible to do this on a single Windows 7 Ultimate machine? I've got RDP open on my home computer and would be interested in setting this up.

 

[BigJayDogg3]

give it a strong password that you rotate every 30 days.

Why rotate it? If its a strong password today, why wouldn't it be secure tomorrow? If we are talking about brute-forcing a password, the hacker is just as likely to hit upon your new secure password as they are the old one. If the password can be retrieved, they can just as easily retrieve the new password as they would the old. The only thing I could see is if you have someone that had legitimate access to the passwords and no longer does (roommate moving out, employee terminated, etc) so you would want to make sure they haven't copied someone's password. In this case that isn't the concern. Is there some aspect of this I'm blind to?

 

[XOR != OR]

Why rotate it? If its a strong password today, why wouldn't it be secure tomorrow? If we are talking about brute-forcing a password, the hacker is just as likely to hit upon your new secure password as they are the old one. If the password can be retrieved, they can just as easily retrieve the new password as they would the old. The only thing I could see is if you have someone that had legitimate access to the passwords and no longer does (roommate moving out, employee terminated, etc) so you would want to make sure they haven't copied someone's password. In this case that isn't the concern. Is there some aspect of this I'm blind to?

The only thing I would add is that if, by chance, someone does grab the password, then they only have it until the next password change. I'll grant you, these are unlikely scenarios, but then people aren't going to attack the likely targets. Well, and I'm paranoid.

Can actually do that within Windows too.....right in local policies....XXX failed connection attempts and RDP host will stop answering for XXX minutes. You fill in the XXX.

Someone runs a grinding attempt on you...now even a 4 letter password will take years to figure out.

I didn't know that, actually. But I still prefer to have the rate throttling handled externally, by a completely unrelated box. It makes me happy.

Of course, this should be combine with account lockout policies ideally.

 

[Madnes5]

changing your RDC port to something else does nothing, really does nothing, you should run RDC over a VPN instead.

It would very likely stop a worm that took advantage of the RDP protocol.

People need to get over the "security through obscurity doesn't work" mantra. It does work. Just not by itself and as the sole method of protection.

 

[YeOldeStonecat]

It would very likely stop a worm that took advantage of the RDP protocol.
.

Like Morto?
Default port still works fine s'long as you have good passwords and an account lockout policy. Allows one to sleep very well at night.

 

[MekoSuka]

Like Morto?
Default port still works fine s'long as you have good passwords and an account lockout policy. Allows one to sleep very well at night.

My understanding is that Morto scanned tcp/3389 by default. If Morto were to scan all 64000 ports above 1024 it would take a LONG time to scan and it wouldn't be as effective. Let me know if you see this differently.

I like to think that RDP access over the internet is best provided taking a layered security approach especially when it comes to personal use. In business cases you'd likely have other controls (IPS, L7FW's, log correlation etc...) which could alert you to such issues and even shutdown offending connections.

There are myriad of ways you can remotely access your PC w/o paying a buck (i.e. LogMeIn etc...). SSH port forwarding comes to mind.

Here is a typical approach I would recommend to RDP on a home network which is accessible via the public internet in which you don't know how to setup SSH port forwarding.

*Use best practices around Windows accounts and policies.
*Change default listening port to something higher and above 20K
*Filter IP addresses which are allowed to connect
*Where you can't filter IP addy's, use a VPN service provider so you can filter
*Don't want to pay for VPN service provider? Build your own VPN service in Windows
*There is also a known MiTM attack against RDP which if you use TLS you can better mitigate.

Sounds like a lot for sure. Still, using a VPN service and nailing down the IP which is allowed into your network can go a long way is limiting your exposure.

 

[thee_rook]

Take off the tinfoil.
Limit RDP connections to just 1 or 2 accounts. No need t be opened up to the Domain Users group.
RDP is safe as any other remote accessing applications. I have seen ssh exploited in linux installs like CentO/S due to default root account settings. The person installing did not know that the particular distro he was installing enabled access to root via ssh.
You are only as good as your weakest link and every chain has links.
AN open port is an open port and blackhats know how to use every port to get what they want. Never be complacent about your edge. You can re-direct this port, close that port, use this protocol, change account passwords constantly.
This all just makes you a more mobile target but yet you still are a target.

Take off the tinfoil, reduce your footprint as much as possible and still maintain sanity, and watch your packets.

 

[Wrench00]

What should I look for that may have been compromised? Whoever accessed my server, installed the following applications on 8/15.

Safari
Apple Software Update
Bonjour
SeaMonkey 2.12
Opera 12.01
Netscape Navigator
Mozilla Mainenance Service
Mozilla Firefox 15.0
Maxthon 3
K-Meleon 1.5.4
Flock
Comodo Dragon

I've always left port 3389 open so I can easily remote desktop to my server. I'll be changing the port. The other mistake I made was leaving the Administrator account enabled. Normally I rename the account to something much harder to guess. Unfortunately I rebuilt the server recently and left the admin account enable. When I did a search for files modified on the 15th, I found that the person was using the administrator account to download and install all these applications.

I have a SonicWALL router, but the log files only show todays date.

Anyone ever see a hack where they install tons of Internet browsers?

uhm.. What kind of server is this?
If its SBS you should be using RWW.
If its a 2003 use Sonicwall and setup a SSL setup.
if its 2008 install RDWeb or use Sonicwall SSL.

 

[dashpuppy]

uhm.. What kind of server is this?
If its SBS you should be using RWW.
If its a 2003 use Sonicwall and setup a SSL setup.
if its 2008 install RDWeb or use Sonicwall SSL.

shouldn't you be doing trouble tickets ?

sonicwall Or Untangle OR pfsense.

 

[MekoSuka]

Take off the tinfoil, reduce your footprint as much as possible and still maintain sanity, and watch your packets.

You'll have to excuse me as I'm new here but your post is a bit all over the place.

Was the tinfoil comment for me?

Recommending using a VPN rather than opening a list of services is hardly putting on a tinfoil hat. When you further consider that the same VPN can also be used to get at other more risky services you begin to see that the VPN all along reduces the edge footprint. Personally, I'd rather manage 1 entry point into my environment than 10, 20 or more.

I'm not sure where you're going with 'blackhats know how to use every port to get what they want'. Can you clarify this as it's poorly written. An example would help.

 

[Red Squirrel]

It would very likely stop a worm that took advantage of the RDP protocol.

People need to get over the "security through obscurity doesn't work" mantra. It does work. Just not by itself and as the sole method of protection.

Yeah I personally like using some obscurity along with proper methods. Changing a port, in the case of SSH and other apps that produce logs, also helps keep the logs clean of the millions+ of bot attempts per day that you will get.

 

[blinx]

You should never leave your admin account enabled without a strong password. Any attacker worth damn can bypass any clever name you might rename it to. The local admin is always security ID (SID) 500.

 

[thee_rook]

No the tinfoil comment was the which protocol is better security banter.
Tell me the exact reasoning behind RDP is unsafe to use?
I understand VPN as a good security, but most enterprises have it set to auto-run on "field" units at boot so as the person who just go the free laptop from Starbucks this morning, I am in your network without having to put a finger to a key.
So in short I am saying the the only REAL way to ensure your edge is being secured the best way possible is to parse logs, find consistencies, and make the human decision to stop whomever y;all just let scan 65000 ports without alarm as it seems is possible on some of your networks.
Every Edge has holes and every edge has appliances. Do not let complacency let you think you are more secure because of VPN|RDP than the guy who uses porperly, logs, and restricts RDP via a RD gateway or just AD.

 

[Wrench00]

No the tinfoil comment was the which protocol is better security banter.
Tell me the exact reasoning behind RDP is unsafe to use?
I understand VPN as a good security, but most enterprises have it set to auto-run on "field" units at boot so as the person who just go the free laptop from Starbucks this morning, I am in your network without having to put a finger to a key.
So in short I am saying the the only REAL way to ensure your edge is being secured the best way possible is to parse logs, find consistencies, and make the human decision to stop whomever y;all just let scan 65000 ports without alarm as it seems is possible on some of your networks.
Every Edge has holes and every edge has appliances. Do not let complacency let you think you are more secure because of VPN|RDP than the guy who uses porperly, logs, and restricts RDP via a RD gateway or just AD.

RDP is not safe, mainly because its done through clear text. If you use a TS Gateway and encrypt thats a different story.

Only stupid people have an "administrator/admin/root" accounts. Super accounts should always be renamed or disabled to something else and always have a complex password.

 

[MekoSuka]

No the tinfoil comment was the which protocol is better security banter.

Again, you seem to be a little all over the place with your reply so I'll try my best here to answer your questions and comments. I also apologize if this comes across a little harsh/ It's not meant to be. I try to cite examples in each of my responses.

With any secure solution you look to maintain 3 things: Confidentiality, Integrity and Availability. I didn't just make this up. You can find this information in NIST special pubs, CIS Security published whitepapers and standard MS docs which speak to protecting information.

Tell me the exact reasoning behind RDP is unsafe to use?

RDP out of the box lacks full session encryption and is susceptible to a MiTM attack (however unlikely). Depending on what you do with RDP, this may or may not be of concern to you. How do you overcome these weaknesses? There's a multitude of ways of which VPN is one.

I understand VPN as a good security, but most enterprises have it set to auto-run on "field" units at boot so as the person who just go the free laptop from Starbucks this morning, I am in your network without having to put a finger to a key.

I've worked many places and have to say your claims of auto-boot and you're in the network w/o touching a key seem a bit exaggerated. I know for a fact that if a shop is worth its weight in IT, it's not deploying VPN in this manner. Not saying people don't do it, just that it's a bit *dumb*. There is no patch for human stupidity.

Saying

So in short I am saying the the only REAL way to ensure your edge is being secured the best way possible is to parse logs, find consistencies

This couldn't be further from the truth. You're putting all your stock in 'monitoring'. I can tell you from experience that even the $100,000 SIEM products will full blown log correlation miss things all the time, and don't get me started on IPS, WAF, and other signature based prevention systems which can't do DPI to save their lives. Do you really think an IPS log is going to show you an exploit if it doesn't have the signature to flag it in the first place or an IDS is going to detect all packets when it's using a SPAN port to get its informatioN? Are you going to read the 10 billion packets that hit your firewall every day? How much tension do you think the Slowloris DoS puts on your line? It doesn't just jump out at you from a bandwidth perspective.

Putting all your stock in monitoring is not the only real way to make sure you're edge is secure. The best possible way to ensure your edge is secure is to setup the edge correctly in the first place. You do this by employing best practices as they apply to CIA in your environment.

 

[XOR != OR]

RDP is not safe, mainly because its done through clear text. If you use a TS Gateway and encrypt thats a different story.

You don't need to use TS Gateway to get an SSL secured channel. By default, I believe RDP ( service, not protocol ) uses a self-signed cert. But you can get a cheapy SSL cert from a trusted root, and you're good to go.

TS Gateway is nice, but not required.

 

[mngl1500]

I also make administrator accounts be denied access to the terminal server.

 

[MekoSuka]

You don't need to use TS Gateway to get an SSL secured channel. By default, I believe RDP ( service, not protocol ) uses a self-signed cert. But you can get a cheapy SSL cert from a trusted root, and you're good to go.

TS Gateway is nice, but not required.

So I thought I'd share this link to put this all in perspective.

http://tech.slashdot.org/story/12/0...p-vulnerability-should-be-patched-immediately

RDP is safe as any other remote accessing applications

Well that's comforting. The article makes mention of the recent issue with Symantec's PCAnywhere and I know personally of issues in DameWare, another remote admin tool that's FIPS 140-2 certified. VNC goes w/o mentioning HardForum

 

[XOR != OR]

So I thought I'd share this link to put this all in perspective.

http://tech.slashdot.org/story/12/0...p-vulnerability-should-be-patched-immediately

What's your point? Software has bugs/flaws. Good security is comprised of many parts, not the least of which is ensuring a good patch policy.

I had my TS servers tested and patched a day after that advisory went out.

 

[MekoSuka]

I agree with you on good security and patching being part of that.

However, I'm of the opinion (as are many in the industry) that simply sticking RDP out on the net showing its butt at tcp/3389 is begging for trouble.

Determining what to do can be based on risk. It may be fine if you're a home user. All I'm trying to show is the risk associated with that decision. Some may call it a tinfoil hat. I don't believe that one bit.

 

[MrGuvernment]

lots of good info, and in the end it seems to come down to a good I.T Team and people who have a clue or are willing to learn to make things better.

 

[XOR != OR]

I agree with you on good security and patching being part of that.

However, I'm of the opinion (as are many in the industry) that simply sticking RDP out on the net showing its butt at tcp/3389 is begging for trouble.

RD with a TLS transport layer ( default ) is no riskier than ssh, technically. In fact, given the history of each service an argument could be made that RD via TLS is more secure than SSH ( not the least of which because it supports the full ssl trusted root thang ).

Now, certainly, a bad admin can make one less secure than the other. But if we limit our discussion to merely the technical aspects of it, one is no more secure than the other.

 

[MekoSuka]

RD with a TLS transport layer ( default ) is no riskier than ssh, technically. In fact, given the history of each service an argument could be made that RD via TLS is more secure than SSH ( not the least of which because it supports the full ssl trusted root thang ).

Now, certainly, a bad admin can make one less secure than the other. But if we limit our discussion to merely the technical aspects of it, one is no more secure than the other.

I continue to agree with you on SSH vs. RDP or other like protocols. It's also why I stress leveraging VPN as a broker technology when employing remote systems administration from an untrusted network (internet). Surely any reasonable admin would agree that protecting a high profile server should require more than a username and password?

As a service provider that offers TS, this may not be possible but there are other controls in the areas of 2FA which can compensate if password complexities are difficult enforcing around end users.

On a side note, MiTM can still be performed on 2008 RDP sessions if the admin does not take the necessary steps to tighten things up. Out of the box RDP configurations are still susceptible to MiTM. See this MSDN blog for details.

I mention this because of how easy it is to perform MiTM with basic arp poisoning. It's enough to make you cringe when you see users doing work on an open WiFi at a local starbucks..

 

[XOR != OR]

I continue to agree with you on SSH vs. RDP or other like protocols. It's also why I stress leveraging VPN as a broker technology when employing remote systems administration from an untrusted network (internet). Surely any reasonable admin would agree that protecting a high profile server should require more than a username and password?

The benefits must be weighed against the overhead. Cost/setup/maintenance...these all play into it. While we would all like to harden our servers against every known and unknown attack vector, constraints often conspire against us.

On a side note, MiTM can still be performed on 2008 RDP sessions if the admin does not take the necessary steps to tighten things up. Out of the box RDP configurations are still susceptible to MiTM. See this MSDN blog for details.

I mention this because of how easy it is to perform MiTM with basic arp poisoning. It's enough to make you cringe when you see users doing work on an open WiFi at a local starbucks..

Well yes. File that under "Bad Admin".

 

[athlon1.2]

You don't need to use TS Gateway to get an SSL secured channel. By default, I believe RDP ( service, not protocol ) uses a self-signed cert. But you can get a cheapy SSL cert from a trusted root, and you're good to go.

TS Gateway is nice, but not required.

I don't think you can change the certificate unless you use TS gateway. Was trying to do it the other day, but didn't find any option to change it.

 

[MekoSuka]

The benefits must be weighed against the overhead. Cost/setup/maintenance...these all play into it.

Agreed although in 2012 if a business is making an investment in technology cost, setup, and maintenance of an IPsec VPN is not at all difficult to come by. For home users, I can see this as being a formidable challenge (fwd tcp/3389 vs. building a VPN).

I agree a poor admin (poor in terms of knowledge and experience) would be one who uses the default RDP config and self-signed certs. It is unfortunately, the norm.

 

[XOR != OR]

I don't think you can change the certificate unless you use TS gateway. Was trying to do it the other day, but didn't find any option to change it.

You can, but it's non-obvious.

Aside from importing a previously generated cert, you can use the cert tools in IIS to generate a new CSR, ect...

 

[XOR != OR]

Agreed although in 2012 if a business is making an investment in technology cost, setup, and maintenance of an IPsec VPN is not at all difficult to come by. For home users, I can see this as being a formidable challenge (fwd tcp/3389 vs. building a VPN).

You are only looking at up front hard costs ( and are disregarding remote workstation control hard costs at that ). However, there is far more to the landscape then VPN setup costs. Unfortunately, remote support situations can dramatically increase support/training overhead. In situations where you may not be able to access the remote workstation, it's usually better to keep things simple to the point where you can walk the user through support issues.

VPNs are a level of complexity I do not want to talk my user through over bad cell connections.

I agree a poor admin (poor in terms of knowledge and experience) would be one who uses the default RDP config and self-signed certs. It is unfortunately, the norm.

Unfortunately, I must agree with you there.

 

[MekoSuka]

You are only looking at up front hard costs ( and are disregarding remote workstation control hard costs at that ). However, there is far more to the landscape then VPN setup costs. Unfortunately, remote support situations can dramatically increase support/training overhead. In situations where you may not be able to access the remote workstation, it's usually better to keep things simple to the point where you can walk the user through support issues.

VPNs are a level of complexity I do not want to talk my user through over bad cell connections.Unfortunately, I must agree with you there.

I can respect your opinion and thought process on the matter. I don't pretend to understand the business you're in or currently work for. I can only draw from my industry experience and training. I've been supporting VPN's for the better part of 15 years even when I had users who were predominantly based in 3rd world countries. I've not found the issues you speak to large in nature.

As for my overall picture and how I come to the conclusion of leveraging VPN's, I look at far more than up front costs. Supporting a VPN can bring its own set of challenges. However, those challenges are just another tick box in a risk assessment. What you're missing here is that it shouldn't be up to me or you to decide whether or not we want to talk a user over a bad cell while supporting VPN . It should be up the the business and identifying what level of risk they are willing to accept. Bad cell connections are a risk with a risk likelihood that is probably lower than you think. Your business type would better define both of these however. Just assign it a rating and move on. Then offer the business options and know where you stand if regulatory compliance or local regulations apply.

Circling around back to the original question this post raised, it's clear that incident response is something that is missing structure-wise. While there are certain steps you can take to clean up a hack I believe a complete rebuild in this case seems warranted especially when it appears there is no baseline by which you can compare. Along with the myriad of other suggestions made, I'd also encourage the use of HIDS on the box in the future since it's clear it's a target.