Herocraft Plugin Devs being DDOS'd by Brazilian server

K

Kainzo

Gawd
Joined
Nov 12, 2009
Messages
972
Below is a brief explanation of what is happening and why we are being attacked. If you don't know of Herocraft Development, we produce Bukkit/Spout addons.

Here's a list of plugins we develop:
http://dev.bukkit.org/server-mods/vault/
http://dev.bukkit.org/server-mods/herobounty/
http://dev.bukkit.org/server-mods/herochat/
http://dev.bukkit.org/server-mods/heroes/
(Downloads totaling over 1,000,000)

http://herocraftonline.com has been target'd because we wouldn't push out Herochat 1.3 fast enough for this server. The user Spunkiie, aka the owner of http://CraftLandia.com.br has said he will not stop DDOS'ing Herocraft until we are less than a 10 person server.

I have brought this to the attention of the developers and the community of Bukkit and they have all been very caring about what has been going on. The response from the Bukkit community has erected a large retaliation against them - after they tucked their tail - they stopped the DDOS last night and requested a truce.

Today, Evilseph (Mojang/Bukkit Lead), has removed all names / information that CraftLandia is behind this because "they dont want to be involved" so CraftLandia / Spunkiiez have resumed ddosing our services without fear of being beaten back by a healthy / thriving community. They are taunting us on IRC - spunkiiez has been network banned from esper IRC 4 times.

From the server admin side, I am working as quickly as I can on a solution. It means our server costs will increase 5-6 times the amount they are now (very cheap, $200 at this time but will be easily over $500)

(For Herocraft players and plugin users)
I don't like leaving you in the dark here. I want to let you know whats going on. Thank you for your patience in these times. I am quite unhappy by the outcome of this and will fight tooth and nail with every dollar Herocraft has to keep us up and thwart these injustices.

Below are quotes that the Bukkit team removed and had deleted because it was "offensive" to the server in question. They are a compilation of the harassment we have received in PM's, other emails and things that this server and person have done. They told us to take down this information or the DDOS's will not stop.

Update #1 - In response to these threads being created, CraftLandia/Spunkiie has target'd my home internet, the place where I work out of the home 2/5 days of the week. I am now in the office to post this.
http://forums.bukkit.org/threads/treat-plugin-devs-better.91128

http://forums.bukkit.org/threads/herocraft-dev-being-ddosd-no-plugin-updates.91705

Well, I thought it was something just childish, because we wouldn't 'jump' to one person's whiney attitude fast enough. This person [spunkiie, admin of CraftLandia] is now targetting our users and our main server (Herocraft).

The user in question admits to ddosing our services and laughs about it.
http://pastie.org/private/cpfu89abmjraug03uyjzya
Is the combined log of all that was said/done - the previous thread we created was set to private and locked.
More info of the user - IRC chat logs..
http://pastebin.com/2azysbST
http://pastie.org/4446456
Emails of him being "superior" over other wanna-be server owners/devs
http://pastie.org/private/rbivfj5zgs93kjgd452x3a
Pm of one of the "admins" inviting me to their team 6+ months ago
http://pastie.org/private/pz5yfnt1br4vbsfzl2joa
Random users offering help in any way to stop this tyrant.
http://pastie.org/private/3lenesags77hly1i63w
We believe hes running a slow-bot attack, its enough to steadily knock out our services every 5-10mins to create chaos in the userbase. He has increased the DDOS to our website, our TS3, our repo/build server and our test servers.
We are a non-profit organization and a non-profit development team. We don't have a lot of cash to fling around for anti-ddos protection and we can't develop anything if our build servers / test servers are offline. I felt it necessary to spread this to as many people as possible. Our resources are limited but our friends are many, I feel that the work we have pushed out in the last two years more than covers the amount of resources we'd need to thwart this attack.



reposted:
http://www.minecraftforum.net/topic...opment-ddosd-by-brazilian-craftlandia-server/
http://www.reddit.com/r/Minecraft/comments/xzf4m/herocraft_plugin_devs_being_ddosd_by_brazilian/
http://herocraftonline.com
 
Last edited:
I have no idea what you're asking for, but I work in Datacenter Operations during the day dealing with DDOSs at times, build custom routers/firewalls at night, and double as a minecraft server dev/admin during night-night.

I can help you with Snort rules and server/firewall configs, but if you're being DDOS'd to 100% of your link speed, you need to be talking to your provider or throw bandwidth capacity at the problem.

I see you mention a slow loris attack... Do you mean you're being slow loris'd or it's a similar low-bandwidth attack? If so, that's an an easy fix with proper firewalling. If you have no idea what is doing on, can you give us some packet captures so we can find it?

There are so many ways to protect against this. I fortify all my servers even before I get DDOS'd just incase. This doesn't even need a huge amount of cash, it's just configuration. Hopefully you have hardware you can run something like pfSense on? If not, I'm sure we can work with something...
 
Last edited:
Sorry, been busy for most of the day at work. During the day I do systems support :) Unfortunately, the box that was being DDOs'd is being shipped back to Dallas, TX for repairs - the HDD failed at the same time of the DDOS.

My new box however is now at oplink.net and I'm looking to fortify against attacks.
 
You need to first identify what type of attack it is and then move from there.

You haven't given any real specifics in your posts about what assistance you require if any..
 
Only way to fortify against DDoS is to put a large enough pipe in front of the connection to mitigate the brunt attack, and handle the rest with firewall/QoS on a border router or your server. If you're just being attacked from one host (which is a DoS, not a DDoS) you can just contact your uplink provider and have them null route it.

If the attack exceeds 10gbit, good freaking luck.
 
Fully saturating a decent physical link is one type of DDoS, there are many others.

We won't know until we get details.
 
Alright. I am back - it was a very long day sorry for the delays.

Basically - this involves several different facets of our websites.
Website (Herocraftonline.com) hosted/rented from a hosting company. This DDoS has mostly been mitigated by cloudflare.com and having the host throw some more bandiwdth. They took it down 3-4 hours but after Cloudflare has been placed its golden.
TS3 - same as above, on the same box.

Minecraft Hardware 1 (main) -
http://206.212.246.7/mrtg/206.212.240.134_42-week.png
(Hardware is being shut down for Minecraft Hardware 2) This has been the primary target. It's very strange what they've done here - at first our connection was being throttled by our data center (colostore.com) but after they "fixed it" the DDoS went from 10-14Mbps to a straight 25-30Mbps hit. Users could connect fine for about 3-5 mins and then they would drop suddenly. I was able to find anyone using the netstat command in Debian. No erratic users.
We also enabled syn-cookies for the kernel and it didn't seem to find anything.

Minecraft Hardware 2:
http://i.imgur.com/p5r2x.png
Is the new host we're moving to, I won't release the ip, though it appears that they have already found it. I'm seeing a constant 50Mbps with 3 users online - it isnt hard crashing our machine like it was on Hardware 1 - but this machine is also newer and has some better stability than the other with a custom $200 NIC in it / $500 raid card.

I'm more than happy to retrieve logs while we're being ddos'd on this test server - Thanks for listening and I'll be gathering more info as I can. We were DDOS'd on our home connection today so it took a few hours to resolve that with the ISP.

I'm going to be taking a break and sleeping - if you have any ideas of what to run and what to do - lemme know
 
I don't know why you spent $200 on a NIC and $500 on a raid card for a minecraft server. You only need some SSDs in redundancy pairings to hold the world folders.

The NIC is almost meaningless, a modern server with a built in Intel NIC would probably just do as well.

You need to setup some proper firewalling, though. Can you get a box to run pfSense on? Heck, even iptables on your box might be enough with something like:

Code: 
sudo iptables -A INPUT -p udp --dport 25565 -m state --state NEW -m limit --limit 50/minute --limit-burst 200 -j ACCEPT

What are your link speeds by the way, are those DDOSs maxing them out? This just seems like insignificant amounts of traffic to me, I've been hit by 75mbps DDOSs at home on my minecraft server and we never dropped a player. You either have a special attack, or a terrible misconfiguration somewhere.
 
Last edited:
Post on 4chan with relevant technical details.
Pray and hope CraftLandia gets DOSS'd beyond existence.

:D

and gets hacked and trashed to death by Anonymous

*trollface*
 
Brak710 said:
I don't know why you spent $200 on a NIC and $500 on a raid card for a minecraft server. You only need some SSDs in redundancy pairings to hold the world folders.

The NIC is almost meaningless, a modern server with a built in Intel NIC would probably just do as well.

You need to setup some proper firewalling, though. Can you get a box to run pfSense on? Heck, even iptables on your box might be enough with something like:

Code: 
sudo iptables -A INPUT -p udp --dport 25565 -m state --state NEW -m limit --limit 50/minute --limit-burst 200 -j ACCEPT

What are your link speeds by the way, are those DDOSs maxing them out? This just seems like insignificant amounts of traffic to me, I've been hit by 75mbps DDOSs at home on my minecraft server and we never dropped a player. You either have a special attack, or a terrible misconfiguration somewhere.
Click to expand...

Hi, we have iptables set up - and a switch in front of our machine at the datacenter. We have 100Mbps full duplex running. What does that command for iptables do exactly?
 
If you have money, consider DDoS proxy services such as Prolexic or BlackLotus. They basically have a lot of bandwidth, you point your web servers to them, and they take the brunt of the DDoS, and pass on clean traffic to your servers.
 
Kainzo said:
Hi, we have iptables set up - and a switch in front of our machine at the datacenter. We have 100Mbps full duplex running. What does that command for iptables do exactly?
Click to expand...

Roughly, if a source tries 200 connections in a minute, it gets limited to to 50 connections a minute maximum.

That rule would need tweaked for the traffic of your server, I would need to know server slot count, and if any traffic besides the normal minecraft traffic will be expected.

That 200->50 rule is more for webservers, so you might want to try less, but maybe you'll need more. It would need tested. I can build a lab and test this, but I need to know what the exact attack your getting is.

Is the server hardware choking, the FA port getting flooded, or is that attack killing java/bukkit?
 
Last edited:
hawk82 said:
If you have money, consider DDoS proxy services such as Prolexic or BlackLotus. They basically have a lot of bandwidth, you point your web servers to them, and they take the brunt of the DDoS, and pass on clean traffic to your servers.
Click to expand...

Some ISPs (such as the one I work for) can also offer network-based-firewall services that will take the brunt of a DDoS for you.
 
If your link speed is not maxing out your ports/bandwidth allotment, your issue is configuration and software.

You do not need money to solve something like that, just knowledge and time.
 
(FYI, I do think this is an interesting thread and am paying attention to it. This area of networking/security is in my field of interest.)
 
aronesz said:
(FYI, I do think this is an interesting thread and am paying attention to it. This area of networking/security is in my field of interest.)
Click to expand...

If we can get some details on this, we can go even more in-depth.

I understand the OP is busy, but I would hope we could save him and his community some donations by not having to throw money at the problem.
 
Alright. I've been gathering information about what's going on. We have successfully thwarted the DDOS on our website (Herocraftonline.com)
The DDOS'ers are attacking the DNS (not single ip's) Of our services. Here's a few they are targeting.
hco.zapto.org
herocraft.zapto.org
mc.herocraftonline.com

It is apparent that they are ONLY going through the MC port 25565 - when I shut down the minecraft process for a server running 25565 - the DDOS ends. (Note this port is TCP not UDP)

Here are the technical connection details using "iftop" in console. It's pastie but you can still see whats going on and the massive flood of ips and the LOW packet count.

Here's a iftop of when we aren't getting DDOS'd
http://pastie.org/private/3dumapwcmgj8idczllduuw

Here's one when we are ddos'd (Mass logs at the bottom)
http://pastie.org/private/jikksafwlrlrrayxbxs2q

Sorry for the delays - I run a massive community and I'm attempting to play damage control and get services up where I can.
 
Is there any way you could dump the state tables for me (in it's entirety)? A packet capture replay would be nice.

Given the extremely small amount of traffic from each IP, I feel this attack is really hurting the Bukkit jar more than anything. I would assume the attack is draining ticks away from the server.

Does java.exe CPU spike on a core when this ddos happens?

If that is basically how the attack works, we're going to need to cook up a rule that only lets a state survive if the minecraft server starts talking back because it's a legit client.
 
Do you mind hitting me with the cmd I need to do to get that for you?

I have a network admin whos very MIA at times and I'm not exactly sure what you want and how to get it. I know my way around Debian but some of this stuff I haven't done.
 
Well, basically setup pcapdump to watch your interface and record all traffic to a file.

As for dumping the state table, "netstat -a > netstat.txt" should get them all? I can't even remember, lol
 
Last edited:
I installed tcpdump and ran a query on our port 25565, when the ddosing wasnt happening and when it was, here are the raw files (in txt)
http://www.mediafire.com/?ste1spvz8sg2n81

using wireshark it shows that they are spamming UDP, ACK and SYN - 95% of these packets are being filtered at the NIC (fragments arent being accepted and its overloading the NIC card on our box)
 
In the DDOS log, <Snip: Kainzo's IP> is the attacker, and 216.230.231.122 is your server, correct? <Snip: Kainzo's IP> seems to be doing the most talking out of everyone, but was that attack really enough to harm the server by itself? A packets/connections per second rule would stop this guy dead on most routers.

What's the reason behind the EC2 box at Amazon connecting? Part of your server?

Basically, just running "(((!(ip.src == <Snip: Kainzo's IP>)) && !(ip.dst == <Snip: Kainzo's IP>)) && !(ip.src == 50.19.140.154)) && !(ip.src == 216.230.231.122)" in wireshark as a filter seems to clear out all the suspicious traffic.

Also, what NIC are you using, and how are you sure it's being overloaded versus java choking over Minecraft dealing with the packets?
 
Last edited:
Just going off of what my network admin / linux guy is telling me.

I'll have to look up the NIC for you, it was around $100-200 on newegg.

Also the ip listed is safe and the other is the server.
 
Kainzo said:
Just going off of what my network admin / linux guy is telling me.

I'll have to look up the NIC for you, it was around $100-200 on newegg.

Also the ip listed is safe and the other is the server.
Click to expand...

<Snip: Kainzo's IP> is a good guy?

Are you sure?
 
Last edited:
Also, what NIC did you buy? I can't really see paying $200 for even an Intel server NIC unless it's 10Gig, they usually are like $100 or so for the best 1Gigs.
 
So I went ahead and bought a protected line, it protected up to 10TB of clean traffic. Here's what they said

As of 8/15/12 @ 8:30AM the DDOS has not stopped or slowed down.
"You are currently getting attacks in the amount of 954.8Mbps , SYN / ACK attacks and also UDP. "
 
Kainzo said:
So I went ahead and bought a protected line, it protected up to 10TB of clean traffic. Here's what they said

As of 8/15/12 @ 8:30AM the DDOS has not stopped or slowed down.
"You are currently getting attacks in the amount of 954.8Mbps , SYN / ACK attacks and also UDP. "
Click to expand...

Welp. Nothing we can do then on the server front. A maxed port is a maxed port.

You either need to work with that host on getting that null routed (they should be doing this with your protected line, no?), or get a 10gig line and throw money and bandwidth at it.
 
Brak710 said:
Welp. Nothing we can do then on the server front. A maxed port is a maxed port.

You either need to work with that host on getting that null routed (they should be doing this with your protected line, no?), or get a 10gig line and throw money and bandwidth at it.
Click to expand...

Right, it's not often that this size of an attack is held for 7+ days. This started on Monday and was brought to light on Thursday (After much trial and error we couldnt resolve on our own)
 
Kainzo said:
Right, it's not often that this size of an attack is held for 7+ days. This started on Monday and was brought to light on Thursday (After much trial and error we couldnt resolve on our own)
Click to expand...

Is your host working on it, or are they basically admitting defeat?
 
You must log in or register to reply here.
Back
Top