Stop 10 Gbps of DDoS?

[muad_dib]

Hello Guys,

a friend of mine asked me to help him with a very nasty issue.

He has 2 x 10 Gbps Fiber connection to the local upsteam.

This connection was already in place when he entered the building, as he use no more than 1 gbps of bandwidth.

Recently he started experiencing DDoS in the range of 8-9 Gbps. It is not enough to fill his BW, but it is much more than needed to fry his firewalls.

He is looking around for commercial solutions, but I hope I can convince him to go opensource, possibly pfsense. Eventually we could make up for the added complexity and less support of open source solutions by hiring a part time admin fully dedicated to this issue.

The idea is to layer several x86 machines to gradually filter his traffic until only clean traffic reaches his network. Each machine use a different set of rules, so that we can reach hopefully an high enough PPS ratio,

I was thinking about using this hardware:

Sandy Bridge EP E5-2643 (less cores but more Mhz)

64 (96 ?) Gb of ECC RAM

4 x Intel 120 (300 ?) GB SSD

Intel motherboard

Intel Fiber NIC x 2

Do you think that by layering several machines we could reach enough throughput?

What would be the best configuration?

I thought about placing 2 machines directly attached to the upstream, each with 8 (4 ?) VM instances of pfsense, with load balancing and failover provided by CARP, with another machine in the back doing the last filtering of the traffic.

Any other idea?

Am I crazy/dreaming, or do you think that this is feasible?

 

[hawk82]

What kind of DDoS? Has his provider been helpful in mitigating the DDoS?

 

[extide]

I tend to say run pfsense on bare metal, and use lower end stuff, you don't need that much horsepower but can stay with xeon if you want that level of platform, just go with a low end e3 and look into squid caching, it may or may not be of use in your environment. I just setup my in laws with a pfsense box built from their old desktop that does caching and helps them out pretty good.

 

[muad_dib]

What kind of DDoS? Has his provider been helpful in mitigating the DDoS?

I wish I could reply, but I won't be able to have a deeper look until next week, if this can help any.

From the little forsenic I collected, it seems like mostly TCP SYN flood attack, but the attack has changed pattern, once he has been attacked from a single dedicated server of a famous company (obviously hacked) which had 1 gbps of bandwidth.

The attacks happens mostly on the weekend. The ISP is slow during the week, in the weekend he is simply unreachable. Anyhow I've been told that in the future the situation will increase dramatically, as this ISP has being bought.

I tend to say run pfsense on bare metal, and use lower end stuff, you don't need that much horsepower but can stay with xeon if you want that level of platform, just go with a low end e3 and look into squid caching, it may or may not be of use in your environment. I just setup my in laws with a pfsense box built from their old desktop that does caching and helps them out pretty good.

do you think low end hardware can handle millions of packets per second?

 

[toast0]

Need way more info, but here's some things to think about.

!!!!) like hawk82 said, contact upstream provider; they may be able to help in a big way. If you're flexible enough, they can often block the abuse before it gets into their network, which helps them and helps you.

a) for the love of god, don't try to do high performance in a VM. You'll be able to do more (of whatever you're doing) as one machine with real hardware than 8 or 4 VMs with fake hardware. VMs are useful for sharing resources between things that need to be kept separate.

b) you want parallelism, not serialism. Split the traffic from the upstream box to multiple machines somehow. The simple way to split traffic is to use multiple ips, but you can split traffic to the same ip too (if the upstream equipment cooperates, or you can get one box to handle enough traffic to divy it up to others)

c) you might need lots of ram, but maybe not; I would be surprised if you need lots of I/O. Most firewalls barely have storage.

Things that would help us help you. What does the legitimate traffic look like, what does the abusive traffic look like. How does legitimate traffic know how to contact you.

 

[muad_dib]

!!!!) like hawk82 said, contact upstream provider; they may be able to help in a big way. If you're flexible enough, they can often block the abuse before it gets into their network, which helps them and helps you.

Ok, but since we can handle the volume of traffic, being able to be self-sufficient would be great.

Until the ISP fixes some internal turmoil we are on our own.

b) you want parallelism, not serialism. Split the traffic from the upstream box to multiple machines somehow. The simple way to split traffic is to use multiple ips, but you can split traffic to the same ip too (if the upstream equipment cooperates, or you can get one box to handle enough traffic to divy it up to others)

I have to disagree on this one. Multi-level filtering is used by most of the players in this market.

c) you might need lots of ram, but maybe not; I would be surprised if you need lots of I/O. Most firewalls barely have storage.

SSDs are for uptime

Things that would help us help you. What does the legitimate traffic look like, what does the abusive traffic look like. How does legitimate traffic know how to contact you.

It's the site of a local newspaper, with a bit o media streaming. The profile of attacks change very often, so he needs to be fully covered. Maybe a d

 

[toast0]

Do you have SYN cookies enabled? Not sure how well that scales, but it's a decent first line defense. Also make sure you're using accept_filter on freebsd or TCP_DEFER_ACCEPT on linux so clients that don't send any data aren't wasting your application resources (will still waste kernel resources )

 

[toast0]

Web server lockdown:

a) load balancey thingie (or round robin DNS) to split traffic

b) async I/O http proxy server, something like squid* (easier to configure, but multiprocess is new) or Apache Traffic Server (harder to configure, but multiprocess is stable).

Use this layer to make sure clients follow basic rules, like only using GET and HEAD on resources they shouldn't POST to. Requests aren't too long, etc. May be able to do ip limiting here too. Don't forward requests until client sends the whole thing, if client is trickling request, close them down.

c) maybe a caching layer, if your pages are cacheable (could be integrated into b, depending on resources)

d) your regular webserver for dynamic content which now doesn't have to wait for slow clients (in either direction). maybe ip limiting here

d') your other webserver for static content (css, js, streaming media?)

e) backendy stuff (pretty isolated from users now)

Depending on the ratio of work, you might want a tier of proxies in front of your webservers, or it might make sense to put it on the same machine as the webservers and just get more webservers.

The key thing is that squid or traffic server will be I/O limited until you hit pretty big simultaneous connections (and then CPU will hit), but dynamic servers can be CPU limited at pretty small numbers of simultaneous connections if your backend services are quick... limit the number of connections here to increase throughput. If the proxy can't get a response from the dynamic server in time, use a cached page or send a failwhale. Incidentally most backend services don't really work well with a lot of concurrency either, so often you can get a lot more throughput by bringing the number of PHP processes (for example) way down.

* but be aware that squid is basically single processor unless you work hard: http://wiki.squid-cache.org/ConfigExamples/MultiCpuSystem

 

[extide]

with a good build of pfsense you could set up like ~4 of them to be synchronized to each other and split up the load. You could use squid as a reverse proxy for your web site, this will also move the ssl encryption load to the pfsense box. If you want to make use of squid for caching then you can actually use up a fair amount of ram, just depends on how it's all configured. Good point in the post above about multi-threading and squid, however. Multiple physical boxes will aid you with that, as well.

I would say get the following hardware: (this list will keep you with server grade stuff yet still be inexpensive)
$234 Intel Xeon E3-1225 V2 Ivy Bridge 3.2GHz (3.6GHz Turbo) (This one looks like a good price/performance ratio, has on-board graphics and is 22nm Ivy Bridge)
$35 Kingston 4GB (2 x 2GB) 240-Pin DDR3 SDRAM ECC (might as well go ECC, its hardly any extra cost)
$109 Intel 313 Series Hawley Creek 2.5" 20GB SATA II SLC SSD (SLC flash for durability, Intel for reliability, these drives would be PERFECT for something like this.)
$64 Rosewill CAPSTONE Series CAPSTONE-450 450W 80 PLUS GOLD Certified (I have read that this is a good PSU, also 80+ Gold so it will be nice and efficient, but not too expensive.)

$427 TOTAL plus a case and PSU

And then you can go for super small like this:
$159 Intel DBS1200KP Mini ITX Server Motherboard LGA 1155 Intel C206 DDR3 1066/1333 (Dual onboard LAN)
$586 TOTAL With the Mini ITX Mobo


Or normal size and this gets you Quad on-board LAN.
$199 ASUS P8B-C/4L ATX Server Motherboard LGA 1155 Intel C202 DDR3 ECC UDIMM
$626 TOTAL with the quad onboard lan


$586 - $626 x 4 = $2344 - $2504 and all you'd need is cases, and put pfsense on them all and configure them in parallel. You could accomplish multiple layers of security on one box, make all 4 be identical and synchronized and redundant. That's what I would suggest. I have used pfsense and I love it. I plan to be moving myself and some other customers to it soon.



EDIT: Changed CPU as initially I had the 1220 v2 which does NOT have graphics, changed to 1225 v2 w/ graphics.
Another CPU Option could be the
$314 Intel Xeon E3-1265L V2 Ivy Bridge 2.5GHz (3.5GHz Turbo) 8MB L3 Cache LGA 1155 45W NOTE: This is 45w so may use a bunch less power. It really depends on the workload, if the cpu above is idle it will clock down anyways. It is also about a $100 upgrade, which IMO is not worth it.



This is definitely the way to go. Running a bunch of VM's is pointless, it will just make your setup a lot more complex and less reliable. Go with ~4 machines in PARALLEL so that you could get by with just one if you had to, but have four. Make them with identical hardware and I would say buy an extra kit of ram, an extra PSU, and maybe an extra SSD to have spare in case something fails.

 

[toast0]

Also, check out this list of things to make your site faster, many of these will reduce packets, and less packets == better for you. Some of them are client side performance which doesn't really make a difference though.

http://developer.yahoo.com/performance/rules.html/

Specific things I'd suggest you look into:

Add an Expires or a Cache-Control Header (might not fit with your customization needs)
Use a Content Delivery Network (might not be within budget)
Minimize HTTP Requests (less traffic maybe = less work)
Gzip Components (reduces traffic at cost of CPU, is often a gain, because you usually have a little cpu room and more packets == more CPU)
Minify JavaScript and CSS (more work at publish time = less data to send)
Avoid Redirects (where appropriate... fewer redirects == fewer requests)
Remove Duplicate Scripts
Configure ETags (mostly turn them off)
Make Ajax Cacheable (if appropriate)
Use GET for AJAX Requests
No 404s (don't have script/image/css links to things that aren't there)
Reduce Cookie Size
Use Cookie-free Domains for Components
Optimize Images (crunch down images while publishing, less data == less work)
Make favicon.ico Small and Cacheable
Avoid Empty Image src (fetches the page again, lots of extra work for nothing)

 

[JTY]

At that attack level, you want ASIC based filtering. Otherwise, go parallel with many systems taking shards of traffic. Basically use bonding to split the traffic across several ethernet links, each link going through a filtering box. The switches upstream and downstream of the filtering systems will take care of balancing the traffic.

That said, your best off going with a filtering appliance. And, some attack detection method that injects ACLs into your switches and routers so they can drop traffic from attackers using ASIC based filtering.

 

[Deleted member 12106]

Can't iptables drop the traffic?

 

[Igal@Incapsula]

Can't iptables drop the traffic?

Not if its a botnet DDoS. In this scenario, blocking IPs will only play into the attackers hands.

For botnet DDoS, deflation is the only viable form of mitigation, and although many here suggested a hardware solution I would suggest looking into front-gate Cloud CDN since it's much more cost-effective, much easier to setup and requires zero maintenance (cost and time wise).

 

[Vashypooh]

Clear solution is to fight fire with fire.

2x 10gbps can mess some stuff up.

 

[Ockie]

Clear solution is to fight fire with fire.

2x 10gbps can mess some stuff up.

This. Corporations will quickly fix their security holes once you turn those guns back onto them

 

[hawk82]

I think you're probably over your head. You should contact a company that specializes in mitigating DDoS attacks, either through proxies or other solutions. They'd have the best solution for you, either short term or long term.

 

[idmud]

DDoS attacks should be stopped as close to the source as possible. It's not generally something you do at your perimeter, even though you should have mechanisms in place to take care of that which your ISP does not block.

If you get 10Gbit from your ISP, they surely offer DDoS protection services. The ones using arbor networks work really well.