• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Gigabit VPN options?

P

Phog

Limp Gawd
Joined
Aug 21, 2012
Messages
275
I'll be getting Google Fiber in a month or so and I was wondering if there are any affordable options that might be able to support 500+mbps VPN links. I know OpenVPN is the big daddy, but it seems to have max throughput issues from my testing. I'd rather not have to hack around with it to make it work, but perhaps that's my only choice.

Anyone else know of any good VPN solutions for that kind of throughput?
 
Phog said:
I know OpenVPN is the big daddy, but it seems to have max throughput issues from my testing.
Click to expand...

Explain?

If you're seeing throughput "caps" I would imagine its due to hardware not being able to keep up with not only the bandwidth, but the encrypting/decrypting of said traffic.

Time to setup a pfsense openvpn lab. I've had pfsense route @ almost gig speeds. "Almost" because I as limited by hard drive speed.
 
I setup the openVPN access server on a VM, and connected to it locally over a 1 gig network. I found the speed to cap around 160mbps (and CPU usage was low). With Google Fiber, they force the use of the network box, so if I used pfsense, I'm guessing I would just forward the VPN ports from the network box to the pfsense VM?
 
OpenVPN is most likely _not_ going to do it since its single threaded.
//Danne
 
diizzy said:
OpenVPN is most likely _not_ going to do it since its single threaded.
//Danne
Click to expand...

This, I'd be interested to what the CPU Usage was. "Low" as in 25% on a quad core? 13% on an 8 core.... etc...

What were the hardware specs of the vm? Most notably CPU model and its clock speed
 
I'm also very interested in any open source (or free) solutions that can push gigabit speeds given the right hardware.
 
jadams said:
This, I'd be interested to what the CPU Usage was. "Low" as in 25% on a quad core? 13% on an 8 core.... etc...

What were the hardware specs of the vm? Most notably CPU model and its clock speed
Click to expand...

As previously mentioned, it is single threaded, so I only saw usage on one core. The server was allocated four cores (8gb ram), each running at 2.4ghz. (the host server has Dual Xeon E5530's). I believe it was around 30% usage, and I was seeing around 110 mbps.
 
Open source is probably the affordable route to take to get gigabit throughput. Now with that being said look at this from a couple of different angles.

1. how many concurrent connections are you going to have?
2. where will your connection be coming from?
3. If its just you connecting to your home do you want to dedicate VM resources or a separate computer just to have VPN access?

If its just you connecting from starbucks then go and pickup a Cisco ASA-5505, Watchguard XTM 25, or a Sonicwall tz 105. They all support SSL VPN and cisco and Sonicwall have an SSL VPN app for mobile devices. Yes the VPN throughput is between 60-100mbps, but I highly doubt you're going to be connecting from free public wifi that has similar bandwidth as your connection you can use just for your session. To get a product that can handle 1gbps VPN throughput it will cost as much as a small mortgage.
 
EV42TMAN said:
To get a product that can handle 1gbps VPN throughput it will cost as much as a small mortgage.
Click to expand...

I take it there is no open source software available to run on commodity hardware that can pull this off? You have to go with cisco/juniper etc?
 
Phog said:
As previously mentioned, it is single threaded, so I only saw usage on one core. The server was allocated four cores (8gb ram), each running at 2.4ghz. (the host server has Dual Xeon E5530's). I believe it was around 30% usage, and I was seeing around 110 mbps.
Click to expand...

Yep that sounds about right. So you'd need a 16Ghz processor to pull that off. haha.

What was the Crytopraphic settings? Is there different hits in performance based off different methods? 64bit, 128bit, etc...

Also with v2.0.1 and above I believe there are hardware encryption options that you can specify in the server config that will offload from the cpu.
 
Last edited:
@jadams

As far as I know, pfSense from 2.1.x onwards has supported for the AES-NI instructions in modern Intel and AMD CPUs. Look here for more info. Obviously this means being installed on bare metal; I'm not sure that even vSphere or ESXi will pass those CPU instructions through.
 
Depending on what type of VPN and what encryption a Routerboard Cloud Core Router would at least get close.

It does have a hardware encryption engine, and it IIRC there are multithreaded VPN protocols (not OpenVPN) to take advantage of the 32x 1.2GHz cores.

You could always bond several OpenVPN tunnels together too...
 
parityboy said:
@jadams

As far as I know, pfSense from 2.1.x onwards has supported for the AES-NI instructions in modern Intel and AMD CPUs. Look here for more info. Obviously this means being installed on bare metal; I'm not sure that even vSphere or ESXi will pass those CPU instructions through.
Click to expand...

I'm setting up a 2.0.3 lab right now. Going to test this. I'll try the latest 2.1 snapshot after too.
 
I guess I don't understand the purpose behind the project. 1gbps is a pretty good amount of bandwidth just to route, let alone encrypt/decrypt at wirespeed. And on the OP's home Google Fiber connection? What's the point? Providing wirespeed secure access to his personal pron collection?

That being said. Even using OS software, the hardware for this will not be cheap. Is it worth it?
 
I'll just jump on the end here. what can pfSense achieve with VPN's / IMIX traffic, the website is a bit vague.

Are we taking 50Mbit AES/3DEC on an i5, or is it likely to be higher, and how is it with inter vlan routing?
 
Liquidkristal said:
I'll just jump on the end here. what can pfSense achieve with VPN's / IMIX traffic, the website is a bit vague.

Are we taking 50Mbit AES/3DEC on an i5, or is it likely to be higher, and how is it with inter vlan routing?
Click to expand...

As some of us have pointed out its going to be limited to a single thread of the cpu. I should have some stats here soon to get some sort of a baseline.

I have two pfsense vm's and two client vm's setup in hyperv. The client pc's virtual hard drives are on different storage devices. I should be able to get full gigabit speeds when I get the vpn up and running. Though I'm running into a small issue at the moment...

This is my first time setting up pfsense's in hyper-v. I can get the pfsense vm's to talk to each other over wan, but what I cant do is ping the WAN's from the "remote" clients... IE client on site 2 pinging wan on pfsense 1. I have more testing to do but it wont be until the weekend at least.
 
jadams said:
As some of us have pointed out its going to be limited to a single thread of the cpu. I should have some stats here soon to get some sort of a baseline.

I have two pfsense vm's and two client vm's setup in hyperv. The client pc's virtual hard drives are on different storage devices. I should be able to get full gigabit speeds when I get the vpn up and running. Though I'm running into a small issue at the moment...

This is my first time setting up pfsense's in hyper-v. I can get the pfsense vm's to talk to each other over wan, but what I cant do is ping the WAN's from the "remote" clients... IE client on site 2 pinging wan on pfsense 1. I have more testing to do but it wont be until the weekend at least.
Click to expand...

I'm curious if you've made any progress on this?
 
bleomycin said:
I'm curious if you've made any progress on this?
Click to expand...

I'm sorry, I havent. Got busy with work and then the holiday. Still having that issue of getting the sites to talk to each other. I'll try to get it this week.
 
Got the WANS to talk. Was really weird. Just briged the WANS onto my normal LAN and it worked. Couldnt get them to talk on their own virtual network. Going to post some initial screenshots of CPU usage with only routing through NAT. No VPN yet. I might be able to get to that later this week. but at least this will give us a little clue into CPU usage.
 
Assuming you get a fast AES-NI capable CPU, you should be able to build a gigabit OpenVPN router. I've built them for customers before, although I can't provide any proof of the real-world traffic levels near gigabit speed... once they're in the DC, that's pretty much the end of it from what I get to see.

Your issue will be getting a VPN provider who also has equally powerful gear feeding your session.
 
http://i.imgur.com/lwVBcM1.png

45% cpu usage of a dual core 2.83Ghz Xeon E5440. This is ONLY basic throughput. No VPN.... yet.

I'll be setting that up shortly. Time to go follow my own guide. lulz
 
Last edited:
OpenVPN is really bad if you want to use hardware acceleration due to the design of it but good luck and you would be much better off on 10-CURRENT :)
//Danne
 
bleomycin said:
I just found this, it's looking like 800+ Mbit is easily done with openvpn and aes-ni capable hardware, very encouraging!
Click to expand...

Whered you find that? Link?

I hope I can run that in my test lab I'm setting up.
 
So heres my findings:

Screenshot of the open file shares. Top window is going through NAT and the bottom one going through VPN. Windows goto the exact same file share on the exact same machine.
http://imgur.com/wEytkoQ

Transfer Speed and cpu usage again through NAT/Routing
http://imgur.com/oVqLxxq

Transfer speed and cpu usage through vpn
http://i.imgur.com/wV2qMmb.png

About 10% the performance. I was WAY off lol.

The only Hardware Crypto options I have in this VM are none, and BSD Crypto engine. BSD crypto option only gives me about another 2-4MegaBYTES/sec

EDIT: And WOW! Half way through transferring the 3GB file pfsense actually crashed and rebooted. First time I've ever been able to crash pfsense. I feel honored.
EDIT2: After it came back up my "remote" client can connect to the vpn, but the OpenVPN server crashes.

I have to say, I am thoroughly disappointed in pfsense and openvpn here.
 
Last edited:
Well, 8.X-branch is old and you'll need to do a lot of manual tweaking to get get anywhere near the performance you're looking for. As I said before, you'll be much better off using IPSec than OpenVPN due to design. There are also other VPN that probably scales better and running pfsense in a VM isn't really ideal.
//Danne
 
Curious, why with openvpn when you connect it only states it is at 10Mbps whne you check task manager?
 
diizzy said:
Well, 8.X-branch is old and you'll need to do a lot of manual tweaking to get get anywhere near the performance you're looking for. As I said before, you'll be much better off using IPSec than OpenVPN due to design. There are also other VPN that probably scales better and running pfsense in a VM isn't really ideal.
//Danne
Click to expand...

I understand all these things. I have a physical box I could possibly run this on. However its only a Core 2 Duo. Nonetheless I expected better performance, and certainly didnt expect it to crash and then become unusable afterwards.

MrGuvernment said:
Curious, why with openvpn when you connect it only states it is at 10Mbps whne you check task manager?
Click to expand...

I'm not sure. The TAP adapter shows does show that but I've sent more than 10Mbps through it. Sent almost 10x through it according to my test.

Additionally what I do find interesting is a cpu usage of ~70%. Its always been my understanding that it was single threaded. If this were the case then cpu should have spiked and 50%. I may reinstall and do quad core to see if performance is any better.
 
Comparing performance to my production pfsense running OpenVPN

Intel Atom D510 @ 1.66Ghz. I'm able to max your my Max upload form the vpn server @ 15Mbps. Cpu usage is around 10%

Saturating the download @ 25Mbps the cpu usage is @ 15%.

My production pfsense still runs v1.2.3 at least until tomorrow :D

The only difference I have between the OpenVPN configs is the cryptography method. My production pfsense uses BF-CBC 128bit, the one above in this test uses AES-128-CBC. I'm no crypto expert. I dont know just how much this effects performance.
 
Visualization isn't always "the way to go", you have a lot of interesting issues that can affect underlying software greatly. Clock skew is one that comes to mind that can make alot of stuff break.

While OpenVPN has never been "fast" or at least advertised as fast its not that bad either considered that its single-threaded meaning that it only uses one core. I'm not sure if that's true for openvpn overall or just per session/client though. You have also a lot of data going back and forth between the kernel and userland which is bad especially on low-end devices such as soho routers running *WRT.

FWIW my EdgeRouter Lite maxes out at around 500kbyte/s and that's a pretty fast 500Mhz MIPS Dual Core CPU running OpenWRT with 192-bit encyption. For comparison my old 266Mhz MIPS single core does about 800-900kbyte/s using vtund

As for the TAP/TUN interface its just cosmetical, there's no way of tellning the link speed except estimating it.

Running a few x86/64-bit and MIPS boxes I've never encountered issues with running 9.1 (9-STABLE) or 10-CURRENT and network services.

//Danne
 
diizzy said:
While OpenVPN has never been "fast" or at least advertised as fast its not that bad either considered that its single-threaded meaning that it only uses one core. I'm not sure if that's true for openvpn overall or just per session/client though. You have also a lot of data going back and forth between the kernel and userland which is bad especially on low-end devices such as soho routers running *WRT.
Click to expand...

I always found this to be the case with pfsense in general. That its single threaded. I've had it running on modest (although multicore) hardware before and put it under heavy load in the past. I once had it running on a C2D and was only ever able to get the cpu to go above 50%.

When a multi core cpu meets a single threaded application the most youll ever get for cpu utilization is 100/number of cores. In my C2D scenario I was never able to get it to go over 50% usage, which was to be expected.

However in my testing on the VM I saw some different behavior. CPU usage shooting up to 70ish%. So i decided to run top while doing a file transfer through vpn and this is what I got.

http://imgur.com/uvvciXQ

The "State" column continuously switches between CPU1 and CPU0 and both are usually around 80-85%. If we have two cores that are continuously between 80-85% thats an overall processor utilization of 60-70% . On the other hand if you have an application that is only utilizing 80% of one core then then that would result in an overall cpu utilization of 40%. This all assumes a dual core.

So I'm curious as to why the OpenVPN server is behaving like multi threaded application.
 
pf or rather networking in general was greatly improved in FreeBSD 9.X and 10 (-CURRENT) offers even more enhancements. Please have in mind that 8.X isn't bad in anyways but pfsense does add patches that isn't in the main branch for several reasons and some are due to performance. As scary as -CURRENT (bleeding edge) sounds it works well on my mips machines and pf is multithreaded too.
Using VMs often gives you somewhat strange results which is why you shouldn't base performance on VMs but on real hardware in general. Not saying that VMs are all bad but you might find some strange quirks that isn't on real hardware. Of course work is being done to fix these issues but there's a lot more to debug and fix in that case. Regarding the core switching its probably the scheduler trying to make all processes run as smoothly as possible and when your VM gets CPU performances "boosts" it switches processes between cores. In your case I'd try a plain -CURRENT image and benchmark.
//Danne
 
I have to use the current image due to it having the drivers for hyperv's default NIC's. You can use the Legacy Virtual NIC's but you're limited to 10/100. Not sure if newer images have this embedded or not.

I've gotten pretty quick at pfsense install and open vpn server config. I can gernally have it done in about 15 mins. I'll try an updated image maybe this weekend if i have a chance.
 
You must log in or register to reply here.
Back
Top