Pfsense WAN load balancing CPU requirements

Machupo

Gravity Tester
Joined
Nov 14, 2004
Messages
5,754
Anyone doing dual-WAN on a pfsense box? I am wondering what kind of CPU power is required for load balancing two decently high speed connections (1gbps and 300mbps) and would love to hear your tales to know if I'm barking up the wrong tree with my current pfsense box.

Thanks in advance!
 
What's your current specs? I haven't done it on pfsense, but been running multi-wan since 2004 so may have some insight. (y)
 
The biggest CPU resource hogs with pfsense come from addons that process the traffic. The CPU power required merely to handle the bandwidth, with minimal/stock addons, will be much less. They recently made AES-NI hardware encryption a bigger part of pfsense, and Sandy Bridge is the first CPU generation to support that. I would not go any older than that. You can find cheap Dell Quad-Core Sandy-Bridge/Ivy Bridge SFF systems these days for like $50. That is what I would recommend unless you use tons of extra addons to process traffic. I recommend Dell mainly because they have a good track record of providing reasonably up-to-date BIOS updates even for their older systems.
 
What's your current specs? I haven't done it on pfsense, but been running multi-wan since 2004 so may have some insight. (y)
Sorry about that, I should have included the current specs.

I am rocking a Supermicro X10SDV-TLN4F (Xeon D-1541, Broadwell chip) in the edge device.

As far as services, I run pfblockerNG with DPI/local cert and have been dabbling in suricata.
 
Sorry about that, I should have included the current specs.

I am rocking a Supermicro X10SDV-TLN4F (Xeon D-1541, Broadwell chip) in the edge device.

As far as services, I run pfblockerNG with DPI/local cert and have been dabbling in suricata.
All good. And with your current setup, what is your current wan speed and what type of max cpu/memory are you seeing?
 
All good. And with your current setup, what is your current wan speed and what type of max cpu/memory are you seeing?

I don't have WAN right now, that's why I was asking the question -- looking at a 300mbit (allegedly) vdsl+LTE offering and whatever starlink can get to me (Ookla was seeing a 120mpbs average in Q1). Generally, I am pretty sure this can handle these speeds, but want it to be good enough to deal with the VDSL WAN and a 1gbps WAN in the event that the new VLEO starlink shell gets built out and it gets up to gigabit.
 
I don't have WAN right now, that's why I was asking the question -- looking at a 300mbit (allegedly) vdsl+LTE offering and whatever starlink can get to me (Ookla was seeing a 120mpbs average in Q1). Generally, I am pretty sure this can handle these speeds, but want it to be good enough to deal with the VDSL WAN and a 1gbps WAN in the event that the new VLEO starlink shell gets built out and it gets up to gigabit.
Gotcha. I would say just run with what you got and watch the cpu and memory. If either one is above 50% with 300Mbps, then you'll need to beef up a bit. Unfortunately with the D-1541 not being socketed, it would mean a motherboard swap, but even a sandy bridge like mentioned above would be an cheap and effective upgrade.

Just keep in mind that enterprise off-the-shelf solutions may be cheaper and more effective. This watchguard M200 while older is a steal at open box new as it can handle up to 7 wans, and in my experience it handles 500/50 and 100/15 without even moving the cpu from zero or the memory from bootup levels:
https://www.ebay.com/itm/354121653597

Or there's the netgate boxes loaded with pfsense too.
 
As an eBay Associate, HardForum may earn from qualifying purchases.
pretty sure my mobo can stomp that ivy bridge xeon all day long, but appreciate the linkage.

Other than "hook it up and see what happens", anyone got any real world experience with the cpu requirements for load-balancing... from what I've seen in the thread so far: not a very big impact on cpu performance (certainly less than DPI and other traffic shaping / intrusion prevention packages I am running?)
 
pretty sure my mobo can stomp that ivy bridge xeon all day long, but appreciate the linkage.

Other than "hook it up and see what happens", anyone got any real world experience with the cpu requirements for load-balancing... from what I've seen in the thread so far: not a very big impact on cpu performance (certainly less than DPI and other traffic shaping / intrusion prevention packages I am running?)
Then I think you'll be on good shape. A lot of the guys on that STH thread will run some of the packet inspections and whatnot so they'll tax that box and if yours kicks it hard then it will be fine.
 
Late to the party as usual, but you got me curious now about my setup. I was running active-backup with 2 carries since one was dsl. But I just got Verizon home internet which is comparable to spectrum. I run a pfsense virtualized on a xeon L5638. pfsense has 4 vcpu /8gb ram. Running a standard speedtest on spectrum, it peaks at 39% cpu usage on a 400/20 with minimal addons (ntopng, open-vm-tools, and openvpn-client-export)
 
Late to the party as usual, but you got me curious now about my setup. I was running active-backup with 2 carries since one was dsl. But I just got Verizon home internet which is comparable to spectrum. I run a pfsense virtualized on a xeon L5638. pfsense has 4 vcpu /8gb ram. Running a standard speedtest on spectrum, it peaks at 39% cpu usage on a 400/20 with minimal addons (ntopng, open-vm-tools, and openvpn-client-export)
Are you running baremetal? Seems like gigabit in your current config might tax it to 100% if the cpu usage grows linearly.
 
I agree about the linear speeds, at least on this setup. But it is worth noting the age and specs of my hardware vs what you are using. No baremetal. On some hold HP gear running esxi. Unfortunately, I don't have the same speeds as you to test with but my Spectrum is 400/20 and Verizon is 300/10 if I can get it back to the window. It works in the basement by the server cabinet but drops significantly (and expectedly) to 100/1 so I will have to run some cabling upstairs for a permanent setup.
 
I agree about the linear speeds, at least on this setup. But it is worth noting the age and specs of my hardware vs what you are using. No baremetal. On some hold HP gear running esxi. Unfortunately, I don't have the same speeds as you to test with but my Spectrum is 400/20 and Verizon is 300/10 if I can get it back to the window. It works in the basement by the server cabinet but drops significantly (and expectedly) to 100/1 so I will have to run some cabling upstairs for a permanent setup.
Oh, I'm not the OP. :D And being virtual there might be a bit more that could be squeezed out of your cpu in a bare metal setup. It will be interesting to see if you can hit 70% cpu with both wans totally 700Mbs. That would also show any overhead in running dual wans.

As far as the OP, with a D-1541, there's almost 2x single thread performance and a few more cores and threads, so I think they will be okay for their 1Gb/300Mb setup:
https://www.cpubenchmark.net/compare/Intel-Xeon-D-1541-vs-Intel-Xeon-L5638/2718vs1262
 
Oh, I'm not the OP. :D And being virtual there might be a bit more that could be squeezed out of your cpu in a bare metal setup. It will be interesting to see if you can hit 70% cpu with both wans totally 700Mbs. That would also show any overhead in running dual wans.

As far as the OP, with a D-1541, there's almost 2x single thread performance and a few more cores and threads, so I think they will be okay for their 1Gb/300Mb setup:
https://www.cpubenchmark.net/compare/Intel-Xeon-D-1541-vs-Intel-Xeon-L5638/2718vs1262

I agree with you, they have a much newer CPU and it's dedicated to pfsense. I don't see an issue either. I was looking at systems with the same cpu.

All good. I'm just trying to give some data since it's mostly what you find on the internet/or guessing. I think I got it working, although the tests completed over speedtest and fast are sometimes inconsistent (some tests were only 200/30, 400/8 or 650/5 or some variations). Maybe from my running constantly? Running the tests back to back, I was able to hit a peak of 64% at 650Mb but generally, peaks were under 58% hitting 630-650Mb.

References:
https://www.cyberciti.biz/faq/howto-configure-dual-wan-load-balance-failover-pfsense-router/#
https://docs.netgate.com/pfsense/en/latest/multiwan/load-balance-and-failover.html

Verizon from my basement:
speedtest-verizon-basement.JPG


Verizon from my desk:
speedtest-verizon-desk.JPG


Spectrum:
speedtest-spectrum.JPG


Load balanced- mostly consistent if I wait a bit in between tests. (58% CPU peak)
speedtest-spectrum_verizon.JPG


Some load balanced uhhhs on Fast.com lol It was showing as high as 2.2Gbps at times.
fastTEST-lbrun1.JPG


fastTEST-lbrun6.JPG


uhhhwhat2.JPG
 
I agree with you, they have a much newer CPU and it's dedicated to pfsense. I don't see an issue either. I was looking at systems with the same cpu.

All good. I'm just trying to give some data since it's mostly what you find on the internet/or guessing. I think I got it working, although the tests completed over speedtest and fast are sometimes inconsistent (some tests were only 200/30, 400/8 or 650/5 or some variations). Maybe from my running constantly? Running the tests back to back, I was able to hit a peak of 64% at 650Mb but generally, peaks were under 58% hitting 630-650Mb.
Thank you for sharing some really great real-world data. (y) After seeing your cpu results, I'm pretty confident that the OP won't have an issue with the D-1541.

Most speed tests still can't handle multi-wan correctly. I see the same wackiness you do with fast as I saw 1.6Gb on our 800Mbs line at one point and that's not even possible since there wasn't a 1Gb link to the modem. :ROFLMAO: dslreports' speedtest was my go to, but something is borked on it and it isn't working for anyone it seems. But that actually handled multi-wan pretty well.
 
Thank you for sharing some really great real-world data. (y) After seeing your cpu results, I'm pretty confident that the OP won't have an issue with the D-1541.

Most speed tests still can't handle multi-wan correctly. I see the same wackiness you do with fast as I saw 1.6Gb on our 800Mbs line at one point and that's not even possible since there wasn't a 1Gb link to the modem. :ROFLMAO: dslreports' speedtest was my go to, but something is borked on it and it isn't working for anyone it seems. But that actually handled multi-wan pretty well.

Thanks!

That is good to know they don't work properly. dslreports says it cannot connect to all four test servers and that it's my fault :) I did try from 3 different browsers on 2 machines with the same result. Using my phone on mobile data, it did appear to work though. The rest of the test sits I found seemed to do the same as the major players.

I found a couple docs that talk about checking the functionality. The key reminder from netgate is that pfsense is doing round robin load balancing which I was able to follow the tp-link doc and it worked as they described when testing from more than one desktop. My desktop shows Verizon, but two VMs show spectrum
https://www.tp-link.com/us/support/faq/2079/
https://docs.netgate.com/pfsense/en/latest/multiwan/test.html
 
Thanks!

That is good to know they don't work properly. dslreports says it cannot connect to all four test servers and that it's my fault :) I did try from 3 different browsers on 2 machines with the same result. Using my phone on mobile data, it did appear to work though. The rest of the test sits I found seemed to do the same as the major players.

I found a couple docs that talk about checking the functionality. The key reminder from netgate is that pfsense is doing round robin load balancing which I was able to follow the tp-link doc and it worked as they described when testing from more than one desktop. My desktop shows Verizon, but two VMs show spectrum
https://www.tp-link.com/us/support/faq/2079/
https://docs.netgate.com/pfsense/en/latest/multiwan/test.html
You're welcome. Yeah, I too have run into the same issue, also testing multiple OS and browsers. Interesting that it works on mobile--maybe that's a clue as to what broke.

Round robin is pretty common for multi-wan. It does break certain things though that require persistent connections such as ssl. If there are other options like failover that would be good. Also if there are options to make certain destinations 'stick' to one wan or another, that would also work for those destinations that be broken. I know my older cisco rv016 had this option.
 
Back
Top