Post Your pfSense Setup/pfSense Sugestions

Joined
May 20, 2011
Messages
898
I've been running pfSense for years now, the last few I've been running it on a Dell T20 that I had collecting dust. It's been super reliable, went almost a year without any down time. I would like to repurpose the Dell as a blueiris server, so I'm looking for ideas on what to replace it with. Something small, low power, but still powerful enough to run my 1000/100 connection. I do run openvpn and would like close to equivalent speeds that my current setup provides. I also run squid, snort, and a few other packages.

Originally I was going to go with pc engines apu2c4 but after some research I don't think it can keep up with my connection, and definitely not my vpn. Now I'm looking at some of the minisys & qotom devices that seem popular and quite capable. Namely the i3 & i5 versions.

Any insight is greatly appreciated!
 

IdiotInCharge

NVIDIA SHILL
Joined
Jun 13, 2003
Messages
14,679
This i5 Qotom is more or less what I've been looking at. I have a J3160-based version now, which is about twice as fast as the AMD mess in the APU-series boxes, and I wonder if it would be able to handle gigabit downloads myself.

There is also ProtectLI that sell rebadged Qotoms (etc.) and have i5-based six-port version for the top-end, but they seem to be really proud of them.
 

rtangwai

[H]ard|Gawd
Joined
Jul 26, 2007
Messages
1,369
Do you happen to have an ESXi server or a Hyper-V box available?

pfSense works quite well for me as a VM in ESXi 6.5, especially as I use Intel NICs that have paravirtualized drivers - they run pretty close to bare-metal speeds so no need to do passthrough.

That way you don't have to buy another box, at most you have to buy a NIC.
 

IdiotInCharge

NVIDIA SHILL
Joined
Jun 13, 2003
Messages
14,679
Do you happen to have an ESXi server or a Hyper-V box available?

pfSense works quite well for me as a VM in ESXi 6.5, especially as I use Intel NICs that have paravirtualized drivers - they run pretty close to bare-metal speeds so no need to do passthrough.

That way you don't have to buy another box, at most you have to buy a NIC.

I thought about this- prototyped pfSense in Hyper-V first- and wound up doing Untangle (free) for the moment on the appliance.

One consideration: server maintenance takes out your LAN :D.

That's why I have a separate box. Need my tinkering buffer ;).

[Note- I wound up with Untangle because I couldn't get Sophos XG Home running on the appliance, and I couldn't get pfSense (or OPNSense) to easily run in passive mode, which Untangle did effortlessly- I'm keeping my Edgerouter4 as router/edge firewall/DHCP server for now]
 

rtangwai

[H]ard|Gawd
Joined
Jul 26, 2007
Messages
1,369
I know what you mean, fortunately for me I use Bell Fibe and a HomeHub 3000. The 3000 doesn't go into bridge mode, instead you designate a MAC address as the DMZ and that NIC gets an external instead of an internal IP. That means the 3000 can provide Internet access simultaneously and independently of pfSense should I screw up and need to look up what I did wrong this time and how to fix it. I have the 3000's WiFi running so I don't even have to move cables around when pfSense/ESXi is down.
 

IdiotInCharge

NVIDIA SHILL
Joined
Jun 13, 2003
Messages
14,679
That's pretty nice-

On another note, found ProtectLI's source on AliExpress, not much of a discount after all: 7200U Appliance

This is basically my 'perfect'. ECC support would be nice, but overall, that's as small as you're going to get with dual drives for a ZFS mirror for pfSense in a 'modern' platform with real cores. I'd run ESXi on it myself, and perhaps more than one router OS instance alongside my other stuff (Ubiquiti apps, pihole...).
 
Joined
May 20, 2011
Messages
898
Do you happen to have an ESXi server or a Hyper-V box available?

pfSense works quite well for me as a VM in ESXi 6.5, especially as I use Intel NICs that have paravirtualized drivers - they run pretty close to bare-metal speeds so no need to do passthrough.

That way you don't have to buy another box, at most you have to buy a NIC.

I've thought about this as well. I do have an exsi box and have messed around with a pfSense vm. I'd like to do it, but I know I usually have to shut down my host at least once a month. Not sure I could get away with the down time.

Question, why don't you use passthrough for your nics? Is there any benifit other than not needing so many physical nics?


Ah, here's a link to a similar Qotom listing to that ProtectLI i5. Looks like a significant markup if you don't need the RJ-45 console port (which I find useful).

Also note that anything 6000-series and newer is going to need DDR4, while the 4000-series Core builds are DDR3.
On another note, found ProtectLI's source on AliExpress, not much of a discount after all: 7200U Appliance

This is basically my 'perfect'. ECC support would be nice, but overall, that's as small as you're going to get with dual drives for a ZFS mirror for pfSense in a 'modern' platform with real cores. I'd run ESXi on it myself, and perhaps more than one router OS instance alongside my other stuff (Ubiquiti apps, pihole...).

These qotom boxes look perfect for my needs. I'd Iove to get one of the six nic 7th gen one's but I can't justify spending so much on something that'll most likely be underutilized. $300ish is about what I have to spend. I don't think I have any need to run esxi on top, as my other host does pretty much everything I need.
 

IdiotInCharge

NVIDIA SHILL
Joined
Jun 13, 2003
Messages
14,679
These qotom boxes look perfect for my needs. I'd Iove to get one of the six nic 7th gen one's but I can't justify spending so much on something that'll most likely be underutilized. $300ish is about what I have to spend. I don't think I have any need to run esxi on top, as my other host does pretty much everything I need.

The DDR3-based ones would probably work just as well. I mentioned the newest ones because they're likely as fast as you'd want for any of the available routing OS distros for basic services and IPS etc. packages on top.
 

rtangwai

[H]ard|Gawd
Joined
Jul 26, 2007
Messages
1,369
I've thought about this as well. I do have an exsi box and have messed around with a pfSense vm. I'd like to do it, but I know I usually have to shut down my host at least once a month. Not sure I could get away with the down time.

Question, why don't you use passthrough for your nics? Is there any benifit other than not needing so many physical nics?

It is one less variable to deal with when you can avoid passthrough.

I also have NICs teamed in ESXi so I not only have more throughput (when multiple clients connect) I also have redundancy if a NIC or switch port fails. Way easier to do in ESXi than on a managed switch.
 

Machupo

Gravity Tester
Joined
Nov 14, 2004
Messages
5,372
Those qotom boxes look pretty snazzy. I am running a 1u netgate octocore box I picked up secondhand. Throwing every package i can use at it including ssl proxy and a massive inspection database and can't make it sweat. Total overkill for my 100/5 connection.
 
Joined
May 20, 2011
Messages
898
I ended up getting a Qotom Q355G4. It's a four nic i5 5200u model. Should be perfect for my needs for the foreseeable future. $219 shipped from China, supposed to be here early next week. Add in a $25 8gb ddr3l stick and a 64gb msata ssd I have laying around, should be a kick-ass low power pfSence setup.

I'm still interested in virtualizing at some point. I did test it out over the weekend getting virtualized pfSence set up on my esxi box. Worked perfectly actually still have it going running the home network. I did end up using passthrough for the pcie nic I have. After more research I've seen several people set up a hardware failover pfSense box for when they need to take their host down. Some have even set it up to where the failover automatically boots and takes over when the virtualized router goes down. Seems pretty cool, maybe someday I'll dive more into it. For now a simple and in my case a more reliable hardware solution will work. I tend to tinker and over think these things way more than I should. At least I learn a lot while doing so.
 
Last edited:

Biznatch

2[H]4U
Joined
Nov 16, 2009
Messages
2,224
I ended up getting a Qotom Q355G4. It's a four nic i5 5200u model. Should be perfect for my needs for the foreseeable future. $219 shipped from China, supposed to be here early next week. Add in a $25 8gb ddr3l stick and a 64gb msata ssd I have laying around, should be a kick-ass low power pfSence setup.

I'm still interested in virtualizing at some point. I did test it out over the weekend getting virtualized pfSence set up on my esxi box. Worked perfectly actually still have it going running the home network. I did end up using passthrough for the pcie nic I have. After more research I've seen several people set up a hardware failover pfSense box for when they need to take their host down. Some have even set it up to where the failover automatically boots and takes over when the virtualized router goes down. Seems pretty cool, maybe someday I'll dive more into it. For now a simple and in my case a more reliable hardware solution will work. I tend to tinker and over think these things way more than I should. At least I learn a lot while doing so.


I've been running pfsense virtualized in a bunch of production environments for the past 6+ years. It's completely stable/supported and I've never had any issues. Yes when you do maintenance on the host itself you have downtime, but that is a pretty rare event.
 

EniGmA1987

Limp Gawd
Joined
May 2, 2017
Messages
429

JeffBlair

Limp Gawd
Joined
Jul 13, 2009
Messages
333
Right now I've got mine in a ESXi VM. But, the issue I'm having is every time I reboot pfSense, the whole box locks up, and I have to power cycle it. Don't know what's causing it, but it's a pain. Might get that mini-PC that someone listed. Had it running on another one, but kept dropping the network because I was having to use a USB-to-LAN adapter that kept dropping out.
 

Biznatch

2[H]4U
Joined
Nov 16, 2009
Messages
2,224
Right now I've got mine in a ESXi VM. But, the issue I'm having is every time I reboot pfSense, the whole box locks up, and I have to power cycle it. Don't know what's causing it, but it's a pain. Might get that mini-PC that someone listed. Had it running on another one, but kept dropping the network because I was having to use a USB-to-LAN adapter that kept dropping out.

I don't use ESXi but are there any logs you can view on what failed prior to the crash? Are you running the latest software updates/drivers/firmware?
 

JeffBlair

Limp Gawd
Joined
Jul 13, 2009
Messages
333
I don't use ESXi but are there any logs you can view on what failed prior to the crash? Are you running the latest software updates/drivers/firmware?
No, that was the first thing I looked at. It just totally locks everything up. I did just upgrade ESXi to the latest version, and it seams like it reboots now without an issue. But, I should be getting my standalone tomorrow. So, I won't have to worry about that any more.
 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
Do you happen to have an ESXi server or a Hyper-V box available?

pfSense works quite well for me as a VM in ESXi 6.5, especially as I use Intel NICs that have paravirtualized drivers - they run pretty close to bare-metal speeds so no need to do passthrough.

That way you don't have to buy another box, at most you have to buy a NIC.

Curious how this is done from a security standpoint.

Wouldn't that involve hooking up the VM server straight to the internet? I know you would assign the outside NIC only to the pfsense VM so the other VMs arn't actually facing the internet, but I still would feel a bit uncomfortable with that, I like to have a physical barrier between internet and LAN in the form of a physical box with two NICs.

Anyone familiar with the Netgate boxes?

https://www.amazon.ca/SG-1000-micro...id=1544568636&sr=8-3&keywords=pfsense+netgate

I'm kinda thinking of buying one, as it would be way lower power usage than a full blown computer/server like I have right now.
 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
I know what you mean, fortunately for me I use Bell Fibe and a HomeHub 3000. The 3000 doesn't go into bridge mode, instead you designate a MAC address as the DMZ and that NIC gets an external instead of an internal IP. That means the 3000 can provide Internet access simultaneously and independently of pfSense should I screw up and need to look up what I did wrong this time and how to fix it. I have the 3000's WiFi running so I don't even have to move cables around when pfSense/ESXi is down.


This thread might help you: https://www.digitalhome.ca/forum/19...4496-some-tidbits-fibreop-infrastructure.html

I have the same service and I too found it annoying being stuck with a double NAT but I was able to get rid of the Actiontec and replace it with an Asus router and custom firmware that emulates the actiontec but allows for pass through. Might be a lot to read through though, I don't recall which specific parts of the thread that helped me but pretty sure that's the one that should have all the info.

It's even simpler if you don't have TV service and only have internet, as you just need to plug straight into the ONT and assign vlan 35 to the WAN port of pfsense. (been a while since I played with it so don't recall specifics) I ALMOST had TV service working too with just a switch, but it would cut out all the time. There's some type of layer 3 QoS you have to setup and I could not quite get it to work.
 

Biznatch

2[H]4U
Joined
Nov 16, 2009
Messages
2,224
Curious how this is done from a security standpoint.

Wouldn't that involve hooking up the VM server straight to the internet? I know you would assign the outside NIC only to the pfsense VM so the other VMs arn't actually facing the internet, but I still would feel a bit uncomfortable with that, I like to have a physical barrier between internet and LAN in the form of a physical box with two NICs.

Anyone familiar with the Netgate boxes?

https://www.amazon.ca/SG-1000-micro...id=1544568636&sr=8-3&keywords=pfsense+netgate

I'm kinda thinking of buying one, as it would be way lower power usage than a full blown computer/server like I have right now.


You put the WAN on a dedicated NIC, that is only connected to the pfsense box. I haven't heard of any vulnerability on any main hyper-visor that allows bypassing that virtual boundary. Nothing but the pfsense VM is attached, so no other resource has an IP or is listening on that interface. I have a bunch of prod deployments virtualizing pfsense in this way with zero issues.
 

EniGmA1987

Limp Gawd
Joined
May 2, 2017
Messages
429
You put the WAN on a dedicated NIC, that is only connected to the pfsense box. I haven't heard of any vulnerability on any main hyper-visor that allows bypassing that virtual boundary. Nothing but the pfsense VM is attached, so no other resource has an IP or is listening on that interface. I have a bunch of prod deployments virtualizing pfsense in this way with zero issues.


Spectre? Meltdown? Thats why they were so bad for enterprise. They allow escaping the virtual machine boundaries and access full system. This wont be the last time such vulnerabilities are found either.
 

Biznatch

2[H]4U
Joined
Nov 16, 2009
Messages
2,224
Spectre? Meltdown? Thats why they were so bad for enterprise. They allow escaping the virtual machine boundaries and access full system. This wont be the last time such vulnerabilities are found either.

That's a completely different case than a virtual nic, which like I said, I have not seen any vulnerabilities for. Spectre/meltdown gave access to protected RAM due to shitty intel architecture to fluff the up their speeds, requiring a software patch to fix that reduced performance. To my knowledge, it still has not been fixed on the latest gen of intel processors either..... It did not however give access to the full system, just the potential to leak secrets.
 

EniGmA1987

Limp Gawd
Joined
May 2, 2017
Messages
429
That's a completely different case than a virtual nic, which like I said, I have not seen any vulnerabilities for. Spectre/meltdown gave access to protected RAM due to shitty intel architecture to fluff the up their speeds, requiring a software patch to fix that reduced performance. To my knowledge, it still has not been fixed on the latest gen of intel processors either..... It did not however give access to the full system, just the potential to leak secrets.

The point was they are a vulnerability that allows escaping the virtual boundary of a hypervisor, and they are a very specific threat in the case of a router being run on a server through a hypervisor. It is the exact case being talked about as a possible risk of the WAN port being in hardware on the system. But if you specifically want only a vulnerability of a NIC itself in a hypervisor, here are two that affect the newer generation 3 virtual NICs of VMWare:
https://www.vmware.com/security/advisories/VMSA-2018-0027.html
CVE-2018-6981 could allow a guest user to execute code on the host, while the CVE-2018-6982 vulnerability could result in information leakage from the host to a guest.
 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
Spectre? Meltdown? Thats why they were so bad for enterprise. They allow escaping the virtual machine boundaries and access full system. This wont be the last time such vulnerabilities are found either.

Yeah kinda my train of thought, say you have a risky VM that has a port forwarded to it, it's on the appropriate vlan etc, but that VM gets hacked (which on it's own might be fine or even expected), from there they could in theory hop on the firewall and then start changing stuff. Then again you probably want a separate VM server altogether for any VMs that may be a risk. (honey pots, servers etc)
 

tangoseal

[H]F Junkie
Joined
Dec 18, 2010
Messages
9,317
This i5 Qotom is more or less what I've been looking at. I have a J3160-based version now, which is about twice as fast as the AMD mess in the APU-series boxes, and I wonder if it would be able to handle gigabit downloads myself.

There is also ProtectLI that sell rebadged Qotoms (etc.) and have i5-based six-port version for the top-end, but they seem to be really proud of them.

Probably has a Chinese spy chip in it like all of super micro does. Google for proof. Was released info by US government.
 

EniGmA1987

Limp Gawd
Joined
May 2, 2017
Messages
429
Probably has a Chinese spy chip in it like all of super micro does. Google for proof. Was released info by US government.


There is no SuperMicro spy chip, that has been proven false. Even without having testing proving the story to be false, it didnt make sense in the first place. The chip described in the story was of godlike capability. If anyone had the ability to create such a device in the size of a grain of rice, our CPUs would be leaps and bounds beyond what we have now. I mean really, has multi-gigabit Ethernet hardware (already not possible in the size of a grain of rice), has enough memory not only for its own bios but to store multiple OS files that have been corrupted so that it can replace the real ones from an OS, has a small logic core so execute code that exploits OS vulnerabilities to gain root level OS access to replace system files, and processes the keylogger files it stores and sends them back to China. All in the size of a grain of rice. Ya how about no.
 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
I'd be more worried about the Intel ME backdoor (AMD has something like that too I think) than Supermicro as it affects practically every modern PC. Sadly there does not seem to be much known about that backdoor and anyone that talks about how to disable it gets a C&D from Intel. I've briefly tried to see if I can even connect to the port and I can't, so I don't know how it's triggered.

But yeah the Supermicro one has not really been proven 100%. Well I guess the Intel one too, but that one is more certain I think.
 

acascianelli

Supreme [H]ardness
Joined
Feb 25, 2004
Messages
6,913
I'm running pfSense on an PCEngines APU2

https://pcengines.ch/apu2.htm

IIRC, these are good to about 700-800MB/s throughput so if you're running Gigabit service probably not a good option. I only have a 150MBit connection. Last I checked it runs about 8-10W. The whole setup (board, SSD, case, AC adapter) only cost me like $175.
 

klank

Killer of Killer NIC Threadz
Joined
Aug 22, 2011
Messages
2,177
I'm running pfSense on an PCEngines APU2

https://pcengines.ch/apu2.htm

IIRC, these are good to about 700-800MB/s throughput so if you're running Gigabit service probably not a good option. I only have a 150MBit connection. Last I checked it runs about 8-10W. The whole setup (board, SSD, case, AC adapter) only cost me like $175.


You will not get over 400-500Mbit of service through an APU2. I know because I have one. Basic firewall with no other filtering it would not break 500Mbits/s.
 

Big[H]

Gawd
Joined
Aug 23, 2001
Messages
774
I just got a SFF Dell Optiplex off craigslist to set up pfSense on. Has an i5 4570 and 8GB DDR3 1600 RAM, total overkill. For the NIC cards, I got a (hopefully genuine) Dell/Intel quad port PCI-e for wired and a D-Link DWA-556 for wireless N. Hopefully it will be a routing monster once those NICs come in.
 

Valnar

2[H]4U
Joined
Apr 3, 2001
Messages
3,446
How much power do those QOTOM or other fanless boxes on Amazon take? My APU2 takes less than 10 watts, but wouldn't mind an upgrade.
 

Grebuloner

[H]ard|Gawd
Joined
Jul 31, 2009
Messages
1,027
I just got a SFF Dell Optiplex off craigslist to set up pfSense on. Has an i5 4570 and 8GB DDR3 1600 RAM, total overkill. For the NIC cards, I got a (hopefully genuine) Dell/Intel quad port PCI-e for wired and a D-Link DWA-556 for wireless N. Hopefully it will be a routing monster once those NICs come in.

I built mine with an i5 4590 I got super cheap, spare parts and a MB off the F/S and an Intel i350-t4 4 port nic for WAN and 2 VLANs. Ironically I put it in one of the largest cases I own: an old Supermicro 750 server tower :cool:. You won't find bottlenecking, even with full VPN active (claim not yet tested on gigabit). What are you going to do with that internal N adapter? Not run a wireless network off it, I hope?
 

IdiotInCharge

NVIDIA SHILL
Joined
Jun 13, 2003
Messages
14,679
How much power do those QOTOM or other fanless boxes on Amazon take? My APU2 takes less than 10 watts, but wouldn't mind an upgrade.

They're usually using ultrabook parts, so the CPU is likely ~15W?

Things run off a wall wart.

[Main issue with the APU2 and further is that the AMD SOC used has low single-thread performance versus modern Core CPUs, and runs at low clockspeeds- it's great for what it is, but it is limited- if they updated it with a mobile Ryzen APU...]
 

Big[H]

Gawd
Joined
Aug 23, 2001
Messages
774
I built mine with an i5 4590 I got super cheap, spare parts and a MB off the F/S and an Intel i350-t4 4 port nic for WAN and 2 VLANs. Ironically I put it in one of the largest cases I own: an old Supermicro 750 server tower :cool:. You won't find bottlenecking, even with full VPN active (claim not yet tested on gigabit). What are you going to do with that internal N adapter? Not run a wireless network off it, I hope?

Nice! I am on gigabit but don't plan to run VPN yet. The internal N adapter will serve minor devices possibly (or give me wireless when I am not using it as a pfSense box). If it ends up being too slow, I do have a real 802.11ac router that I can put after pfSense and let it do its thing. I am wanting to put this before it because the firmwares are very buggy on it as of recent.
 
Top