Esxi Pfsense VM

Modder man

[H]ard|Gawd
Joined
May 13, 2009
Messages
1,771
I have my home router LAN connected to the WAN port of the Pfsense VM. Currently all of my VM's are pulling dhcp from the home router not from the Pfsense box. Do I have the vSwitch mis-configured or?

 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,700
It's not clear from your post if you have a problem, or what you are trying to do :( If you are wanting the VMs to pull dhcp from pfsense, they need to be on the pfsense LAN, not the WAN. Why are you even doing this? Just make the pfsense connect to the cable modem or whatever and go with that...
 

Modder man

[H]ard|Gawd
Joined
May 13, 2009
Messages
1,771
I want the home network to remain intact. I cant do that if i connect the pfsense box directly to the modem. the vms are on the lan from the picture of the vswitch?
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,700
I'm not being clear. The home router serves up addresses and internet access. Does it do anything magic that can't be done by the pfsense VM? What does 'remain intact' mean? What exactly are you trying to accomplish?
 

rtangwai

[H]ard|Gawd
Joined
Jul 26, 2007
Messages
1,369
Looks right to me, are you using a separate subnet for the LAN side vs. the WAN side?
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,700
Well, I'm not so sure. If the intent is the VMs pull DHCP from the home router, the pfsense would need to be acting as a transparent bridge or something. Without knowing what he is trying to accomplish here, it's hard to really make a helpful suggestion...
 

dbwillis

[H]F Junkie
Joined
Jul 9, 2002
Messages
8,195
Shouldnt it be the modem (cable/DSL, etc) connected to VMNic0 w/virtual switch connecting only to the WAN of the PFSense VM, then the LAN of the PFSense VM, connects to VMNic1 w/virtual switch

Turn off DHCP on the router too.
(if the VM switch0 is connected to the LAN port of the router, anything connected to that vswitch can grab DHCP from the router, not PFSense)
 

Modder man

[H]ard|Gawd
Joined
May 13, 2009
Messages
1,771
The home router hands out 192.168.X.X range and the pfsense box hands out 10.10.X.X This is the way my lab used to be configured but i was using a physical pfsense box at the time.
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,700
You still haven't told us what you are trying to do though. So is the idea that the pfsense sits on the home lan network, and serves up IP addresses on its lan interface? So the two routers are (effectively) daisy-chained together? e.g. the VMs in ESXi are segregated on a separate virtual network? If so, I assume from your post, that it isn't working? What specifically isn't working?
 

Modder man

[H]ard|Gawd
Joined
May 13, 2009
Messages
1,771
That is what i am attempting to do. I am not getting and responses from the pfsense box. as in no DHCP or even if i set a machine static i am unable to communicate with the pfsense box. All vms are still receiving thier addresses from the home router.
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,700
ah, i think i see. i think you have the switches messed up. the management network for ESXi needs to be somewhere on your lan (usually) so you can get to it with a real workstation. i think you need to rename the WAN vswitch to something like 'Virtual LAN' and move all of the other VMs to that switch. Then interchange the vswitches the two pfsense vnics are on. e.g. you want the pfsense WAN vnic to be in the ESXi LAN vswitch, and the pfsense LAN vnic to be in the ESXi Virtual LAN vswitch. All the VMs also go in the Virtual LAN vswitch, so they talk to the DHCP server on the pfsense "LAN". Make sense?

p.s. if you do this, I don't think you need an actual nic in the 'Virtual LAN' vswitch?
 

Modder man

[H]ard|Gawd
Joined
May 13, 2009
Messages
1,771
Yeah i was wondering that. It occured to me that perhaps i didnt need a nic assigned to the "virtual Lan switch" That said i was considering also running that virtual LAN nic to a physical switch in my rack to run all the hardware in my rack on the pfsense router.
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,700
You could certainly do that, yes. But that's orthogonal to the design error you have here. I'd give that a go...
 

Modder man

[H]ard|Gawd
Joined
May 13, 2009
Messages
1,771
With this proposed network design how many physical nic's would i be using? 2 correct?
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,700
Correct. The LAN vswitch/nic would go to your normal home LAN. The Virtual LAN vswitch/nic would plug into the switch servicing your rack hosts...
 

efishta

Limp Gawd
Joined
Oct 12, 2004
Messages
184
If you want to run pfsense behind the cable modem/router, and not touch the rest of your network, it is going to have to be double NATted, with the cable modem LAN connection going into the WAN connection of pfSense (this part you have correct)
Additionally, since this new segment it is going to be accessed only by the VMs, there is no reason for your pfSense LAN vSwitch to be connected to physical NIC (pNIC). This is where the issue is arising above.

I hope I am understanding what you are trying to do correctly. Here is the general configuration I am picturing for your current setup:

3 vSwitches:

1. Regular LAN access and ESXi management, connected to 1 pNIC then to physical LAN switch
2. pfSense WAN, connected to 1 pNIC then to physical LAN switch - this is how pfSense gets its internet access. This is sort of redundant, as you can actually just connect it to the above vSwitch and share that pNIC for both host management purposes and to feed internet into the WAN port of pfSense.
3. pfSense LAN - this one does not need a pNIC connected to it, as it is only being accessed by the VMs, which are then getting THEIR OWN connectivity from pfSense through this vSwitch.
.
Currently, even assuming the connections are set up and verified correctly in the pfSense VM and the pfSense LAN is connected to to the vSwitch that is running your management network, then the VMs are likely picking up DHCP IPs through the cable modem router because they have a direct connection to that LAN segment through this vSwitch.
By isolating the VMs into a vSwitch that has no physical connection to your home network, you're making diagnosing possible. Currently you can't even tell if your pfSense virtual router is working (without packet tracing I suppose).
 
Last edited:

Modder man

[H]ard|Gawd
Joined
May 13, 2009
Messages
1,771
ok here is what i have now. not getting a wan address on the pfsense box. I do have a VM getting the proper address from the Pfsense box LAN though

 

efishta

Limp Gawd
Joined
Oct 12, 2004
Messages
184
Cool, that's half the battle.

As for your WAN not picking up an address, I'm assuming you've set up the pfSense WAN interface to pick up an address from your cable modem via DHCP, right? Or static assignment with the correct settings for your cable modem's 192.168.1.x subnet?

Can you post a screenshot of your pfSense WAN configuration page?

Try enabling Promiscuous mode on the WAN vSwitch. I don't think this will do anything in this scenario, give it a shot any way. In my own virtualized pfSense setup, the WAN port does not have Promiscuous mode enabled - though I am directly connected to the WAN in pfSense.

Oh, and it should go without saying, but reboot your networking equipment, just in case. Then start actually diagnosing :)
 

efishta

Limp Gawd
Joined
Oct 12, 2004
Messages
184
Awesome, glad you got it resolved. It's odd that it didn't work in the Physical LAN vSwitch1, though, provided everything else is identical between vSwitch configs and connectivity.
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,700
I assume it didn't work in vswitch1 because the home router is in vswitch0 (the management network), which is presumably the rest of his home network - by default pfsense would have had dhcp set for the WAN nic, so it would be expected to get an IP from the home router. I don't understand why you created an extra switch - there was no need for that. Delete vswitch2, rename vswitch1 to LAB Virtual LAN and move the two guys on vswitch2 to vswitch1 (that's what I was suggesting to begin with...)
 

efishta

Limp Gawd
Joined
Oct 12, 2004
Messages
184
I assume it didn't work in vswitch1 because the home router is in vswitch0 (the management network), which is presumably the rest of his home network - by default pfsense would have had dhcp set for the WAN nic, so it would be expected to get an IP from the home router. I don't understand why you created an extra switch - there was no need for that. Delete vswitch2, rename vswitch1 to LAB Virtual LAN and move the two guys on vswitch2 to vswitch1 (that's what I was suggesting to begin with...)

I was operating under the assumption that vSwitch0 and vSwitch1 were connected to diff ports on the home LAN switch. I probably should not have.

As for the unnecessary vSwitch, you are right, of course. I mentioned as much in my previous post, as well. I was attempting to separate all the physical connections and vSwitches so the OP would have a clearer picture of what and why each connection was where it needed to be (and a bit for the sake of best practices - isolating Mgmt Network)

EDIT: Yeah... I don't think there was a home switch mentioned anywhere in the thread. Just my fault for assuming ;)
 

Modder man

[H]ard|Gawd
Joined
May 13, 2009
Messages
1,771
Both of them were hooked up to the home LAN network that is what threw me off. As efishta said I was trying to isolate the management network from either of the other networks.
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,700
Curious why it didn't work then. Probably some funky vswitch setting. I guess I'm confused - if the hosts on the physical lan can't talk to the management network subnet, you cant' access/manage the vsphere host. If they can, it isn't isolated. Not sure what you were getting at...
 

efishta

Limp Gawd
Joined
Oct 12, 2004
Messages
184
Curious why it didn't work then. Probably some funky vswitch setting. I guess I'm confused - if the hosts on the physical lan can't talk to the management network subnet, you cant' access/manage the vsphere host. If they can, it isn't isolated. Not sure what you were getting at...

Poor choice of words on my part. I used "isolate" when I should have said "separate" (as in, a separate, dedicated pNIC for Management purposes).
 

The_Moves

Limp Gawd
Joined
Oct 8, 2008
Messages
183
I have a DSL modem with the same setup, however when I have pFsense do the PPPoE authentication, it will randomly fail the authenticate, causing me to have to rebooot both the DSL Modem and pfSense VM. To be able to have pfSense authenticate my PPPoE credentials, I had to setup my modem up in bridge mode. In the end though, I put my modem in its default setup, having it authenticate my PPPoE credentials, and put my pfSense with a static IP in the modems DMZ. I tried this with my cable before switching to DSL, with the Cable modem in passthrough mode, and it worked without issue.
 

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
32,873
Since my local computers on my home network interact with my guests on my ESXi box all the time, I just keep them on the same network,

I have eliminated the consumer router all together in favor of pfSense.

Just in case there are any security concerns with vswitches, I have passed through one of my server nics directly to the pfsense guest and use that as wan.

Over the years I have used two separate setups for the lan side.

1.) pfsense lan port -> vswitch (with guests) -> second server NIC -> external hardware switch for lan.

2.) pfsense lan port is direct forwarded second NIC -> external hardware switch (with home clients) -> back in to internal viswitch using a third NIC.

So, the advantage of #1 is that it is cleaner, uses fewer ports / pcie slots hardware

#2 gets (marginally) better latencies for home clients (virtual nics and virtual switches add more latency than direct forwarded nics and hardware switches). The difference is extremely tiny though. (0.1 to 0.2ms?) The downside of this approach is that you need to install an additional NIC though, as I have yet to encounter a server motherboard with three NIC's.

In the end I settled for #2, because I installed a second quad port NIC anyway, because I wanted to increase bandwidth to the server using link aggregation.

In either case, most better routers will allow you to turn them into dumb wireless access points instead of routers, and let you rely on pfSense for DHCP/gateway.

Eventually I just got rid of my consumer router all together, and started using a Ubiquiti Unifi wireless access point. They perform SO much better than consumer Wifi solutions in areas like mine where the 2.4 ghz band is highly congested.
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,700
+1 for the unifi units. I have 2 of them - one upstairs and one downstairs. I run pfsense totally virtual too. Works just fine...
 

Keljian

[H]ard|Gawd
Joined
Nov 7, 2006
Messages
1,386
Zarathustra[H];1041886318 said:
I have eliminated the consumer router all together in favor of pfSense.

Just in case there are any security concerns with vswitches, I have passed through one of my server nics directly to the pfsense guest and use that as wan.

Over the years I have used two separate setups for the lan side.

1.) pfsense lan port -> vswitch (with guests) -> second server NIC -> external hardware switch for lan.

2.) pfsense lan port is direct forwarded second NIC -> external hardware switch (with home clients) -> back in to internal viswitch using a third NIC.

#2 gets (marginally) better latencies for home clients (virtual nics and virtual switches add more latency than direct forwarded nics and hardware switches). The difference is extremely tiny though. (0.1 to 0.2ms?) The downside of this approach is that you need to install an additional NIC though, as I have yet to encounter a server motherboard with three NIC's.

.

Wow.. just Wow.

Points:
1. If there were security concerns with vswitches there would be many businesses in a lot of trouble. You are not operating a bank from home. Best use virtual switches where they are meant to be used
2. What you're doing is unnecessarily using hardware and losing features (eg vmotion)
3. Each gigabit hop is about 0.3-0.7 ms - so you're adding latency (more than you're suggesting) - test it with hrping http://www.cfos.de/en/ping/ping.htm - Total latency with your proposed setup would be in the order of 2-3ms at a guess
4. Paravirtualised nics have less latency through esxi - where possible, use vmxnet3/open-vm-tools/vmtools - long and short of it is that you're talking about 10 microseconds latency difference between passthrough and paravirtualised (https://www.vmware.com/files/pdf/techpaper/network-io-latency-perf-vsphere5.pdf) - Only microtransaction traders should be worried about this.
5. Latency across vswitches between VMs is extremely small

The solution is simple:
Nic-Wan (vswitch0) - >PFSense
Nic-Lan (vswitch1) -> PFsense/Other VMs/network

You will save 10 microseconds of latency if you passthrough the WAN, and another 10 microseconds if you passthrough the LAN - IMO this is within margin for error.

IF you want to isolate your lab from the rest of the network set up vlans using PFsense and a vlan aware switch..
 
Last edited:
Joined
Nov 2, 2005
Messages
615
I would add a ebay intel nic and use it for dedicated pfsense pass-through

one of the ports would go to the modem, the other to the switch. I would give pfsense another internal one to the virtual switch
 

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
32,873
Wow.. just Wow.

Points:
1. If there were security concerns with vswitches there would be many businesses in a lot of trouble. You are not operating a bank from home. Best use virtual switches where they are meant to be used
2. What you're doing is unnecessarily using hardware and losing features (eg vmotion)
3. Each gigabit hop is about 0.3-0.7 ms - so you're adding latency (more than you're suggesting) - test it with hrping http://www.cfos.de/en/ping/ping.htm - Total latency with your proposed setup would be in the order of 2-3ms at a guess
4. Paravirtualised nics have less latency through esxi - where possible, use vmxnet3/open-vm-tools/vmtools - long and short of it is that you're talking about 10 microseconds latency difference between passthrough and paravirtualised (https://www.vmware.com/files/pdf/techpaper/network-io-latency-perf-vsphere5.pdf) - Only microtransaction traders should be worried about this.
5. Latency across vswitches between VMs is extremely small

The solution is simple:
Nic-Wan (vswitch0) - >PFSense
Nic-Lan (vswitch1) -> PFsense/Other VMs/network

You will save 10 microseconds of latency if you passthrough the WAN, and another 10 microseconds if you passthrough the LAN - IMO this is within margin for error.

IF you want to isolate your lab from the rest of the network set up vlans using PFsense and a vlan aware switch..

In my own testing I have found the latencies added by using vswitches to be 10x higher than you suggest.

Personally I have no use what so ever for vmotion. I'm not running a datacenter. I have a single free license server, and use ESXi for one reason only, and that is consolidation of my home servers.
 
Top