HA/vMotion/etc with physical NICs?

TeeJayHoward

Limpness Supreme
2FA
Joined
Feb 8, 2005
Messages
13,415
I've got two basically identical ESXi hosts, each with an I350T4 NIC in them. Is it possible to give each VM a port on the NIC and still have the VMs swap over to the remaining host when one goes down? If I understand correctly, passing through the NIC ruins any sort of failover for the VM. What if I created a "Virtual Machine" connection and associated it with a single port on the NIC? Would that work if I were to pull the plug on the host? Should I create a new connection, add all four ports to it, and let ESXi sort out the pathing?

Basically, how should I set up my physical and virtual networks to take advantage of every feature ESXi offers?
 
Don't pass through hardware devices. That breaks HA/vMotion.

Very few good reasons to even do it.
 
Don't pass through hardware devices. That breaks HA/vMotion.

Very few good reasons to even do it.
Server 2012 has that whole SMB multichannel/multipath/whatever-it's-called thing where I can combine the bandwidth of multiple NICs. Previously, I had just passed through the NICs to the VM, and everything worked great. However, now that I'm fiddling around with HA, it's causing (obvious) issues. Plus, until I can afford a decent NAS, I'm stuck with my drives sitting in the hosts. Gotta pass through the controller.

Also, I wanted to ask... Is HA/vMotion for the hosts, or the VMs? Can you have 1 VM with passthrough that doesn't fail over, and 3 more without that do?
 
HA will restart any VM it can on a surviving host. By "it can" I mean VMs not tied physically to a host and assuming there are resources on surviving hosts. So..yes.
 
You can do multiple nics in the guest if you want without passing through the physical hardware - individual virtual machine networks bound to single network cards is the easiest way to make it work like it would in the "physical" world - but route based on whatever (depending on your network hardware) would also do pretty well since the vnic isn't actually limited by any kind of speed (if you're using VMXNET), and we'll handle the rest outside the guest if you have more than one thing talking to it (and htus more than one thing to hash).

Don't overthink it ;) At least that's my understanding - I'm not a network Guru.
 
Looks like I'll be virtualizing the network as well. Is there an advantage to supplying more than one NIC to vMotion?
 
vMotion can use as many connections as you give them as long as the others hosts also have that many connections.
 
Do you REALLY care how fast vmotion works? Unless you're running VMs with gigs upon gigs of ram, it really doesn't make that much of a difference :)
 
Do you REALLY care how fast vmotion works? Unless you're running VMs with gigs upon gigs of ram, it really doesn't make that much of a difference :)

It matters when it's after hours and you have to update a cluster of hosts with only 1Gb of VMotion bandwidth. :)

MOAR BECAUSE MOAR.
 
It matters when it's after hours and you have to update a cluster of hosts with only 1Gb of VMotion bandwidth. :)

MOAR BECAUSE MOAR.

Yeah, this. Did a N1Kv upgrade one night at a customer. Evacuating all VMs off the hosts took 25 to 30 mins each and they had 6 hosts. So figure 3 hours off hours watching status bars update....
 
For production sure :p For a lab (like this sounds like)? :p
Oh, it's definitely a lab. I just now have 3x I350T4s that I bought intending to pass through to the VMs, and I'm realizing that I don't really need them. Selling 'em would be a pain in the arse, so I'm trying to find a use for 'em. I'm starting to wish I had of gone the Infiniband route. 32x the speed at 1/2 the cost.
 
For production sure :p For a lab (like this sounds like)? :p

In my lab I'm using multi-NIC VMotion for a total of 2Gb when 1Gb would suffice. Even in my lab when it's time to patch or upgrade my hosts I don't want to wait any longer than I have to for them to clear. :)

Anyway, that's just personal preference. For a lab environment there are definitely no compelling reasons outside of convenience and practice to go with more than 1Gb.
 
I don't like dedicating physical ports to specific jobs besides storage if I can help it, for example if you can use LTB. Are you using NFS or iSCSI?
 
Last edited:
Don't pass through hardware devices. That breaks HA/vMotion.

Very few good reasons to even do it.

You're right that it does block those, but you're wrong about the few good reasons statement. All of those features are great features very important in their own scenarios. Performance of full virtualization is always going to be worse than PCI passthru - just that in many cases it's still good enough. But there are plenty of scenarios for which it's not that good.
 
He meant LBT (Load Based Teaming). On a distributed switch you can change the load balancing mode to LBT which will automatically move traffic to less busy uplinks should an uplink become too busy.

Ah! I know that acronym. Thanks CoW.
 
Yep, LBT (late night typo). It works best with when you have many vmkernel ports so it can spread them around. If you are still in the Ent Plus trial period or have NFR license I would definitely try it. Make all four ports on that i350 vDS uplinks and go crazy with vmkernel ports.
 
Yep, LBT (late night typo). It works best with when you have many vmkernel ports so it can spread them around. If you are still in the Ent Plus trial period or have NFR license I would definitely try it. Make all four ports on that i350 vDS uplinks and go crazy with vmkernel ports.

That's what I do in my home lab. A single VDS with 4 uplinks running Mgmt, FT, NFS, VMotion, and VM traffic. LBT moves them to various uplinks as needed (except VMotion since I'm using multinic VMotion).
 
You're right that it does block those, but you're wrong about the few good reasons statement. All of those features are great features very important in their own scenarios. Performance of full virtualization is always going to be worse than PCI passthru - just that in many cases it's still good enough. But there are plenty of scenarios for which it's not that good.

Shens (aka, prove it). ;)

While it'll always be "worse", at this point we're talking less than a percentage point or two for most things (heck, we're virtualizing HPC loads now with less than a few percent loss off the top). SR-IOV is going to eliminate that even more.
 
You're right that it does block those, but you're wrong about the few good reasons statement. All of those features are great features very important in their own scenarios. Performance of full virtualization is always going to be worse than PCI passthru - just that in many cases it's still good enough. But there are plenty of scenarios for which it's not that good.

Love to see those "plenty of scenarios". Where are you hitting limits on this with the hypervisor? Not theoretical...where are you hitting them?
 
Yep, LBT (late night typo). It works best with when you have many vmkernel ports so it can spread them around. If you are still in the Ent Plus trial period or have NFR license I would definitely try it. Make all four ports on that i350 vDS uplinks and go crazy with vmkernel ports.

The only downside to LBT is that a single vmkernel or VM can only use one NIC's worth of bandwidth. Behind the scenes it is the same as Hashing by Virtual Port ID except it'll move some Port IDs/MACs to lesser used NICs. But if you have a VM that talks to 100 clients you are probably better serviced with IP Hashing as that can hash conversations over all available NICs.

But I'm a big fan of LBT. It's simple. It's the only proactive hashing mechanism. It works for 98% of people.
 
The only downside to LBT is that a single vmkernel or VM can only use one NIC's worth of bandwidth. Behind the scenes it is the same as Hashing by Virtual Port ID except it'll move some Port IDs/MACs to lesser used NICs. But if you have a VM that talks to 100 clients you are probably better serviced with IP Hashing as that can hash conversations over all available NICs.

But I'm a big fan of LBT. It's simple. It's the only proactive hashing mechanism. It works for 98% of people.
We got 10Gb at work last year (Nexus 5000 baby) so it's a no brainer. But you make a great point, it would be cool if they could make it work with link aggregation so you can pick and chose between LBT and IP hashing per port group while still using all your uplinks. Although I'm not sure if that is even possible?
 
Last edited:
Back
Top