NIC bonding without switch

[mamruoc]

Hello,

I am setting up a NAS using OpenIndiniana to serve all files for my VMware ESXi 4.1 server.
I have chosen to use NFS/SMB so all the VMs can access the different files, mainly a large storage containing all the stuff.

My question is: 1 GB NIC will may be easily saturated but the VMs, so I need to bond the nics.

Is it possible to connect bonded nics between these two servers without having to connect them through a switch?

 

[PigLover]

Question is better asked in the "Networking and Security" forum.

Short answer: yes, you can run the NICs in a "back to back" configuration if you use static IP addressing. And you can team them. But I don't think you will get the results you desire.

Longer answer: There are many types of "NIC Bonding" (better name: teaming) available and both ends of the link need to be the same to have good results. VMWare supports a couple of types of teaming that are better suited to reliability and resiliency designs than getting increase bandwidth out of the path. Basically, because they only implement layer-2 teaming in its simplest forms, traffic between any two nodes in the network can never exceed the speed of a single link. In the VMWare case they "hash" by the "virtual" MACid or IPaddress so a "node" is a VM rather than the physical machine. This means that if you have several VMs and one NAS then each VM could get up to 1Gb of bandwidth to the NAS, but no single VM (or the VMk) would ever see more than that no matter how many NICs you include in the team.

If that does what you want - great.

If not, I suggest embedding the NAS onto your VM server as a VM itself (the "all in one" approach). In this case, the "Virtual" LAN between the VMs and the NAS will report itself to be a 10Gbe lan (and actually run even faster than that). Alternatively, you could always get 10Gbe NICs and run them back-to-back between the machines,

 

[tangoseal]

What kind of team do you desire?

802.3ad requires a switch that supports it to my knowledge.

However I never tried it direct to a pc using crossover cabling.

Please share your results if you try it.

 

[/dev/null]

if you have a linux box, try "modinfo bonding" for bonding module info. Should have the mdoes it supports as well.

 

[mamruoc]

First of all, thanks for taking time to answer me even I have posted in wrong forum.

Short answer: yes, you can run the NICs in a "back to back" configuration if you use static IP addressing. And you can team them. But I don't think you will get the results you desire.

Why? will I get worse performance than going through a switch or is it based on the info about how vmware is handling "teaming"?

Longer answer: There are many types of "NIC Bonding" (better name: teaming) available and both ends of the link need to be the same to have good results. VMWare supports a couple of types of teaming that are better suited to reliability and resiliency designs than getting increase bandwidth out of the path. Basically, because they only implement layer-2 teaming in its simplest forms, traffic between any two nodes in the network can never exceed the speed of a single link. In the VMWare case they "hash" by the "virtual" MACid or IPaddress so a "node" is a VM rather than the physical machine. This means that if you have several VMs and one NAS then each VM could get up to 1Gb of bandwidth to the NAS, but no single VM (or the VMk) would ever see more than that no matter how many NICs you include in the team.

If that does what you want - great.

If not, I suggest embedding the NAS onto your VM server as a VM itself (the "all in one" approach). In this case, the "Virtual" LAN between the VMs and the NAS will report itself to be a 10Gbe lan (and actually run even faster than that). Alternatively, you could always get 10Gbe NICs and run them back-to-back between the machines,

I tried the all in one approach, with a Intel Q6600 and 8 GB RAM and multipe PCI+PCIe cards, the performance was so bad that I couldn't serve 2 concurrent DVD backup isos to the LAN without stutterings...

I have now maniac of a virtual host, and upgraded my NAS with an Intel SASUC8I. 8 GB RAM should also be enough for ZFS, that's why I had this problem.

I haveonly 2 PCI and 1 PCIe x1 slot left on the NAS. I was thinking of adding 2 Intel Pro 1000GT Desktop NICs to do LACP (802.3ad) using 3 NICs at each end...

 

[mamruoc]

What kind of team do you desire?

802.3ad requires a switch that supports it to my knowledge.

However I never tried it direct to a pc using crossover cabling.

Please share your results if you try it.

I think I will use a switch, if I max out the link, I may try crossover cables.

 

[mamruoc]

if you have a linux box, try "modinfo bonding" for bonding module info. Should have the mdoes it supports as well.

Mm, unfortunately I will be suing VMware ESXi and OpenIndiana (Solaris)

 

[Red Falcon]

if you have a linux box, try "modinfo bonding" for bonding module info. Should have the mdoes it supports as well.

I might have to give this a try, thanks!

 

[PigLover]

First of all, thanks for taking time to answer me even I have posted in wrong forum.

Your welcome. I wasn't bashing when I suggested the networking forum - just noting that more people who have the expertise to help you are regular readers/responders there.

Why? will I get worse performance than going through a switch or is it based on the info about how vmware is handling "teaming"?

No. Performance won't be worse than with a single NIC. But it probably won't be better either. Most people think that if they have 4 1Gbe NICs in a team or bonded group that they can achieve 4Gbe of throughput. That isn't quite right. What you can achieve are 4 streams of 1Gbe each. In a home VMware setup it is rather unlikely that you have enough concurrent usage from each VM to the NAS at the same time to actually see improved performance over having a single NIC. What you will get is improved reliability/resiliency against a link failure - but copper NICs don't fail all that often and - for you, without a switch in the path - all you have left to fail is a piece of wire.

If you were talking about a NAS serving several dozen client machines spread across an enterprise then you might have something where teaming would be worthwhile.

I tried the all in one approach, with a Intel Q6600 and 8 GB RAM and multipe PCI+PCIe cards, the performance was so bad that I couldn't serve 2 concurrent DVD backup isos to the LAN without stutterings...

I have now maniac of a virtual host, and upgraded my NAS with an Intel SASUC8I. 8 GB RAM should also be enough for ZFS, that's why I had this problem.

I have only 2 PCI and 1 PCIe x1 slot left on the NAS. I was thinking of adding 2 Intel Pro 1000GT Desktop NICs to do LACP (802.3ad) using 3 NICs at each end...

There's nothing wrong with this. Just don't expect to get 3Gbe of throughput out of it.

 

[mamruoc]

No. Performance won't be worse than with a single NIC. But it probably won't be better either. Most people think that if they have 4 1Gbe NICs in a team or bonded group that they can achieve 4Gbe of throughput. That isn't quite right. What you can achieve are 4 streams of 1Gbe each. In a home VMware setup it is rather unlikely that you have enough concurrent usage from each VM to the NAS at the same time to actually see improved performance over having a single NIC. What you will get is improved reliability/resiliency against a link failure - but copper NICs don't fail all that often and - for you, without a switch in the path - all you have left to fail is a piece of wire.

If you were talking about a NAS serving several dozen client machines spread across an enterprise then you might have something where teaming would be worthwhile.

Oki, I see the point, I think I'll stick to 1 NIC at first, if the link seem to be max, I can start trying to use LACP.

Btw, if connected to a switch how does LACP bonded hosts work with non-LACP capable hosts?

 

[PigLover]

Oki, I see the point, I think I'll stick to 1 NIC at first, if the link seem to be max, I can start trying to use LACP.

Btw, if connected to a switch how does LACP bonded hosts work with non-LACP capable hosts?

The same way any other hosts with different "speeds" communicate through a switch (e.g., a 100Mbit host and a 1Gbit host one the same switch). In one direction, no problem at all...in the other, well, the switch sends all it can and if there is too much volume it drops the rest. The layer-3/4 protocols (TCP, etc) are expected to manage the flow control.

 

[dandragonrage]

Ugh. Never run a NAS in VMWare. Disk performance will be awful.

UNLESS

You use a separate RAID controller for the NAS, have VT-d available, and pass through that card to the NAS. ESX can do this, though Xen is better for it.

Without VT-d, you cannot get good disk performance out of VMWare. Period.

 

[danswartz]

How is xen better? Doesn't need a RAID controller (unless he wants RAID), in some cases the sata controller on the motherboard can be passed thru (I do this with the cougar point sata controller on my x9scl).

 

[dandragonrage]

How is xen better? Doesn't need a RAID controller (unless he wants RAID), in some cases the sata controller on the motherboard can be passed thru (I do this with the cougar point sata controller on my x9scl).

You can't only pass through certain ports of it unless it presents itself as multiple different devices to the host... And you can't pass through a controller that you boot the VM from. My P67 SATA controller is presented as one device, at least to Windows. Though I suppose I could try passing through the JMicron or Marvell controllers if I wanted to use this computer for VMs...

Xen has far more complete VT-d support than VMWare. You're not limited to 2 devices and you can even pass through video cards (though the support isn't perfect yet).

 

[danswartz]

I'm not sure I get your point. The same caveats you listed (all ports or none and no use by the hypervisor) also apply to a separate raid card, no? As far as a two device limitation, and the video card restriction, the point being discussed was virtualizing a storage system - if you have two controllers with 8 ports each, that is 16 hard drives you can present to the NAS/SAN - hardly crippling, and the video card issue is even less germane, no? Re-reading your post, I think what you were getting at was that you need something to boot the virtualized VM from - if so, yes, I agree. I have a low-end LSI card (4-port) which I use as the datastore for esxi, and the OI VM lives there. ESXi itself boots and runs from flash drive.

 

[dandragonrage]

What I meant was that the RAID card would be for the NAS VM and that onboard would be for VMWare and the other VMs.

As for the VT-d stuff, I was simply saying that Xen has more complete VT-d support. You should be using VT-d for network cards as well, IMO. There could be two storage networks, one for internal (10Gbps if possible) and a VT-d gigabit card to your switch. If you only need one NIC then you'll be okay with HD controller + NIC passed through, but that's not enough for all setups... The video card part is not important for a server - no - just augmenting that Xen has more complete VT-d support.

 

[danswartz]

Not sure it's really that much of a win to use a nic in vt-d mode, unless you have 10gbe, is it?

 

[mamruoc]

By having an elder XENON cpu with no VT-d and wishing to run Windows in some of the VMs forced me to go with VMware to begin with...