• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

storage controllers, multipathing, vlans, & vswitches

T

Thuleman

Supreme [H]ardness
Joined
Apr 13, 2004
Messages
5,833
So I am having this discussion with a colleague of mine and we cannot come to a consensus on "best practices" or even on "short of best practices but standard operating procedure regardless".

Consider a storage array that has a total of five network ports, sNICm (management), and sNIC0-3 for iSCSI traffic. Consider an ESXi host which has (more than) five network ports. eNICm and eNIC0-3.

We all know that it is advisable to place NICm into a management traffic vlan (v942), no argument there. Both the storage array and the ESXi host have access to the iSCSI vlan (v1010) and the vMotion vlan (v1020).

The way I look at it is that I pair up NICs and place them as follows:
iSCSI v1010 assigned to sNIC0, sNIC1, eNIC0, eNIC1
vMotion v1020 assigned to sNIC2, sNIC3, eNIC2, eNIC3

On the ESXi host there is only one vSwitch per vlan, so eNIC0 and eNIC1 are connected to one vSwitch for v1010.

When provisioning datastores in ESXi I can check the paths to the storage and the appropriate number of paths show up without that any kind of MPIO needed to be explicitly configured (as expected in vSphere 5.1).

My colleague is passionately insisting that I am doing it wrong, so much so that I am actually questioning whether I may possibly be doing it wrong even though I didn't have any issues.

His take on this is that each on the storage array and ESXi host ought to be in its own vlan, and on its own vswitch, all else is madness. His approach would require two additional vlans and would look like this:

iSCSI (v1010) assigned to sNIC0, eNIC0, vSwitch1
iSCSI (v1011) assigned to sNIC1, eNIC1, vSwitch2
vMotion (v1020) assigned to sNIC2, eNIC2, vSwitch3
vMotion (v1021) assigned to sNIC3, eNIC3, vSwitch4

The additional vlans seem to create a lot of admin overhead, especially if extrapolated to additional storage arrays (one new vlan per new storage array port, wtf?!).

However, I am willing to entertain the possibility that there are significant benefits to the one vlan per port approach. Are there significant benefits?
 
I think your colleague is complicating things. I am not able to find any reasonable reason you should need each sNIC, etc. on its own VLAN just to have it on its own VLAN. Admin overhead big time in my mind, but it has been a long day.
 
You don't need them on different VLANs. Just throw all 2 or 4 on the same VLAN and do 1:1 mapping of vmkernels to pNICs and you're good.
 
First, you should see what the "best practice" of the storage array vendor dictates, but since you stated that it was working, then more than likely option two below is the way it should be set up.

Some iSCSI arrays, example, Dell Powervault MD, active/active require separate IP subnets per controller therefore you need to bind nics to vmkernels in each subnet.

Some iSCSI arrays, EqualLogic for example, Active/Passive controllers, don't require different subnets per controller as the Passive controller takes the IP configuration from the Active during a failover.

In the first case, yes, you need at least two VLANs for iSCSI traffic, untagged, the second case, no. In either case you don't need multiple vSwitches

Also, for vMotion, they should be in the same VLAN even if you do multi-nic vMotion...meaning 2 separate vmkernels for vMotion one nic active/one standby and vice versa on the second vMotion vmkernel.

Why does the storage array have access to vMotion?:confused:
 
Last edited:
Even though one VLAN will work for iSCSI, I still prefer to use two iSCSI VLANs so I can dictate exactly where my iSCSI traffic goes. I don't want my iSCSI traffic going cross-stack if I can help it otherwise I have some paths with one switch hop and some with two.
 
iSCSI is more redundant within the same VLAN

If you lose ports on one side and controllers on the other you can at least hop the switch stack and still make the connection

Here's the way I've always seen it done

vmnic0
vmnic1
vmnic2
vmnic3

vSwitch0
vMotion-1 vmnic0 active, vmnic1 standby, VLAN 1020
vMotion-2 vmnic1 active, vmnic0 standby, VLAN 1020

vSwtich1
iSCSI-1 vmnic2 active, vmnic3 unused VLAN 1010
iSCSI-2 vmnic3 active, vmnic2 unused VLAN 1010

PS, what storage arrays are you using?
PS, 10Gb NICS, Distributed Switches, NIOC and Converged Networking is where it's at
 
Last edited:
0V3RC10CK3D said:
iSCSI is more redundant within the same VLAN

If you lose ports on one side and controllers on the other you can at least hop the switch stack and still make the connection
Click to expand...

Unless the iSCSI storage controllers don't have two ports each, they should be cross connected to the two switches anyway so you can withstand a controller and switch failure.

Although it only adds nanoseconds of latency, I just prefer to know iSCSI-A is going through one physical switch straight to a controller and the same for iSCSI-B. I want to know what route my traffic is taking and all it takes is an extra VLAN.
 
Child of Wonder said:
Unless the iSCSI storage controllers don't have two ports each, they should be cross connected to the two switches anyway so you can withstand a controller and switch failure.

Although it only adds nanoseconds of latency, I just prefer to know iSCSI-A is going through one physical switch straight to a controller and the same for iSCSI-B. I want to know what route my traffic is taking and all it takes is an extra VLAN.
Click to expand...

Double failures are called "Max Cost Failovers" and have a demonstrably lower chance of success (although it's still very high). :)
 
Child of Wonder said:
Unless the iSCSI storage controllers don't have two ports each, they should be cross connected to the two switches anyway so you can withstand a controller and switch failure.

Although it only adds nanoseconds of latency, I just prefer to know iSCSI-A is going through one physical switch straight to a controller and the same for iSCSI-B. I want to know what route my traffic is taking and all it takes is an extra VLAN.
Click to expand...

True. Each controller would need to have multiple NIC's capable of being placed in separate VLAN's. Then it would work nice and dandy. Still too much trouble for me, our 10Gb switches have a LAG of two 40Gb connectors for 80Gb of switch to switch bandwidth.
 
0V3RC10CK3D said:
True. Each controller would need to have multiple NIC's capable of being placed in separate VLAN's. Then it would work nice and dandy. Still too much trouble for me, our 10Gb switches have a LAG of two 40Gb connectors for 80Gb of switch to switch bandwidth.
Click to expand...

I'm just old school I guess. I try to treat iSCSI networks like I would FC by creating two or more "fabrics."
 
Child of Wonder said:
I'm just old school I guess. I try to treat iSCSI networks like I would FC by creating two or more "fabrics."
Click to expand...

I wouldnt call that old school, i would call that best practice. I dont care how much bandwidth you have, separate as physically as possible your storage from your 'core' traffic. Or you just begin asking for issues.
 
4saken said:
I wouldnt call that old school, i would call that best practice. I dont care how much bandwidth you have, separate as physically as possible your storage from your 'core' traffic. Or you just begin asking for issues.
Click to expand...

Yet the "in" thing today seems to be one subnet and one VLAN. I would never set up an ISL between my FC switches and allow my storage traffic to traverse that, why would I do the same thing with my iSCSI?

I'm not saying one subnet and VLAN won't work. I just don't want paths 1 and 3 going through one switch while paths 2 and 4 go through 2. It may only be a few nanoseconds latency difference but there is a difference.

Plus, all the one subnet and VLAN gains is a little simpler set up and management. I'd rather do it "right" where I know every storage path has the same number of hops, I can easily trace where traffic is going, and all it requires is another VLAN to keep track of.

Hell, I was on an implementation a few months ago with a certain ZFS storage vendor and they said "we're just going to put all our 10Gb ports in an LACP trunk with one IP and call it good!"

p0ye.jpg
 
Don't forget ISL limits, loops, etc - and path management systems get confused sometimes by that too.

KISS when it comes to storage fabrics - make it SIMPLE.

Oh, and that LACP trunk - waffles at SCSI failovers.
 
4saken said:
I wouldnt call that old school, i would call that best practice. I dont care how much bandwidth you have, separate as physically as possible your storage from your 'core' traffic. Or you just begin asking for issues.
Click to expand...

I don't think storage vs network traffic separation was ever in question.
 
As mentioned already it comes down to what SAN vendor you're working with. Equallogic for example runs off of the flat storage network(single vlan). In this case seperate VLANS would not work. EMC on the other hand can work with more of a mesh fabric like you would do with FC fabrics.

One thing does come to mind though, how is your storage switching configured? Do you have stacking implemented or discrete switches?

Another thing is how does ESX route traffic on interfaces in seperate VLANs that share the same IP address space?

IE
vmkiSCSI0 is on vSwitch 0 is tagged with 555 with an address of 192.168.20.5
vmkiSCSI1 is on vSwitch 1 is tagged with 666 with an address of 192.168.20.6
vmkiSCSI2 is on vSwitch 2 is tagged with 777 with an address of 192.168.20.7

Storage interface A is on 555 with an address of 192.168.20.55
Storage interface B is on 666 with an address of 192.168.20.66

What is keeping ESX from sending traffic for the interfaces you've placed on 555 out the 777 interface. Also at the storage switch level are you replacing the tags with a single tag?
 
Last edited:
geiger said:
As mentioned already it comes down to what SAN vendor you're working with. Equallogic for example runs off of the flat storage network(single vlan). In this case seperate VLANS would not work. EMC on the other hand can work with more of a mesh fabric like you would do with FC fabrics.

One thing does come to mind though, how is your storage switching configured? Do you have stacking implemented or discrete switches?

Another thing is how does ESX route traffic on interfaces in seperate VLANs that share the same IP address space?

IE
vmkiSCSI0 is on vSwitch 0 is tagged with 555 with an address of 192.168.20.5
vmkiSCSI1 is on vSwitch 1 is tagged with 666 with an address of 192.168.20.6
vmkiSCSI2 is on vSwitch 2 is tagged with 777 with an address of 192.168.20.7

Storage interface A is on 555 with an address of 192.168.20.55
Storage interface B is on 666 with an address of 192.168.20.66

What is keeping ESX from sending traffic for the interfaces you've placed on 555 out the 777 interface. Also at the storage switch level are you replacing the tags with a single tag?
Click to expand...

Hello, using the same subnet across isolated networks will cause issues. When ESXi initiates traffic connections it actually does the handshake over the lowest vmk number.

If network connections are not possible they need to be on a separate subnet. VLAN optional but preferred to minimize the broadcast domain.
 
0V3RC10CK3D said:
Hello, using the same subnet across isolated networks will cause issues. When ESXi initiates traffic connections it actually does the handshake over the lowest vmk number.

If network connections are not possible they need to be on a separate subnet. VLAN optional but preferred to minimize the broadcast domain.
Click to expand...

that's fixed in 5.5 and (IIRC) later patches 5.1
 
0V3RC10CK3D said:
Hello, using the same subnet across isolated networks will cause issues. When ESXi initiates traffic connections it actually does the handshake over the lowest vmk number.

If network connections are not possible they need to be on a separate subnet. VLAN optional but preferred to minimize the broadcast domain.
Click to expand...

I know, I was trying to allude to the point that if the storage array doesn't require it, or there isn't some work done on the storage switching, then it might not work.
 
You must log in or register to reply here.
Back
Top