SAN dual controllers & VMware iSCSI path selection, IOPS per path setting

[Thuleman]

Storage: Dual controllers, active/active, 4x 1 GB NICs per controller.
Hosts: 2x 1 GB physical NICs per host connected to storage vLAN
Physical Network: dual Cisco something or other, Flow Control on, Jumbo off
Virtual Network: standard vSwitch, 2x VMkernel, 2x physical NICs, iSCSI bound correctly

So the 8 storage NIC ports are hooked into 2 switches, 2 ports on each controller to each physical switch. Inside the storage some raid groups have controller 1 set as preferred and others have controller 2 set as preferred.

By chance I thumbed through a write-up by Nimble for their arrays (the array in question is not a Nimble, it's a JetStor) and they recommend that the path selection for a datastore within vSphere ought to be set to Round Robin rather than the default of Most Recently Used. In addition there was a mention of altering the default IOPS per path from 1000 to 0 (zero) for each datastore on each host.

Looking into it a bit further I have found that there are some other storage vendors who recommend to set the default IOPS to 1. It seems that if you set the IOPS per path to 0 then the IOPS per path are ignored altogether and the queue depth is used to determine when a path is switched. By that reasoning setting it to 1 would mean that the path switch is forced after just one operation?

How significant is monkeying with the IOPS per path setting? Likewise, does round robin rather than MRU actually provide a tangible benefit in a 2 host NIC situation considering multiple raid groups per array which are all set to prefer a specific controller?

 

[peanuthead]

I think it's dependant on the storage array and what the manufacturer suggests. What does the VMware device lookup say? (I forget the page's exact name but it is where you can look up the suggested storage setup per vendor and product.)

 

[Dytralis]

How significant is monkeying with the IOPS per path setting? Likewise, does round robin rather than MRU actually provide a tangible benefit in a 2 host NIC situation considering multiple raid groups per array which are all set to prefer a specific controller?

You'll find that screwing with the IOPS per path setting could give you an increase in IOPS, but it ends up being under very specific workloads:

http://www.yellow-bricks.com/2010/03/30/whats-the-point-of-setting-iops1/

And to answer the question about round robin: use it, you won't regret it

 

[Child of Wonder]

Any advantages to monkeying with the Round Robin settings disappear as your host count goes up. Round Robin is a path balancing mechanism to equalize IO across paths so you're not having to manually do it yourself or overloading ports on your storage array. Eventually the performance gains are outweighed by the additional management overhead of editing the RR settings. Powerpath/VE would offer load balancing and better performance but for the cost you may as well invest in a faster transport like 8Gb FC or 10Gb ethernet if you haven't already.

KISS

 

[lopoetve]

1, nimble should have corrected that - I caught it right after publication. No touchy the setting.
2. It only helps for single host / single vm per lun setups
3. don't bother, see item 1.
4. Use RR.

 

[peanuthead]

1, nimble should have corrected that - I caught it right after publication. No touchy the setting.
2. It only helps for single host / single vm per lun setups
3. don't bother, see item 1.
4. Use RR.

It's funny seeing this as Synology recommends MRU on my NAS, but I prefer RR for the reasons listed above.