iSCSI and MPIO in ESX4

cyr0n_k0r

Supreme [H]ardness
Joined
Mar 30, 2001
Messages
5,360
From my previous threads I have setup a new vswitch using vkernel. Added 4 physical NIC's to this vswitch and configured it for round robin. I have enabled iSCSI and setup the initiator and when done I see 2 nic's active (i/o) and 2 nics standby.

However this article seems to suggest that there is even more configuration necessary to get true MPIO. Can anyone confirm if all this is necessary, or if my above setup is sufficient for failover and multipathing.

http://virtualgeek.typepad.com/virt...-post-on-using-iscsi-with-vmware-vsphere.html
 
There's more needed. That gets you basic failover, but it's not gonna be the cleanest, and the logs will get confusing. Do the esxcli commands :) Lock it so those nics and vmkernel ports (hint, you need 1-1 mappings) are for iscsi.
 
So following that guide will do what you suggest? Or more is needed past what the guide shows you?
 
Yes, you must follow that guide (or take similar steps) to use iSCSI MPIO, including round robin. It's always better to use storage layer failover/pathing than network layer, especially since you will seldom see more than a single NIC worth of performance from the network.

You'll need at least two NICs with separate addresses on both ESX and storage server/array in order for this to be effective, however.
 
Ok, I've followed the guide and I believe everything is working correctly. However my only concern is that when I go to manage paths under the storage configuration for the iscsi san, I am seeing paths that say they are active i/o, however the target field is blank. Is that normal?

Here is a screen shot of every path that shows up as I scroll down.

iscsi1.jpg

iscsi2.jpg

iscsi3.jpg
 
That seems a bit odd, as it looks to me like that's 16 paths for a single target and lun. Did you set your vmknics to only use a single (and different) uplink each?
 
Yes, each vm kernel port has only 1 active physical nic, and all others are unused (not standby).
I am using 4 physical NIC's in my vswitch for iscsi.

iscsi4.jpg

iscsi5.jpg
 
I think this is what you are asking for. If not, let me know how I can get the information.

iscsi6.jpg
 
Do you know if the storage actually has multiple numbered physical interfaces, or if those ip addresses are virtual? Path count is valid regardless (4 initiator ports x 4 target portals), of which the esx maximum of 8 paths are being used. However unless it's separate physical ports on the target, you're still going to run up against the maximum speed of a single ethernet link. That said, if it does have multiple physical ports, then I wonder how it would be possible to configure and verify a 1:1 mapping of them... to get that accomplished, you will probably need to pair vmknics and intended target ports into their own subnets, such as mentioned here:

http://www.delltechcenter.com/page/VMware+ESX+4.0+and+PowerVault+MD3000i
 
The 3000i I'm running has 4 physical iSCSI ports. 2 on each controller.
I am not running them in separate subnets because it isn't needed. The switches all 4 links are plugged into are stacked, so different subnets isn't necessary.

And I think my configuration is actually correct. If you look at the very last 2 images on the article you mentioned, you will see that there are paths in the screenshot that also don't have targets on them.

The article has 2 physical nics being used with 4 paths. So logically scaling it up to 4 nics I should see 16 paths, which is exactly what I am seeing.
 
Yep, it is correct, but I was contemplating the logistics of the setup in general. At a maximum you really only have four links worth of bandwidth, so depending on which paths are being used, the esx server might not be deriving the most performance out of this. Benchmarks might help in determining this. Regardless, it's already better than using network layer nic aggregation :)
 
How do I benchmark an ESX 4 iscsi system if I don't have any VM's running yet? Is there some programs that can run inside esx, or do I need to setup a windows vm and do my benchmarks from there?
 
You will need a virtual with a disk on the datastore to be benchmarked, ideally using the paravirt scsi virtual adapter. Iometer should work nicely for the actual benchmark tool.
 
What parak said.

4k and 32k reads, 50% read, 1024 outstanding IO, one worker :) limit the size of hte file it tests against so that it doesn't take forever too, and I recommend doing it on a second disk (not the boot disk) to make life easier too. :)
 
Do you know if the storage actually has multiple numbered physical interfaces, or if those ip addresses are virtual? Path count is valid regardless (4 initiator ports x 4 target portals), of which the esx maximum of 8 paths are being used. However unless it's separate physical ports on the target, you're still going to run up against the maximum speed of a single ethernet link. That said, if it does have multiple physical ports, then I wonder how it would be possible to configure and verify a 1:1 mapping of them... to get that accomplished, you will probably need to pair vmknics and intended target ports into their own subnets, such as mentioned here:

http://www.delltechcenter.com/page/VMware+ESX+4.0+and+PowerVault+MD3000i

There is no ESX path maximum - we've got customers running 16 paths to a device in some cases :)

You are correct though in that a single link at a time is used for IO.
 
There is no ESX path maximum - we've got customers running 16 paths to a device in some cases :)

You are correct though in that a single link at a time is used for IO.

Weeeell, actually, as per http://www.vmware.com/pdf/vsphere4/r40/vsp_40_config_max.pdf page 4 on the bottom... :)

To further clarify, with round robin, all active /paths/ are being used. Technically speaking, they are not used concurrently, as ESX switches paths every X I/O operations (tweakable), which sort of means that only one is used at a time, but the switching is so fast that it looks like all of them are used. But I digress; the links that I mentioned are MAC to MAC links, of which there can only be a maximum of 4, meaning a theoretical maximum of 4Gbps. The dilemma that I'm pondering is how to make sure that multipathing takes the most advantage of those 4 links, as with 16 possible paths (8 active), it's possible to leave one or more physical link unused, I think.
 
Interesting, looks like they updated the info on that. The SW initiator will actually handle more, but some arrays don't play well with it.

Default RR path switching is every 1000io - there are reasons not to switch that (mostly having to do with operations that require SCSI reservations, as the RR algorithm is not intelligent and will require a reserve / release at every path switch, no matter how many iops are required for the actual operation, unlike things like PowerPath that can have the RR portion switched off for certain operations).
 
Back
Top