OpenIndiana IPMP for iSCSI Host

W

wheelz

Weaksauce
Joined
Feb 4, 2011
Messages
100
Does anyone have any links to good how-to for setting up active/active with probe based failure detection or can tell me the correct steps? I've found several for various versions of opensolaris but none for openindiana (google thinks these are synonyms). I tried to use the ones I found and I had some success but still not quite right. There must be some slight differences (or the articles are just not 100% accurate). I did a bunch of trial and error and tried to combine what I found from multiple articles but still can't quite get it there. Thanks for any insight!
 
brianmat, thanks for the great link! I still had some trial and error but I think it is working now except for the probe based failure detection is still disabled. However I think that is because I don't have enough IPs for it to hit on that VLAN. I'll do some more testing...
 
the ipmp probes should just work, assuming you setup the static routes correctly for it to use to probe.

In my case I setup static routes to my esxi servers and the default gateway for it to use.
 
From what I read I thought it had an automatic way of finding hosts to probe. First I thought it would try using the default gateway and if there is none then it would do a multicast to find any hosts. However even when I put static routes in to 3 hosts, the status of the probe still says disabled... hmm...
 
Still having some issues with this. I got one ipmp to show the probe stat as ok, but not the other and I'm not sure why. Here is my output:

# ipmpstat -g
GROUP GROUPNAME STATE FDT INTERFACES
iscsi0 iscsi0 ok 10.00s aggr1 aggr0
host0 host0 ok 10.00s e1000g1 e1000g0
# ipmpstat -i
INTERFACE ACTIVE GROUP FLAGS LINK PROBE STATE
aggr1 yes iscsi0 ------- up unknown unknown
aggr0 yes iscsi0 --mb--- up unknown unknown
e1000g1 yes host0 ------- up ok ok
e1000g0 yes host0 --mb--- up ok ok
# ipadm show-addr
ADDROBJ TYPE STATE ADDR
lo0/v4 static ok 127.0.0.1/8
host0/? static ok 192.168.50.11/24
iscsi0/? static ok 192.168.48.135/26
aggr0/? static ok 192.168.48.136/26
aggr1/? static ok 192.168.48.137/26
e1000g0/? static ok 192.168.50.12/24
e1000g1/? static ok 192.168.50.13/24
lo0/v6 static ok ::1/128

I'm not sure why most of them have a "/?". Any ideas?
 
It looks to be saying your aggr is bad. You sure you configured them correctly?

I am using nfs over ipmp over aggr, but not for iscsi, iscsi can do multipath, and is much better to let it handle this, than using ipmp for iscsi.

# ipmpstat -g
GROUP GROUPNAME STATE FDT INTERFACES
ipmp0 ipmpg1 ok 10.00s aggr2vlan4 aggr1vlan4
# ipmpstat -i
INTERFACE ACTIVE GROUP FLAGS LINK PROBE STATE
aggr2vlan4 yes ipmp0 ------- up ok ok
aggr1vlan4 yes ipmp0 --mb--- up ok ok
# ipadm show-addr
ADDROBJ TYPE STATE ADDR
lo0/v4 static ok 127.0.0.1/8
aggr1vlan4/? static ok 10.1.0.103/20
aggr1vlan6/v4static211 static ok 10.1.17.211/24
aggr2vlan4/? static ok 10.1.0.104/20
aggr2vlan6/v4static212 static ok 10.1.17.212/24
ipmp0/? static ok 10.1.0.105/20
ipmp0/? static ok 10.1.0.106/20
lo0/v6 static ok ::1/128
 
That hasn't been my experience so far. The multipath is dependent on the host connecting and in my case that is VMware. From what I've seen VMware's builtin multipath is either only active/passive or it seems to cause issues particularly if you have any windows guests that use clustering. Also in my testing I got a significant performance boost when I enabled ipmp
The arggregate seems to be healthy based on dladm... Is there another way to check its health?
 
It sounds like you have networking issues.

vmware does multipath iscsi just fine over multible nics, but it is not simple to setup. You need to setup the vknic's correctly, setup iscsi correctly on them, then setup your iscsi luns correctly on all of that.

ipmp giving an proformance boost could only happen if your iscsi isn't setup correctly, and your performance test only involved reading, as ipmp can't improve the performance of writes last a single nic. Because of that, you really want to use iscsi multipathing, so that writes and reads both get benifitted.

Yes, vmwares normally lun config is active/passive, that has nothing to do with iscsi though, and is a simple mouseclick/cli command away to change.

As far as why your aggr's are bad, based on the ipmpstat -i, mine all show up ok ok, and your two aggr's show bad there, but your nics show ok.

I guess based on your output you have 6 nic's or more? two just normal, and 2 in aggr0 and 2 in aggr1?
 
Yes I have 2 onboard NICs for the host and a quad port PCIe NIC for iscsi.

I set up the iscsi according to VMware guide. According to this: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1011340 MRU and Fixed only uses one path at a time (failover only). I can't use Round Robin because I have clustered VMs and I could be wrong but I don't think fixed path with array preference will work with OI. Am I missing something or given this, vmware's multipathing can only be one path at a time in my scenario? I know there are 3rd party plugins that are better but I don't have any of those.
 
I think I found my reason - http://docs.oracle.com/cd/E19082-01/819-6990/gfazs/index.html in the Probe-Based Failure Detection section:

The in.mpathd daemon determines which target systems to probe dynamically. First the daemon searches the routing table for target systems that are on the same subnet as the test addresses that are associated with the IPMP group's interfaces. If such targets are found, then the daemon uses them as targets for probing. If no target systems are found on the same subnet, then in.mpathd sends multicast packets to probe neighbor hosts on the link. The multicast packet is sent to the all hosts multicast address, 224.0.0.1 in IPv4 and ff02::1 in IPv6, to determine which hosts to use as target systems. The first five hosts that respond to the echo packets are chosen as targets for probing. If in.mpathd cannot find routers or hosts that responded to the multicast probes, then ICMP echo packets, in.mpathd cannot detect probe-based failures. In this case, the ipmpstat -i utility will report the probe state as unknown
Click to expand...

I don't have 5 other IPs on my iSCSI vlan yet so I'm guessing it is the same with OI. Thought I would post if anyone else saw this.
 
patrickdk said:
Sounds like an overly complex setup.

http://kb.vmware.com/selfservice/mi...nguage=en_US&cmd=displayKC&externalId=1037959

Says roundrobin can be used, except if using RDM based luns
Click to expand...

It is actually pretty simple if you know how. The Windows clustering is one of my areas of expertise though. But yes, I'm using RDMs so I can't use RR. I've heard from other VMware guys that the built-in multipathing is so-so and there are a number of 3rd party plugins that are far better (like EMC powerpath) but unfortunately I don't have any of those.
 
You must log in or register to reply here.
Back
Top