Fun issue with Starwind iSCSI

[Child of Wonder]

Been running Starwind iSCSI since August or so and haven't had a single problem with it. Noticed that there have been a few updates since then including a fix that disables VAAI as they've seen issues with it.

"OK," I thought. "I have to reboot my file server anyway for Windows updates so let's update Starwind, too."

After SVMotioning my important VMs to local storage, I bounce the SAN. VMFS volumes gone. Hosts can see the devices but can't mount them. Tried restarting Starwind service, rolling back to previous version, removing and re-adding the image devices in Starwind, rescanning the iSCSI adapters. All no good.

Checked the /var/log/hostd.log file and saw this:

2013-03-17T16:04:31.442Z cpu0:575307)WARNING: FSAts: 1304: Denying reservation access on an ATS-only vol 'FS_iSCSI1'
2013-03-17T16:04:31.442Z cpu0:575307)WARNING: HBX: 1955: ATS-Only VMFS volume 'FS_iSCSI1' not mounted. Host does not support ATS or ATS initialization has failed.
2013-03-17T16:04:31.442Z cpu0:575307)WARNING: HBX: 1968: Failed to initialize VMFS distributed locking on volume 5067c817-d3b2bd0b-b58f-001b2145c5e0: Not supported
2013-03-17T16:04:31.442Z cpu0:575307)Vol3: 2359: Failed to get object 28 type 1 uuid 5067c817-d3b2bd0b-b58f-001b2145c5e0 FD 0 gen 0 :Not supported
2013-03-17T16:04:31.442Z cpu0:575307)WARNING: Fil3: 2492: Failed to reserve volume f530 28 1 5067c817 d3b2bd0b 1b00b58f e0c54521 0 0 0 0 0 0 0
2013-03-17T16:04:31.442Z cpu0:575307)Vol3: 2359: Failed to get object 28 type 2 uuid 5067c817-d3b2bd0b-b58f-001b2145c5e0 FD 4 gen 1 :Not supported

Ah ha! ESXi was expecting to use ATS locking on the volumes but Starwind had disabled that. When a VMFS volume is created that supports VAAI, ESXi sets the datastore to use ATS. If that functionality is removed, ESXi can no longer mount it. I changed VMFS3.HardwareAcceleratedLocking to 0, did a rescan, and there they were! But no more VAAI.

Starwind plans to have VAAI fully functional in v8 so until then I guess I'll have to live without it.

Next I'll use vmkfstools to disable ATS mode on the datastores themselves and turn VMFS3.HardwareAcceleratedLocking back to 1. Then when the new Starwind comes out or I switch to another product that supports VAAI, I can create new datastores that will use VAAI and SVMotion off my old ones.

 

[danswartz]

Nice catch. I love when vendors do stuff like this

 

[Olga-SAN]

afaik vaai support was represented as experimental

would love to see it working officially and somebody publishing performance number with it enabled and disabled

Nice catch. I love when vendors do stuff like this

 

[Child of Wonder]

afaik vaai support was represented as experimental

would love to see it working officially and somebody publishing performance number with it enabled and disabled

They enabled two of the four primitives in the initial release of v6.0 but had to roll them back because of issues they saw. I never had any problems with it at all but figured I better play it safe and install the latest patch to avoid any issues.

 

[lopoetve]

Been running Starwind iSCSI since August or so and haven't had a single problem with it. Noticed that there have been a few updates since then including a fix that disables VAAI as they've seen issues with it.

"OK," I thought. "I have to reboot my file server anyway for Windows updates so let's update Starwind, too."

After SVMotioning my important VMs to local storage, I bounce the SAN. VMFS volumes gone. Hosts can see the devices but can't mount them. Tried restarting Starwind service, rolling back to previous version, removing and re-adding the image devices in Starwind, rescanning the iSCSI adapters. All no good.

Checked the /var/log/hostd.log file and saw this:



Ah ha! ESXi was expecting to use ATS locking on the volumes but Starwind had disabled that. When a VMFS volume is created that supports VAAI, ESXi sets the datastore to use ATS. If that functionality is removed, ESXi can no longer mount it. I changed VMFS3.HardwareAcceleratedLocking to 0, did a rescan, and there they were! But no more VAAI.

Starwind plans to have VAAI fully functional in v8 so until then I guess I'll have to live without it.

Next I'll use vmkfstools to disable ATS mode on the datastores themselves and turn VMFS3.HardwareAcceleratedLocking back to 1. Then when the new Starwind comes out or I switch to another product that supports VAAI, I can create new datastores that will use VAAI and SVMotion off my old ones.

I had a nice slide on that in my VMworld presentation

Starwind is part of our partner program now for support - good guys there

 

[Grentz]

Can I just thank you, even though this does not apply to me, for posting a full post with the issue clearly defined and steps to remediation clearly defined.

So often on the internet you run across countless threads where people have issues, post they fixed it, then never post WTF they did to actually fix it.

 

[Child of Wonder]

Thanks. I'm starting a blog soon so I can document this stuff. I tinker around in my lab quite a bit and come across stuff like this.

 

[Olga-SAN]

and that

(and some performance numbers putting different solutions face to face)

would be appreciated!

Thanks. I'm starting a blog soon so I can document this stuff. I tinker around in my lab quite a bit and come across stuff like this.