ESXi dropped iSCSI connections

T

Thuleman

Supreme [H]ardness
Joined
Apr 13, 2004
Messages
5,833
There seems to be an issue between ESXi 4.1 and my EQL SAN. I see a lot of dropped connections that are then re-established. This is what the array logs say:

INFO 3/1/11 8:32:11 AM eql02 iSCSI login to target '10.10.30.102:3260, iqn.2001-05.com.equallogic:0-8a0906-ac5b90a03-d90000002ef4a246-iso' from initiator '10.10.30.50:50399, iqn.1998-01.com.vmware:vmhost1-mc-723c917d' successful, using standard frame length.
INFO 3/1/11 8:32:08 AM eql02 iSCSI session to target '10.10.30.100:3260, iqn.2001-05.com.equallogic:0-8a0906-ac5b90a03-d90000002ef4a246-iso' from initiator '10.10.30.50:54598, iqn.1998-01.com.vmware:vmhost1-mc-723c917d' was closed. Load balancing request was received on the array.

more of it at http://pastie.org/1621530

In ESXi I one vSwitch that has a VMkernel port (that's the 10.10.30.50) attached to one physical NIC and then there's another vSwitch that has the external network and two physical NICs. The two physical NICs are set to NIC Teaming -> Load Balancing -> Route based on the originating virtual port ID.

I believe this to be partially responsible for this problem I poked around a little and came across this post on the Veeam boards: http://www.veeam.com/forums/viewtopic.php?f=2&t=5780

There the guy concluded that:

follow-up. i MAY have found the solution, not 100% verified (needs much more time), but it SEEMS the latest ixgbe driver (2.0.84.9) for intel 10 gb modules on ESXi 4.1 seems to solve the drop problem.
Now what i always find very frustrating with ESXi 4.1: These driver updates have to be installed manually via vsphere cli or vma, no way you get em installed via update manager - and they are also NOT inccluded in ESXi 4.1 firmware updates (not even in the very latest two days ago). Can´t understand that ;-)

But what is nice (we checked this): If you install the new ixgbe driver and then do a ESXi 4.1 firmware upgrade with update-manager, the new driver will stay in place.

BTW: If you don´t know how to get the driver version of your ixgbe on your ESXi 4.1 machine, just enable local or remote TSM, login and enter
vsish -e get /net/pNics/vmnic0/properties
(just replace the 0 with your nic number you are looking for). Scroll above (use putty, is much nicer) and check the driver version. Driver is too old or vanilla? You can update it with vsphere cli via
vihostupdate.pl –server [IP address] –username root –install –bundle [CD/DVD]:\offline-bundle\INT-intel-lad-blablabla-offline_bundle-blablabla.blablabla

NOW ;-) If anyone has a solution how to read out the ixgbe driver version with vSphere Client or even UPDATE it, THAT would be more than great.
Click to expand...

That post also links to a thread on the VMware forums ESX4 swiscsi MPIO to Equallogic dropping although all these posts are old and are referring to issues that have already been fixed or do not apply to me since I am only using a 1:1 mapping between VMkernel and physical NIC anyway.

So my question is; Is there something else I can do to attempt to fix this or do I need to open a ticket with VMware?
 
Thuleman said:
There seems to be an issue between ESXi 4.1 and my EQL SAN. I see a lot of dropped connections that are then re-established. This is what the array logs say:

INFO 3/1/11 8:32:11 AM eql02 iSCSI login to target '10.10.30.102:3260, iqn.2001-05.com.equallogic:0-8a0906-ac5b90a03-d90000002ef4a246-iso' from initiator '10.10.30.50:50399, iqn.1998-01.com.vmware:vmhost1-mc-723c917d' successful, using standard frame length.
INFO 3/1/11 8:32:08 AM eql02 iSCSI session to target '10.10.30.100:3260, iqn.2001-05.com.equallogic:0-8a0906-ac5b90a03-d90000002ef4a246-iso' from initiator '10.10.30.50:54598, iqn.1998-01.com.vmware:vmhost1-mc-723c917d' was closed. Load balancing request was received on the array.

more of it at http://pastie.org/1621530

In ESXi I one vSwitch that has a VMkernel port (that's the 10.10.30.50) attached to one physical NIC and then there's another vSwitch that has the external network and two physical NICs. The two physical NICs are set to NIC Teaming -> Load Balancing -> Route based on the originating virtual port ID.

I believe this to be partially responsible for this problem I poked around a little and came across this post on the Veeam boards: http://www.veeam.com/forums/viewtopic.php?f=2&t=5780

There the guy concluded that:



That post also links to a thread on the VMware forums ESX4 swiscsi MPIO to Equallogic dropping although all these posts are old and are referring to issues that have already been fixed or do not apply to me since I am only using a 1:1 mapping between VMkernel and physical NIC anyway.

So my question is; Is there something else I can do to attempt to fix this or do I need to open a ticket with VMware?
Click to expand...

Seen this before 1st hand and while we never found the true cause what solved it for us was setting everything back to standard frames.

I see your connection is using standard frames but do you have any vnic or vswitch set to MTU 9000?

Tell me more and I maybe able to save you some time from our investigations.

EQL firmware version?
Physical nic model?
What network driver is ESXi using?
Switch model and settings?
Not teaming the iscsi nics correct?
 
Just a note as I looked into this a bit today and it seems a lot of the issues suspected to be part of the problem have been addressed with ESXi update 1 that came out 3 weeks ago. Do you have this installed?

I plan to test this myself in the lab and see if I can reproduce it with a fully patched server when I get time... but it could be awhile until I get time to play.
 
I am running v5.0.2 firmware on the EQL. The iSCSI NICs are broadcom, they are not teamed. The switch is a Cisco Catalyst something or other.

You are right in that this was supposed to be patched. I am running build 260247 while the patch 1 is 320137. Looks like the dropped iSCSI is the least of my problems. The Update Manager plugin is installed, 4.1 critical and non-critical baselines are attached, and it shows I am 100% compliant, which I am clearly not, this is going to be a long night.

EDIT: VUM hasn't updated since January 8th, that's when the most recent file was written to the repository.
 
Last edited:
What version of vCenter are you running? The updates released on 1/4/11 were only for esx 4.0.0. The updates released on 2/10/2011 were only for 4.1.0.
 
When I updated vCenter to 4.1 I probably just installed it on top of the existing 4.0, same with Update Manager. That somehow hosed the way the Update Manager decided whether or not a host is patched properly, even with 4.1 baselines attached.

The solution was to uninstall everything related to Update Manager (which didn't delete the local patch repository (lame!!!)), manually delete all files associated with Update Manager from the disk, then reinstall Update Manager choosing to clear the database upon install, then reinstall the plugins and do the scan, stage, remediate.

All my hosts are on ESXi 348481 now. The problem remains:

INFO 3/3/11 9:26:20 AM eql02 iSCSI login to target '10.10.30.102:3260, iqn.2001-05.com.equallogic:0-8a0906-ac5b90a03-d90000002ef4a246-iso' from initiator '10.10.30.52:50267, iqn.1998-01.com.vmware:vmhost3-mc-12ef88d6' successful using standard-sized frames. NOTE: More than one initiator is now logged in to the target.

INFO 3/3/11 9:26:17 AM eql02 iSCSI session to target '10.10.30.100:3260, iqn.2001-05.com.equallogic:0-8a0906-ac5b90a03-d90000002ef4a246-iso' from initiator '10.10.30.52:59298, iqn.1998-01.com.vmware:vmhost3-mc-12ef88d6' was closed. Load balancing request was received on the array.

This is on a Broadcom NetXtreme II BCM5708 (Dell R2950 III) onboard. I also got a quad port Intel Pro NIC in there, may move my iSCSI over to it and see whether that fixes the problem.

The whole thing is pretty weak on VMware's part though, between the seriously lacking Update Manager uninstall (hey, commercial users only pay a few thousand dollars for a license, no need to make the uninstaller work properly) and this issue with a NIC that's been deployed in the millions on 4.0 HCL hardware, I am not impressed at all.
 
Switching my iSCSI traffic over to the the Intel 82575 NICs didn't may a difference, disconnect messages are still flooding my logs. :(
 
Just as a follow-up, EQL released new FW which I haven't installed yet so I don't want to call VMware till I am up to date with the FW and I won't be updating that till the end of the week.
 
ok, so...

VC will have NOTHING to do with this.
That being said, EQL's kick initiators off on a regular basis to load balance - if that's what you're seeing, nothing odd. If it's not, then there's a problem.

Biggest first question: How many physical nics for iscsi, and how many vmkernel ports bound to them?
 
lopoetve!!! good to see you are still alive!

I currently have a support ticket open with EQL on the generally poor performance of the array, the failure to engage the VAAI array offloading, the event log being clogged up by these messages. We are about a week into it (lowest severity) and nowhere close to a resolution on any of the issues.

Running EQL FW 5.0.4, two physical nics for iscsi bound to two vmkernel ports, override vswitch failover checked, one kernel port per nic, the respective other nic in the unused adapter list.
 
what shows up for:

esxcli swiscsi nic list --adapter VMHBADEVICEFORSWISCSI

(command line, and VMHBAblah should be something like vmhba36 or so)

And yes, I'm still around :) Just been busy :)
So, VAAI... yeah, let me check on that one. Which update of ESX are you on? We've got a set of VAAI patches out that may fix the issue you're running into there.

The other question - you using 10gbe cards?
 
Running ESXi 4.1 348481

vmk1
pNic name: vmnic4
ipv4 address: 10.10.30.50
ipv4 net mask: 255.255.255.0
ipv6 addresses:
mac address: 00:1b:21:2c:71:74
mtu: 1500
toe: false
tso: true
tcp checksum: false
vlan: true
vlanId: 0
ethernet speed: 1000
packets received: 209580
packets sent: 28316
NIC driver: igb
driver version: 1.3.19.12.2-1vmw
firmware version: 1.7-7

vmk2
pNic name: vmnic5
ipv4 address: 10.10.30.53
ipv4 net mask: 255.255.255.0
ipv6 addresses:
mac address: 00:1b:21:2c:71:75
mtu: 1500
toe: false
tso: true
tcp checksum: false
vlan: true
vlanId: 0
ethernet speed: 1000
packets received: 4852
packets sent: 1984
NIC driver: igb
driver version: 1.3.19.12.2-1vmw
firmware version: 1.7-7

Just rebooted the host so no stats yet.

Not 10gbe.
 
So I finally got to the bottom of this myself. Notice how I wrote "just rebooted the host" in my previous post. It appears that what happened was that the EQL MPIO got installed and configured on the host but the host was not actually rebooted after install as was supposed to happen.

Once I rebooted the host everything was fine, no more dropped connection messages in the logs since I rebooted a couple of days ago.
 
Thuleman said:
So I finally got to the bottom of this myself. Notice how I wrote "just rebooted the host" in my previous post. It appears that what happened was that the EQL MPIO got installed and configured on the host but the host was not actually rebooted after install as was supposed to happen.

Once I rebooted the host everything was fine, no more dropped connection messages in the logs since I rebooted a couple of days ago.
Click to expand...

:p

Yeah, that'd do it.

Protip: Make sure the MPIO is actually working. Pull a path, see it fail over, and then put the path back and make sure it COMES BACK. Please. :)
 
You must log in or register to reply here.
Back
Top