• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

VM performance issue to EqualLogic

K

KapsZ28

2[H]4U
Joined
May 29, 2009
Messages
2,114
I guess this could either be in this forum or the storage forum. I don't believe there is an issue with the EqualLogic. It appears to be the ESXi host, but it is a bit odd. Here is the setup.

Two Dell PowerEdge R620 servers using a Broadcom BCM57810 10Gb NIC (2 ports for storage)
Two Dell N4032 10 Gb switches that are stacked
One EqualLogic PS6210X SAN with 10 Gb

The customer has a Server 2008 R2 VM with an application installed that does some conversations. The application and the data are run on drive D which is simply just a VMDK that resides on the EqualLogic. When the app is run on host A, it runs fine and takes about two minutes. If you move the VM to host B and try to run the app again, it freezes a bit and usually takes 6-10 minutes to run. They migrated all VMs except for the test one off of host B to ensure there was no contention.

I ran a simple disk speed test and both hosts and below is the results.

Host A


Host B


I ran the tests several times and the results were always about the same. Clearly host B is not performing as well especially with the sequential and random reads. Multipathing is setup correctly, but I decided to try each 10 Gb NIC individually to see what happens. Same results. All the settings are the same, MTU, port speed, etc. I really couldn't find anything different between the two hosts.

I logged into the hosts directly to run "vmkping 10.51.0.13 -c 10000 -s 8000 -d". That is the IP address of the EqualLogic for the iSCSI connection. Below are the results.

Host A
 8008 bytes from 10.51.0.13: icmp_seq=10 ttl=255 time=0.189 ms
8008 bytes from 10.51.0.13: icmp_seq=11 ttl=255 time=0.216 ms
8008 bytes from 10.51.0.13: icmp_seq=12 ttl=255 time=0.169 ms
8008 bytes from 10.51.0.13: icmp_seq=13 ttl=255 time=0.196 ms
8008 bytes from 10.51.0.13: icmp_seq=14 ttl=255 time=0.231 ms
8008 bytes from 10.51.0.13: icmp_seq=15 ttl=255 time=0.188 ms
8008 bytes from 10.51.0.13: icmp_seq=16 ttl=255 time=0.228 ms
8008 bytes from 10.51.0.13: icmp_seq=17 ttl=255 time=0.145 ms
8008 bytes from 10.51.0.13: icmp_seq=18 ttl=255 time=0.156 ms
8008 bytes from 10.51.0.13: icmp_seq=19 ttl=255 time=0.182 ms
8008 bytes from 10.51.0.13: icmp_seq=20 ttl=255 time=0.188 ms
8008 bytes from 10.51.0.13: icmp_seq=21 ttl=255 time=0.149 ms
8008 bytes from 10.51.0.13: icmp_seq=22 ttl=255 time=0.192 ms
8008 bytes from 10.51.0.13: icmp_seq=23 ttl=255 time=0.155 ms
8008 bytes from 10.51.0.13: icmp_seq=24 ttl=255 time=0.217 ms

Host B
 8008 bytes from 10.51.0.13: icmp_seq=10 ttl=255 time=0.280 ms
8008 bytes from 10.51.0.13: icmp_seq=11 ttl=255 time=0.369 ms
8008 bytes from 10.51.0.13: icmp_seq=12 ttl=255 time=0.245 ms
8008 bytes from 10.51.0.13: icmp_seq=13 ttl=255 time=0.322 ms
8008 bytes from 10.51.0.13: icmp_seq=14 ttl=255 time=0.240 ms
8008 bytes from 10.51.0.13: icmp_seq=15 ttl=255 time=0.254 ms
8008 bytes from 10.51.0.13: icmp_seq=16 ttl=255 time=0.283 ms
8008 bytes from 10.51.0.13: icmp_seq=17 ttl=255 time=0.271 ms
8008 bytes from 10.51.0.13: icmp_seq=18 ttl=255 time=0.412 ms
8008 bytes from 10.51.0.13: icmp_seq=19 ttl=255 time=0.241 ms
8008 bytes from 10.51.0.13: icmp_seq=20 ttl=255 time=0.290 ms
8008 bytes from 10.51.0.13: icmp_seq=21 ttl=255 time=0.274 ms
8008 bytes from 10.51.0.13: icmp_seq=22 ttl=255 time=0.204 ms
8008 bytes from 10.51.0.13: icmp_seq=23 ttl=255 time=0.364 ms
8008 bytes from 10.51.0.13: icmp_seq=24 ttl=255 time=0.248 ms

Both are good response times, but host B is taking 67% longer. Not sure if that matters...

Any ideas on what may be causing the performance issue on host B, or what else I can check?
 
EqualLogic has some specific settings that they recommend for iSCSI and vSphere. Please see my blog post that will point you in the right direction for the information.

http://virtualfng.wordpress.com/2014/07/26/equallogic-iscsi-setup-script-for-vmware-vsphere/

Though it talks about how to use the Pearl Script to easily setup iSCSI properly, look at the beginning of the article for the Dell Rapid EqualLogic Configuration Portal to find all the info you need.

This also includes switch configuration(s) as well.
 
Suggestion - do not stack your iSCSI switches as now you cannot perform maintenance on them without taking the whole stack down. I always configure them independently of one another and use a port channel between them.

Things to check:
1. Delayed ACK disabled by default on swiSCSI initiator and inherited to all static targets discovered dynamically.
2. Use the Dell MEM PSP if you have E or E+ VMware licensing
3. Make sure that if you do not have the ability to use the MEM that you set the IOPS per path to 3. This was more relevant to 1Gb multi link setups, but could be a factor.
4. Make sure that jumbo, flow control, etc. are configured correctly on both your PC switches and both hosts.
5. Like above, check all firmware using Dell Lifecycle controller, or, my preferred method is to make a bootable update ISO with Dell Repository Manager (faster).
6. Obviously any updates to VMware as well.

To me it just sounds like a setting that was overlooked on the host that is not performing as well, but I have been surprised in the past. Definitely share what the issue was once found so we all can learn. Good luck!
 
How about using the Dell ESXi ISO for the installation? I noticed that they used a standard VMware ISO instead of downloading the latest from Dell.
 
Vader said:
That's not silly at all...ya never know...;)
Click to expand...

So I checked on both hosts, and they are the same.

~ # ethtool -i vmnic4
driver: bnx2x
version: 1.72.56.v55.2
firmware-version: FFV7.8.16 bc 7.8.55 phy 1.34
bus-info: 0000:42:00.0

~ # ethtool -i vmnic5
driver: bnx2x
version: 1.72.56.v55.2
firmware-version: FFV7.8.16 bc 7.8.55 phy 1.34
bus-info: 0000:42:00.1

~ # vmkload_mod -s bnx2x | grep Version
Version: Version 1.72.56.v55.2, Build: 1331820, Interface: 9.2 Built on: Sep 18 2013

They are Broadcom adapters, and oddly I didn't see any firmware listed on Dell's website. I see driver versions 17 and 18, but I don't see where to get the exact driver version on the ESXi host. Although I am fairly confident that both hosts have the identical versions on them.
 
Vader said:
EqualLogic has some specific settings that they recommend for iSCSI and vSphere. Please see my blog post that will point you in the right direction for the information.

http://virtualfng.wordpress.com/2014/07/26/equallogic-iscsi-setup-script-for-vmware-vsphere/

Though it talks about how to use the Pearl Script to easily setup iSCSI properly, look at the beginning of the article for the Dell Rapid EqualLogic Configuration Portal to find all the info you need.

This also includes switch configuration(s) as well.
Click to expand...

Thanks. Good info in the article. Love the screen shots! :D All the settings are correct on both hosts. Same settings that are in you post. Obviously no Dell MEM since they are only using ESXi Standard.
 
MEM? Someone please remind me what this stands for?

Is there any cpu affinities set, minimum mhz or storage i/o limits set?
 
Last edited:
MEM=Multipathing Extension Module

Kaps, did you follow SGalbincea's suggestions as well. Since you can't use the MEM, you'll have to manually configure all your settings..

Also..did you look at the switch configs as well per the Dell EqualLogic Rapid Configuration Portal?
 
Vader said:
MEM=Multipathing Extension Module

Kaps, did you follow SGalbincea's suggestions as well. Since you can't use the MEM, you'll have to manually configure all your settings..

Also..did you look at the switch configs as well per the Dell EqualLogic Rapid Configuration Portal?
Click to expand...

The settings appear to be correct based on your blog except for the IOPS. It is not my server to make changes on, so I made the recommendation to change IOPS=3.

I reviewed the network settings last week and they looked good for both servers and switches, but I will double check later.

I made the recommendation to the client about using the Dell Lifecycle or Dell Repository Manager, but again, it is on them.

I am curious, does vmkping mean much of anything?

"vmkping 10.51.0.13 -c 10000 -s 8000 -d"

Good server.
8008 bytes from 10.51.0.13: icmp_seq=18 ttl=255 time=0.156 ms

Bad server.
8008 bytes from 10.51.0.13: icmp_seq=13 ttl=255 time=0.322 ms

I know they are both sub-millisecond, but the server with the issues is 67% slower to respond.
 
Hmmm..interesting. Specify the vmk, -I vmkx where x is the VMkernel and see if it's both interfaces on the Bad server?
 
Vader said:
Hmmm..interesting. Specify the vmk, -I vmkx where x is the VMkernel and see if it's both interfaces on the Bad server?
Click to expand...

Pretty much the same except for 1 of the pings. This is a quad port NIC.

~ # vmkping 10.51.0.13 -c 10000 -s 8000 -d -I vmk1
PING 10.51.0.13 (10.51.0.13): 8000 data bytes
8008 bytes from 10.51.0.13: icmp_seq=0 ttl=255 time=0.391 ms
8008 bytes from 10.51.0.13: icmp_seq=1 ttl=255 time=0.268 ms
8008 bytes from 10.51.0.13: icmp_seq=2 ttl=255 time=0.261 ms
8008 bytes from 10.51.0.13: icmp_seq=3 ttl=255 time=0.231 ms
8008 bytes from 10.51.0.13: icmp_seq=4 ttl=255 time=0.258 ms
8008 bytes from 10.51.0.13: icmp_seq=5 ttl=255 time=0.235 ms
8008 bytes from 10.51.0.13: icmp_seq=6 ttl=255 time=0.232 ms
8008 bytes from 10.51.0.13: icmp_seq=7 ttl=255 time=0.229 ms
8008 bytes from 10.51.0.13: icmp_seq=8 ttl=255 time=0.376 ms
8008 bytes from 10.51.0.13: icmp_seq=9 ttl=255 time=0.275 ms
8008 bytes from 10.51.0.13: icmp_seq=10 ttl=255 time=0.304 ms
8008 bytes from 10.51.0.13: icmp_seq=11 ttl=255 time=1.263 ms
8008 bytes from 10.51.0.13: icmp_seq=12 ttl=255 time=0.226 ms
8008 bytes from 10.51.0.13: icmp_seq=13 ttl=255 time=0.346 ms

--- 10.51.0.13 ping statistics ---
14 packets transmitted, 14 packets received, 0% packet loss
round-trip min/avg/max = 0.226/0.350/1.263 ms
~ # vmkping 10.51.0.13 -c 10000 -s 8000 -d -I vmk2
PING 10.51.0.13 (10.51.0.13): 8000 data bytes
8008 bytes from 10.51.0.13: icmp_seq=0 ttl=255 time=0.460 ms
8008 bytes from 10.51.0.13: icmp_seq=1 ttl=255 time=0.378 ms
8008 bytes from 10.51.0.13: icmp_seq=2 ttl=255 time=0.374 ms
8008 bytes from 10.51.0.13: icmp_seq=3 ttl=255 time=0.267 ms
8008 bytes from 10.51.0.13: icmp_seq=4 ttl=255 time=0.243 ms
8008 bytes from 10.51.0.13: icmp_seq=5 ttl=255 time=0.243 ms
8008 bytes from 10.51.0.13: icmp_seq=6 ttl=255 time=0.290 ms
8008 bytes from 10.51.0.13: icmp_seq=7 ttl=255 time=0.207 ms
8008 bytes from 10.51.0.13: icmp_seq=8 ttl=255 time=0.397 ms
8008 bytes from 10.51.0.13: icmp_seq=9 ttl=255 time=0.298 ms
8008 bytes from 10.51.0.13: icmp_seq=10 ttl=255 time=0.249 ms
8008 bytes from 10.51.0.13: icmp_seq=11 ttl=255 time=0.308 ms
8008 bytes from 10.51.0.13: icmp_seq=12 ttl=255 time=0.390 ms
8008 bytes from 10.51.0.13: icmp_seq=13 ttl=255 time=0.230 ms

--- 10.51.0.13 ping statistics ---
14 packets transmitted, 14 packets received, 0% packet loss
round-trip min/avg/max = 0.207/0.310/0.460 ms
 
Have you tried a different patch cable and/or a different port on either the SAN or the switch yet? If only to isolate the issue further?
 
Nate7311 said:
Have you tried a different patch cable and/or a different port on either the SAN or the switch yet? If only to isolate the issue further?
Click to expand...

I made that recommendation to, but not sure they too my advise. Below is what they said they are planning on doing.

1. We are going to order new NIC cards this week and install them this weekend.
2. We compiled host logs and had them sent to VMware. Their engineers are going to review and get back to us. I would think if it's OS related they will be able to tell us.
3. I think we will also apply VMWare updates this weekend.
 
So, from what I remember about equallogic, it has a active network port and then a standby port that "could" take traffic but really just redirected it to the active interface internal to the equallogic. But, that may have changed when they moved away from eight 1gb copper ports...

This latency issue seems like you are taking an extra hop some place. Is it possible to put host A, Host B, and the equallogic active ports on the same switch? The latency feels like host B is going through the stacking link to get to the equallogic.
OR, if the equallogic still does the standby forwarding to the active port thing.... And host B and the equallogic standby ports share the same switch, maybe it's there for the extra hop.

Let me know if its possible to do some testing with all of the same links on the same switch and the results.

If they are all on the same switch already, see if you can setup a monitor port and wireshark sniff the traffic to see which port the hosts are using on the equallogic. You could have a dirty fiber, bad equallogic port or exploding SFP!

Nicholas Farmer
Http://pcli.me
 
So this issue has been resolved by the client. I was told it was a memory setting issue in the BIOS. I don't know the full details. Hopefully we will have a call at some point so I can get more information as I am very curious what memory setting in the BIOS would have caused this problem.
 
Maybe. Hopefully I have a call with them soon and get the correct information.
 
You must log in or register to reply here.
Back
Top