• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

ESXi Host and VM Disk Latency

K

KapsZ28

2[H]4U
Joined
May 29, 2009
Messages
2,114
This is a four host cluster that was rebuilt last December with ESXi 5.5u2. The ESXi servers have not been rebooted since. On January 6th I migrated all the ESXi server to a new vCenter Appliance. I guess our monitoring solution wasn't set to send out alerts for high disk latency as I just discovered this last Friday when testing out Foglight and vRealize Operations Manager. I saw a bunch of VMs showing high disk latency. After looking closer I see that all these VMs are on the same ESXi host. Here is the Disk Performance for the one ESXi host.



The three other ESXi hosts have an average latency of 2ms while this ESXi host is clearly higher. If you look at the performance metrics of the VMs running on this host, they also show high disk latency. If I log into one of the Windows VMs, they do appear a bit slower. If I run a disk speed test I do see quite a bit of write latency and poor write speeds. If I migrate the VM to a different ESXi host, then the issue goes away.

As far as health, I don't see any issues. No hardware or networking issues reported. Definitely not exceeding capacity especially for networking. The four ESXi hosts have 6x10 Gb NICs going to two Arista switches, and 4x10 Gb to the NetApp. I don't see any network errors.

The only part I find odd is this started right after migrating the ESXi hosts to a new vCenter. The graph below shows when the transition happened.



Prior to Jan 6th and the new vCenter, absolutely no latency. (Ok, less than 1ms of latency.)

The distributed switch was exported from the old vCenter and imported into the new one. Settings on the physical ESXi server didn't change. Obviously the distributed switch settings effect all ESXi hosts, but only one has an issue. I'll probably reboot the server tonight just for the hell of it.

Any ideas what may be causing this or what else I can check?
 
I guess you mean looking at the HBA adapters?



It makes sense that there is a lot of latency on the iSCSI adapter, but I don't know what is causing it. The settings were configured with the NetApp VSC and are identical across all four ESXi hosts.
 
Reboot that sucker - see if that clears it up, but it's definitely outside the host somewhere. /var/log/vmkwarning.log show anythign interesting?
 
I do have plenty of these latency warnings, but it shows up on all the ESXi hosts.

Code: 
2015-03-23T17:08:49.978Z cpu22:32827)WARNING: ScsiDeviceIO: 1223: Device naa.60a98000383030454824462d334e3236 performance has deteriorated. I/O latency increased from average value of 2408 microseconds to 56301 microseconds.
2015-03-23T17:08:50.234Z cpu17:32822)WARNING: ScsiDeviceIO: 1223: Device naa.60a98000383030454824462d334e3236 performance has deteriorated. I/O latency increased from average value of 2408 microseconds to 121356 microseconds.
2015-03-23T17:10:16.623Z cpu15:32820)WARNING: ScsiDeviceIO: 1223: Device naa.60a98000383030454824462d334e3236 performance has deteriorated. I/O latency increased from average value of 2387 microseconds to 47814 microseconds.
2015-03-23T17:10:16.818Z cpu19:32824)WARNING: ScsiDeviceIO: 1223: Device naa.60a98000383030454824462d334e3236 performance has deteriorated. I/O latency increased from average value of 2387 microseconds to 97397 microseconds.
2015-03-23T17:10:41.000Z cpu20:32825)WARNING: ScsiDeviceIO: 1223: Device naa.60a98000383030454824462d334e3236 performance has deteriorated. I/O latency increased from average value of 2386 microseconds to 49202 microseconds.
2015-03-23T17:10:41.299Z cpu19:32824)WARNING: ScsiDeviceIO: 1223: Device naa.60a98000383030454824462d334e3236 performance has deteriorated. I/O latency increased from average value of 2394 microseconds to 111483 microseconds.
2015-03-23T17:48:47.997Z cpu21:32826)WARNING: ScsiDeviceIO: 1223: Device naa.60a98000383030454824462d334e316f performance has deteriorated. I/O latency increased from average value of 1386 microseconds to 28132 microseconds.
2015-03-23T18:05:44.431Z cpu23:32828)WARNING: ScsiDeviceIO: 1223: Device naa.60a98000383030454824462d334e316f performance has deteriorated. I/O latency increased from average value of 1386 microseconds to 28477 microseconds.
 
Cool, thanks. That was pretty much what I was leaning towards. I started playing around with vRealize Operations Manager and setup some graphs for VM read and write latency, but I am not seeing how I can add a graph for host latency.
 
You must log in or register to reply here.
Back
Top