Two New ESXi 5.5Servers - High CPU from Helper Process

K

KapsZ28

2[H]4U
Joined
May 29, 2009
Messages
2,114
I installed ESXi 5.5 U1 on two brand new servers and both are using about 4500 MHz of CPU. I looked in esxtop and it appears to be the Helper process. Can't say that I have seen this before. Last time I had an issue with high CPU, it ended up being the IPMI.





Also while testing the servers, one of the NetApp NFS datastores seems to randomly disconnect. Then again, these servers have been nothing but bad luck since we received them.
 
I recently helped a client that was on ESXi 5.5 and was also having Netapp datastores randomly disconnecting from random hosts. They had upgraded from 5.0 and had no issues there. We followed VMware's KB2016122 on the issue which called for upgrading the Netapp but that didn't help. We updated the Cisco UCS enic drivers, attempted the workarounds in the VMware KB article, but nothing helped. I also reviewed their UCS and VMware infrastructure and everything was configured correctly. Checking the host's logs, all it sees is everything is working fine then suddenly the NFS datastore disappears.

Finally, I just had them roll back to 5.0 and everything was fine again. Now we're waiting on a response from VMware support on what the problem actually was since nothing in the KB article they claimed was the issue helped.
 
Child of Wonder said:
I recently helped a client that was on ESXi 5.5 and was also having Netapp datastores randomly disconnecting from random hosts. They had upgraded from 5.0 and had no issues there. We followed VMware's KB2016122 on the issue which called for upgrading the Netapp but that didn't help. We updated the Cisco UCS enic drivers, attempted the workarounds in the VMware KB article, but nothing helped. I also reviewed their UCS and VMware infrastructure and everything was configured correctly. Checking the host's logs, all it sees is everything is working fine then suddenly the NFS datastore disappears.

Finally, I just had them roll back to 5.0 and everything was fine again. Now we're waiting on a response from VMware support on what the problem actually was since nothing in the KB article they claimed was the issue helped.
Click to expand...

That thoroughly sucks. One work around mentioned in the article is enabling IO control, which I already did. None of our 5.1 hosts have an issue, but part of this upgrade was to use SSD flash caching which is only available in 5.5.
 
KapsZ28 said:
That thoroughly sucks. One work around mentioned in the article is enabling IO control, which I already did. None of our 5.1 hosts have an issue, but part of this upgrade was to use SSD flash caching which is only available in 5.5.
Click to expand...

We still have no idea what the issue was. VMware support doesn't have an answer yet. One possibility is that the ISO image used to upgrade the hosts was corrupt. However, it sounds like you may be seeing the same issue.

Try re-downloading the 5.5 ISO, run a hash check against it to ensure it's good, then reload a host and see if it still drops the datastores.
 
lopoetve said:
Set high performance in the BIOS for the CPUs.
Click to expand...

I confirmed that the CPU was set to Max Performance. I tried changing it to Disabled to test, and the CPU utilization is still the same. Are there any other settings that may affect this? These are the Intel E5-2680V2.
 
Child of Wonder said:
We still have no idea what the issue was. VMware support doesn't have an answer yet. One possibility is that the ISO image used to upgrade the hosts was corrupt. However, it sounds like you may be seeing the same issue.

Try re-downloading the 5.5 ISO, run a hash check against it to ensure it's good, then reload a host and see if it still drops the datastores.
Click to expand...

What is interesting is that I downloaded the latest ISO with the Driver Rollup since this is a fresh install. However, once installed I saw that the build number is 1623387 which is not the driver rollup version. I thought maybe I accidentally downloaded the wrong version, but when I went to download it again, it was the same exact ISO. It seems the latest build should be 1636597. Not sure how much of a difference it will make.
 
KapsZ28 said:
What is interesting is that I downloaded the latest ISO with the Driver Rollup since this is a fresh install. However, once installed I saw that the build number is 1623387 which is not the driver rollup version. I thought maybe I accidentally downloaded the wrong version, but when I went to download it again, it was the same exact ISO. It seems the latest build should be 1636597. Not sure how much of a difference it will make.
Click to expand...

What hardware are you running on?
 
Child of Wonder said:
What hardware are you running on?
Click to expand...

Dual Intel Xeon E5-2680 V2 CPU
Supermicro 2u Heatsinks
Supermicro X9DR3-LN4F+ Motherboard /w Dedicated IP-KVM
16x 16GB DDR3 1600MHz ECC-Registered Memory (256GB)
2x WD2503ABYX 250GB 7200 RPM Cache SATA HDD
Intel I350-T2 NIC
Intel X520-DA2 NIC
LSI 9271-8i SAS/SATA Controller
LSI Battery Backup Unit
Supermicro SC825TQ_R740LPB Chassis w/740w Redundant PSU
 
So I am not overly familiar with NetApp versions. VMware says the issue is resolved with the following versions.

7.3.7P2
8.0.5
8.1.2P4
8.1.3RC1
8.2RC1

We are running version 8.1.2P3. Does that mean we need to upgrade to 8.1.2P4 or 8.1.3?
 
So I am talking to a buddy of mine that had the same issue and he did upgrade NetApp to resolve. The workarounds didn't appear to be a guarantee. How hard is this kind of upgrade, and do you need to failover the heads or anything? Our friggin guy that has done this before is on vacation for two weeks. My options are either upgrade NetApp, or install ESXi 5.1 which doesn't seem to have the issue.
 
KapsZ28 said:
So I am talking to a buddy of mine that had the same issue and he did upgrade NetApp to resolve. The workarounds didn't appear to be a guarantee. How hard is this kind of upgrade, and do you need to failover the heads or anything? Our friggin guy that has done this before is on vacation for two weeks. My options are either upgrade NetApp, or install ESXi 5.1 which doesn't seem to have the issue.
Click to expand...


The OnTAP upgrades are very easy. But yes, they do require a failover.
 
KapsZ28 said:
So I am talking to a buddy of mine that had the same issue and he did upgrade NetApp to resolve. The workarounds didn't appear to be a guarantee. How hard is this kind of upgrade, and do you need to failover the heads or anything? Our friggin guy that has done this before is on vacation for two weeks. My options are either upgrade NetApp, or install ESXi 5.1 which doesn't seem to have the issue.
Click to expand...

Don't know. It's supposed to be an online upgrade where NFS will fail over between the heads but greater minds than I will have to confirm. I've never worked on a Netapp array.
 
Screw that. Even if I could do it, we were specifically told not to make changes we couldn't fix if something went wrong. So I will stay away from NetApp.

I installed ESXi 5.5 without the update, and so far the CPU level is normal. I will see if it stays that way after installing some VIBs. I guess I will also try the MaxQueueDepth of 64.
 
Being that this has been a bug since 5.x, I am surprised we never ran into a problem before. Hopefully it was just update 1 or the ISO. Once I am finished with reinstalling 5.5, I will see if there are any problems. Considering there was only one VM running at the time, I don't think it was actually an IO issue.
 
Definitely do the maxqueuedepth64 with the netapp filers. We run on 2x NetApp 3240s HA pairs. ESXi 5.5 u1 on NFS and didnt run into any issues like yours, yet.
 
4saken said:
Definitely do the maxqueuedepth64 with the netapp filers. We run on 2x NetApp 3240s HA pairs. ESXi 5.5 u1 on NFS and didnt run into any issues like yours, yet.
Click to expand...

Yeah, I was going to set that in additional to the Storage IO Control just to be safe, but I am still going to stay away from U1 for now. Need to go back to the old days of not using the latest release.
 
KapsZ28 said:
So I am not overly familiar with NetApp versions. VMware says the issue is resolved with the following versions.

7.3.7P2
8.0.5
8.1.2P4
8.1.3RC1
8.2RC1

We are running version 8.1.2P3. Does that mean we need to upgrade to 8.1.2P4 or 8.1.3?
Click to expand...

Are you using NFS or iSCSI/FC with NetApp?

None of the version you list here is supported. You need 8.1.4 or 8.2P4. There are known issues for block if you run lower version of OnTap with vSphere 5.5.
 
BTW if your SAN guy(s) get back and want to upgrade to 8.1.4, wait for 8.1.4 p1 which should be out this month. There are couple significant performance bugs in 8.1.4 that you may want to avoid.
 
We are just using NFS. So far everything is working with the reinstalled v5.5, storage IO control, and maxqueuedepth set. We'll probably look into the 8.1.4P1 as you mentioned. Especially since we will be getting another NetApp soon.
 
You have to upgrade your NetApp to 8.1.2P4 at a minimum. I would also recommend adjusting any ESX host settings related to the NFS stack. NetApp should have published best practices regarding timeouts, buffer size, etc. for the vSphere clients.
 
KapsZ28 said:
What is interesting is that I downloaded the latest ISO with the Driver Rollup since this is a fresh install. However, once installed I saw that the build number is 1623387 which is not the driver rollup version. I thought maybe I accidentally downloaded the wrong version, but when I went to download it again, it was the same exact ISO. It seems the latest build should be 1636597. Not sure how much of a difference it will make.
Click to expand...

Maybe this article is relevant to your problem. Seems as if the ISO with the Driver Rollup is giving some more people a bad time and not for the first time.
 
You must log in or register to reply here.
Back
Top