vCenter Server Appliance 5.1 high CPU usage

nry

Limp Gawd
Joined
Jul 10, 2008
Messages
409
I have built myself a lab up of 5 hosts, all was working perfectly for a few days, noticed today that the fans of the server hosting the vCenter appliance were on full speed.
After some investigation it turns out that the vCenter server appliance is running at 100% CPU usage.

I have done some google searches on this and can find a few issues on version 5, but nothing for 5.1.

Anyone any idea what could be causing this?

top:
Code:
top - 13:13:52 up 28 min,  1 user,  load average: 1.02, 0.99, 0.90
Tasks: 139 total,   1 running, 138 sleeping,   0 stopped,   0 zombie
Cpu(s): 50.3%us,  0.2%sy,  0.0%ni, 49.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4057616k total,  3770080k used,   287536k free,    81296k buffers
Swap: 15735804k total,     7556k used, 15728248k free,  1164756k cached

  PID USER      PR  NI  VIRT  RES  SHR S   %CPU %MEM    TIME+  COMMAND
14097 root      20   0 1016m 388m  16m S     99  9.8   5:25.10 java
 5733 root      20   0 11420 1632 1216 S      0  0.0   0:00.55 vmware-watchdog
 5791 ssod      20   0 1629m 496m  11m S      0 12.5   0:35.27 java
13533 root      20   0 1605m 340m  12m S      0  8.6   0:13.52 java
13893 root      20   0  435m 176m  86m S      0  4.4   0:04.98 vpxd
    1 root      20   0 10528  720  676 S      0  0.0   0:00.59 init

Process ID in question
Code:
vcenter:~ # ps -eaf | grep 14097
root     14097     1 99 13:08 ?        00:06:01 /usr/java/jre-vmware/bin/java -Djava.util.logging.config.file=/usr/lib/vmware-vpx/tomcat/conf/logging.properties -Xss1024K -Xincgc -XX:+ForceTimeHighResolution -XX:PermSize=64M -Xmx512m -XX:MaxPermSize=128m -Dvim.logdir=/var/log/vmware/vpx -Djava.util.logging.manager=com.springsource.tcserver.serviceability.logging.TcServerLogManager -Djava.endorsed.dirs=/usr/local/tcserver/vfabric-tc-server-standard/tomcat-7.0.25.B.RELEASE/endorsed -classpath /usr/local/tcserver/vfabric-tc-server-standard/tomcat-7.0.25.B.RELEASE/bin/bootstrap.jar:/usr/local/tcserver/vfabric-tc-server-standard/tomcat-7.0.25.B.RELEASE/bin/tomcat-juli.jar -Dcatalina.base=/usr/lib/vmware-vpx/tomcat -Dcatalina.home=/usr/local/tcserver/vfabric-tc-server-standard/tomcat-7.0.25.B.RELEASE -Djava.io.tmpdir=/usr/lib/vmware-vpx/tomcat/temp org.apache.catalina.startup.Bootstrap start
root     16030 15081  0 13:14 pts/0    00:00:00 grep 14097
 
Does rebooting it help? I've seen my Windows lab vCenter do that (Java pegging CPU) after an unexpected restart (like when I knocked the power cable loose to one host the other day...). It would boot up and kill the CPU. Let if finish starting, reboot it, and it was fine.
 
Need more info, what version of vCenter Appliance are you using?

5.1, 5.1U1, 5.1U1a?

Are the vmware tools updated as well?

Do you have Resource Pools Enabled (DRS?) is there any Limits on Memory or CPU that could create a contention like this?
 
/me knows what it is. Restart inventory service and profile driven storage. Betcha it goes away.
 
VMware vCenter Server Appliance 5.1.0.5300 Build 947940

Which I believe is version 5.1.0b

vmware tools are just the default installed ones with the appliance I have downloaded

DRS has settings set to:
Migration Automation Level: Manual
Power Management Automation Level: Off

I do have some hosts in the cluster currently offline, that shouldn't cause it should it? Never had this problem before when they were off.

The Appliance has 2vCPUs and 4GB of RAM, same issue if it has 8 or 12GB of RAM.

Rebooting the appliance results in the same problem after about 10 minutes, takes about 7 minutes for it to boot up!!

I am learning all of this vmware stuff so please bare with any silly mistakes I am making :)
 
Removed all the hosts which are offline so now just a single host in the cluster.

No change in CPU usage.
 
ok, that's rather odd. let me see what all is going on known with those.
 
Nothing too obvious, Im not too sure what to look for to be honest as I am not that familiar with half of these services.

Inside vws.log I did find one error...

Code:
[2013-06-26 19:39:40,949 pool-3-thread-1  INFO  com.vmware.vim.vws.VwsContextListener] Starting VWS
[2013-06-26 19:39:40,993 VwsInit  INFO  com.vmware.vim.security.authenticate.CertificateUtil] Loading local SSL certificate
[2013-06-26 19:39:41,426 VwsInit  INFO  com.vmware.vim.security.authenticate.CertificateUtil] Replaced certificate for service endpoint: cn=VIMAPI,CN=DD035725-A49E-4BDC-B0EF-0B0C29F5CC16,OU=Instances,dc=virtualcenter,dc=vmware,dc=int
[2013-06-26 19:39:42,777 VwsInit  INFO  com.vmware.vim.cimmonitor.inventory.InventoryPoller] InventoryPoller initialization complete
[2013-06-26 19:39:42,777 VwsInit  INFO  com.vmware.vim.vws.VwsContextListener] Started VWS.
[2013-06-26 19:39:42,858 Timer-0  INFO  com.vmware.vim.common.vdb.VdbConfig] Config name=vcdb
[2013-06-26 19:39:42,858 Timer-0  INFO  com.vmware.vim.common.vdb.VdbConfig] Property file=/etc/vmware-vpx/vcdb.properties
[2013-06-26 19:39:42,858 Timer-0  INFO  com.vmware.vim.common.vdb.VdbConfig] Loaded url from props=jdbc:postgresql://127.0.0.1:5432/VCDB
[2013-06-26 19:39:42,858 Timer-0  INFO  com.vmware.vim.common.vdb.VdbConfig] Overrides=
[2013-06-26 19:39:43,741 pool-14-thread-1  ERROR com.vmware.vim.cimmonitor.vc.stub.VcStub] Failed to acquire CIM ticket for host host-52
com.vmware.vim.binding.vmodl.fault.ManagedObjectNotFound:
obj = com.vmware.vim.binding.vmodl.ManagedObjectReference@63e44e7d
inherited from com.vmware.vim.binding.vmodl.fault.ManagedObjectNotFound: The object has already been deleted or has not been completely created
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
	at java.lang.Class.newInstance0(Class.java:355)
	at java.lang.Class.newInstance(Class.java:308)
	at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:171)
	at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:26)
	at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:33)
	at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:135)
	at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:98)
	at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:84)
	at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:37)
	at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:97)
	at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:245)
	at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:203)
	at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:126)
	at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:98)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:526)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:507)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:295)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:265)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:169)
	at $Proxy48.acquireCimServicesTicket(Unknown Source)
	at com.vmware.vim.cimmonitor.vc.stub.VcStub.acquireCimServicesTicket(VcStub.java:60)
	at com.vmware.vim.cimmonitor.service.impl.CimMonitorImpl.acquireCimServicesTicket(CimMonitorImpl.java:432)
	at com.vmware.vim.cimmonitor.service.impl.CimMonitorImpl.createCimClient(CimMonitorImpl.java:633)
	at com.vmware.vim.cimmonitor.service.impl.CimMonitorImpl.createCimClient(CimMonitorImpl.java:810)
	at com.vmware.vim.cimmonitor.service.impl.HostRefresherTask.refreshHostInt(HostRefresherTask.java:205)
	at com.vmware.vim.cimmonitor.service.impl.HostRefresherTask.run(HostRefresherTask.java:118)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
[2013-06-26 19:39:43,743 pool-14-thread-1  ERROR com.vmware.vim.cimmonitor.service.impl.CimMonitorImpl] Error grabbing CIM ticket on 10.0.20.150/host-52:Failed to acquire CIM ticket for host host-52
[2013-06-26 19:39:43,744 pool-14-thread-1  WARN  com.vmware.vim.cimmonitor.service.impl.CimMonitorImpl] 10.0.20.150/host-52: Increasing refresh interval to 2700000 millis.
[2013-06-26 19:39:43,754 pool-14-thread-1  INFO  com.vmware.vim.cimmonitor.service.impl.CimMonitorImpl] randomly scheduled host-52 for health poll in 2633091 millis
[2013-06-26 19:39:58,725 pool-59-thread-1  INFO  com.vmware.vim.common.vdb.VdbConfig] Config name=vcdb
[2013-06-26 19:39:58,725 pool-59-thread-1  INFO  com.vmware.vim.common.vdb.VdbConfig] Property file=/etc/vmware-vpx/vcdb.properties
[2013-06-26 19:39:58,725 pool-59-thread-1  INFO  com.vmware.vim.common.vdb.VdbConfig] Loaded url from props=jdbc:postgresql://127.0.0.1:5432/VCDB
[2013-06-26 19:39:58,725 pool-59-thread-1  INFO  com.vmware.vim.common.vdb.VdbConfig] Overrides=
[2013-06-26 19:44:46,683 Timer-0  INFO  com.vmware.vim.common.vdb.VdbConfig] Config name=vcdb
[2013-06-26 19:44:46,683 Timer-0  INFO  com.vmware.vim.common.vdb.VdbConfig] Property file=/etc/vmware-vpx/vcdb.properties
[2013-06-26 19:44:46,683 Timer-0  INFO  com.vmware.vim.common.vdb.VdbConfig] Loaded url from props=jdbc:postgresql://127.0.0.1:5432/VCDB
[2013-06-26 19:44:46,683 Timer-0  INFO  com.vmware.vim.common.vdb.VdbConfig] Overrides=
 
looks like your vcenter can't grab the health data from one of your hosts (10.0.20.150), Not sure if that has anything to do with your problem, but it couldn't hurt to fix it.

what's the hostname of your esxi server? I read something about the name of the esxi server affecting the tomcat service because it could be not rfc 952 compliant.

http://kb.vmware.com/selfservice/mi...nguage=en_US&cmd=displayKC&externalId=1013507

try restarting the management and vpxa services on the host. If that doesn't work, remove the last(?) host from your cluster (while it's still turned on and connected) then reconnect it. I would recommend using the root user and password to reconnect to vcenter.
 
Back
Top