omnios in esxi - high cpu usage

[_d3_]

Hi, i run an omnios vm on esxi 5.5 with 2vcpus and vmxnet vnic. When i try to benchmark network (with iperf to other vm on the same host) cpu usage statistics in esxi console shows 200% used (esxtop), but in vm its lower, like 25% (should be +- 50% in esxtop). I think ist unsual and there is problem or misconfiguration somewhere. I tried enabling jumbo frames with LSO disabled but no change in behavior.

 

[peanuthead]

VMtools installed? What is you switch to a e1000 NIC?

 

[danswartz]

Yeah, I've more than once had anomalous behavior from vmxnet3 driver using any kind of solaris derivative.

 

[_Gea]

Problem with ESXi 5.5 and e1000 under OmniOS ?:

I have stability problems when using e1000 on the OmniOS VM under ESXi 5.5
It works when using the VMXnet3 vnic.

my config:
SuperMicro X9 SRH-7TF, 10 GbE
ESXi 5.5 free
All-In-One, with my preconfigured ESXi 5.5 VM appliance
( http://napp-it.org/downloads )


Maybe this problem:
VMware KB: Possible data corruption after a Windows 2012 virtual machine network transfer


Anyone with similar problems?
To the OP. Can you switch to e1000 and compare results

 

[_d3_]

I found out that all IO is causing high cpu load on host, not only network IO but also any kind of disk activity and even idle cpu usage is about 20% (from perf.tab on esxi host, cpu is i5-4560).

Top interrupts from powertop were vmxnet3 and mpt, with 10k-20k wakeups per second. In idle it was <xcalls> unix`dtrace_xcall_func with <1000 wakeups/s.

VMTools installed. Also tried E1000 adapter, but it was totaly unstable (droping frames, high latency), jumbo frames on vmxnet3, upgrading to vmx-10.. still nothing

VM config - Omni OS stable r151006p, 2vcpu, 4gm ram, logic paralel controler with 1 virtual disk on local datastore, 2x phys.RDM.

 

[_Gea]

I found out that all IO is causing high cpu load on host, not only network IO but also any kind of disk activity and even idle cpu usage is about 20% (from perf.tab on esxi host, cpu is i5-4560).

Top interrupts from powertop were vmxnet3 and mpt, with 10k-20k wakeups per second. In idle it was <xcalls> unix`dtrace_xcall_func with <1000 wakeups/s.

VMTools installed. Also tried E1000 adapter, but it was totaly unstable (droping frames, high latency), jumbo frames on vmxnet3, upgrading to vmx-10.. still nothing

VM config - Omni OS stable r151006p, 2vcpu, 4gm ram, logic paralel controler with 1 virtual disk on local datastore, 2x phys.RDM.

update (at least for the e1000 problem)
ESXi 5.5 and e1000 seems ok after disabling tcp offload
by adding the following lines to /kernel/drv/e1000g.conf (reboot required)

#tcp offload disable
tx_hcksum_enable=0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0;
lso_enable=0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0;

 

[_d3_]

will try, thanks.

 

[dasaint]

_GEA,

Just curious is this an issue where you get random packet loss and high pings when putting a load on the E1000?

I saw that when i was running BETA code, the sad response is OMNI isnt a supported OS so i couldn't go the nice path i was going to reach out to the OMNI team to see what they got but i got shipped out to a customer so i just reverted the host back to 5.1 temporarily..

good to know that the VMXnet is also wonky i was gonna test that next!

 

[dasaint]

I have heard that there was possible issues with Solaris and the VMXnet configs i am trying to confirm that from the inside the catch is if it is what i think it is with the VMXNet it will be out in a patch down the road not on the instant...

 

[_d3_]

I switched to 1vcpu and problem with high host cpu usage is gone, but i have another On esxi 5.5 after shutdown quest action (or reboot quest) with vmtools installed and 1vcpu configured, kernel panics and quest is rebooted. When i add second vcpu the same vm cat shutdnow/reboot cleanly.

error:
panic[cpu0]/thread=ffffff00083c0c40: mutex_enter: bad mutex, lp=ffffffffc019d9d8 owner=ffffff018d18fae0 thread=ffffff00083c0c40

ffffff00083c09f0 unix:mutex_panic+73 ()
...
etc

 

[danswartz]

This is known illumos bug (I believe I filed it.) I finally got fed up with crap like this and gave up on illumos derivatives for now.

 

[_d3_]

Workaround for shutdown/reboot bug is editing poweroff-vm-default script (in /etc/vmware-tools) and delete all lines plus add poweroff command. Esxi logs an error after quest shutdown but its working.