High host CPU usage with ESXi 5.1 and pfsense 2.02

S

shadowwyvern

Gawd
Joined
Sep 22, 2005
Messages
604
Hello,

Recently set up an ESX server with the goal of consolidating some systems - notably a fileserver and router.

Specs:
CPU: i5-3570
Motherboard: Asrock Pro4-M
LAN NIC: Intel 82572EI GigE
WAN NIC: Realtek 8168 (on the host, pfsense connected through vSwitch)

pfsense VM has 2 e1000-based virtual NICs and VM tools installed (from the packages page). I also have bandwidthd and the VPN bridging fix packages installed.

I have a 25Mbit/25Mbit (advertised) FIOS connection.

I think I am having some performance problems with the setup. When downloading at 3MByte/s (connection max) I see host CPU usage (via vSphere client) at 35-40% for the pfsense guest. "top" on the guest only reports a fraction of that however.

I have a few ubuntu 12.10 x64 guests on the host, and when idle they report 5-10MHz of usage, while pfsense when totally idle (I made a new VM that has no network load, other than whatever traffic it gets through its port group) is at 40-50MHz (w/o VM tools installed). Meanwhile, my "production" pfsense VM runs at 100-200MHz when routing low-level traffic (a few 100kbit of voip, etc) and 500-1000MHz when I load it with some 1.5-2 MByte traffic.

In vCenter when transferring 2MByte/s I see 20% CPU usage, while top still reports 93-98% idle.

Also, subjectively, the internet seems less responsive - but I'm not taking that as much evidence at the moment, as I can't quantify it.

My previous setup was an 800MHz VIA cpu, and it worked perfectly fine. Why am I now able to use up to 35% of a 3.4GHz ivy bridge? Why is there such a discrepancy between vSphere's usage and "top"?

I have searched some on this topic, and it seems this is a hit-or-miss issue - some people are able to run their VMs without a problem with close to gigabit throughput, others have the same issue as me.

This thread: http://forum.pfsense.org/index.php/topic,41647.0.html has a good discussion (I haven't tried any of the potential fixes because there didn't seem to be any conclusive results)

EDIT: My steam test download finally picked up - currently reading 40% host cpu, 90% idle in top, 3MB/s down.
 
Last edited:
Yep. NetJunkie in that thread is me. You can flip..I think it was Device Polling...and it'll fix it for a while. That's one reason I now run Untangle. Never could find a good fix for it.
 
That's too bad. My only real theory at this point is that its something to do with my Realtek NIC, it seemed in that thread that people with Intel cards had less problems.

I am going to try to run a test that tests a completely virtualized workload: VM->pfsense->VM, that should show some hardware independent behavior. Can't seem to see a clean way of cloning VM's - maybe the free ESX license doesn't support it - but that's a different matter :)
 
two things

did you install the open vm tools?
did you try the latest beta of 2.1

I am using the latest beta and it works very well.
 
Yes, open vm tools were installed, via the package manager. There were 2 different ackages, but I could not tell the difference between them, so I just picked one.

Rody and shiznit, what are you running for host hardware?
 
intel nics, Realtek with linux based OS have been noted many times to use high cpu usage due to crap drivers.
 
Looks like I'm off to ebay to find some new NICs.

Those who say it works well, could I ask:
1) How much traffic are you moving? How much host CPU is used while moving this?
2) "Idle" host CPU usage - routing 150KByte of traffic, I am running at 167MHz
3) I am still suspicious that my virtualized pfsense is not quite as responsive as the bare metal one - it seems to take longer for pages to load, etc. Are you noticing any particular problems?

MrGuvernment, I presume you were referring to the host - ESXi, as it is linux based? I am using the "e1000" drivers on the guest pfsense.
 
Well this is odd...I just checked mine and I'm having that problem too. If I go to cachefly.com and download the 100mb.zip, my CPU usage in esxi is 75% of one core, where as in pfsense its like...6 ~ 9%

AMD 8120 8 core
32gb
ESXi 5.1
Intel Dual Server Gigabit nic passed through to pfSense 2.0.2
 
I purchased a EXPI9402PT from ebay. Hopefully that will help - and if not I can experiment with some aggregation to my desktop.

I tried giving pfsense another virtual core. It's difficult to say if there has been any benefit.

With the cachefly file, I see 50% usage for the pfsense VM according to esxtop.
 
fastforwarding does not appear to have an impact on host cpu usage for me. 3MByte/s download still consumes 50% of the host cpu according to esxtop.

Thanks for the suggestion :)
 
25 Mbit/s for me is 5% in pfsense and 10% in esxtop, and I think you have to reboot after you enable fast forwarding.
 
Last edited:
I run a pfsense box on KVM and typically see about 15-25% cpu usage on a phenom ii 965....
 
Even with that tweak I'm still in the 80% range for 25 Mbit/s down.
 
Since pfSense is wanting to murder an entire core with even little usage, what would you guys recommend I replace it with?
 
To avoid the troubles with a virtualized router I went with a dedicated 1U Intel Atom D525 Supermicro setup, 2 x Intel NICs, sips power. I'm in South Texas, so during the summer I can kill my VM lab and still have a home network.
 
Ironically enough, one of my motivations for virtualizing the router was to save power. The bare metal box uses 26W - the EXI box the virtualized is on uses 47W idle while running, which is a little more than the fileserver it essentially eclipsed.

Further ironically, running a single core at 50% during heavy load completely cancels out these savings.

My FIOS service is having some issues at the moment - only 3Mbit upload on testmy.net, but both speedtest and the verizon speedtest report 25Mbit, as does the real-time bandwidth plot in pfsense.

One interesting outcome of my flurry of speedtests is that pfsense is using less cpu to upload than to download. With the speedtest reporting the same results for both upload and download, esxtop report 49% during the download phase, and only 30% during the upload phase.

shiznit, I have indeed rebooted the vm (but not the host). doing a "sysctl net.inet.ip.fastforwarding" from the shell reports "1".

With regards to the VM Tools, is everyone using the ones from the packages? Or is a manual install of the actual VMware tools recommended? Also, there are two packages in the 2.0.2 package list:
>>"Open-VM-Tools-8.8.1" package version "528969" I am running this one
>>"Open-VM-Tools" package version "Stable 8.7.0.3046 (build-313025) platform: 2.0"

Is any particular one better, or for that matter is one of them outright wrong?
 
I use 8.8.1. If you want to try, there is a guide on the pfsense forums for installing the official vmtools which will let you use the vmxnet3 adapter and may help with your problem.

Ziggo0, if you want to go physical consider Alix, I hear it can handle 50+ Mbit and has hardware acceleration for 128 bit encryption. That said, an atom or bobcat system is roughly the same price and much more capable.

An ARM port would be ideal.
 
^Thanks for the input.


I'm passing through my dual intel gigabit nic to my pfSense VM...sadly with or without vmware tools makes no difference. Hitting cachefly @ 35mbps yields around 70% of one CPU core in esxtop still. Nothing I've found seems to help :(

Edit: I introduced a 250mhz speed limit to my single core pfSense VM and my speeds are still the same internet wise, toping out around 8 ~ 10% usage of 1 core

Edit: Well for raw speed downloading a single file 250mhz works well, but playing games it doesn't seem to care for. Up to 1ghz and things seem to be ok.
 
Last edited:
Ran a rather unscientific test regarding vmxnet2 vs e1000 ethernet adapters on the 2.1 beta. Consistent with other such tests from around the internet, I didn't see a substantial difference.

I had my fileserver serving a file from its disks, a pfsense VM routing them, and then an Ubuntu VM (running vmxnet3) behind the pfsense VM. A better test would have been iperf, but I didn't feel like doing that. I also didn't bother because I am looking for a factor of 5 improvement or so, so its pretty easy to see if the problem is fixed or not.

My new dual port ethernet adapter arrives on Wed. ziggo0 seems to have already tested the setup I was planning on - passthru to the VM.
 
Installed the new intel card, and am seeing no difference in CPU usage. 1627MHz to download 25Mbit of traffic on a VM with 2 vCPUs.
 
I installed m0n0wall which I believe pfSense is based from, ZERO CPU issues there. Passing through the same card, same memory/CPU cores (1) - hitting cachefly gives me a whopping 5% CPU usage (as it should).

Moved down to pfSense 1.2.3....no issues with host CPU usage here. Seems to be that whatever they changed they don't want to fix or aren't acknowledging the countless people having the issue.
 
Last edited:
I primarily use Mikrotik for the traffic shaping but pfSense is my vpn gateway and I'm am not seeing this issue. If I have time one day I'll make it my edge router and see if I can replicate it.
 
You must log in or register to reply here.
Back
Top