ESXi 5.0 Virtual NIC's E1000 vs. VMXNET3

Zarathustra[H]

Extremely [H]
2FA
Joined
Oct 29, 2000
Messages
42,476
Hey all,

More n00bish questions here.

Thus far I have understood to choose VMXNET3 for best performance, and if this doesn't work, use E1000 for compatibility.

What do I lose by using E1000? Is there more host CPU overhead, or what is the issue?

Much obliged,
Matt
 
You can get faster speeds with VMXNET if your guest OS supports it properly. It's not emulating a physical NIC and isn't subject to the same limitations. You will need VMWare Tools installed in the guest to use VMXNET, which can sometimes cause issues when updating the guest OS. If VMWare Tools gets broken, the VMXNET controller can break as well. Our production VMs usually use e1000, but we use VMXNET for some lab and Windows guests.
 
Unfortunately, the vmxnet3 driver for opensolaris seems to be not very reliable. I've been using e1000 across the board for that reason.
 
You can get faster speeds with VMXNET if your guest OS supports it properly. It's not emulating a physical NIC and isn't subject to the same limitations. You will need VMWare Tools installed in the guest to use VMXNET, which can sometimes cause issues when updating the guest OS. If VMWare Tools gets broken, the VMXNET controller can break as well. Our production VMs usually use e1000, but we use VMXNET for some lab and Windows guests.

Interesting.

I've had issues getting VMWare Tools installed on my Ubuntu Oneiric (11.10) Server VM, yet VMXNET3 seems to work just fine in it... :confused:
 
Unfortunately, the vmxnet3 driver for opensolaris seems to be not very reliable. I've been using e1000 across the board for that reason.

Yeah, we've had problems with it on Solaris/Illumos and FreeBSD (though I think FreeBSD has since been resolved). It seems to work fine out of the box on any 3.x Linux kernel (and some late 2.6-series) and on Windows with VMWare Tools.

Zarathustra[H];1038379900 said:
Interesting.

I've had issues getting VMWare Tools installed on my Ubuntu Oneiric (11.10) Server VM, yet VMXNET3 seems to work just fine in it... :confused:

Yeah, I forgot to mention that recent Linux kernels have support for it directly in the Kernel. On kernels that support it, it's the best option for speed. However, we have stuck with e1000 for consistency's sake until all of our legacy Linux guests are upgraded to 3.x kernels.
 
You can get faster speeds with VMXNET if your guest OS supports it properly. It's not emulating a physical NIC and isn't subject to the same limitations. You will need VMWare Tools installed in the guest to use VMXNET, which can sometimes cause issues when updating the guest OS. If VMWare Tools gets broken, the VMXNET controller can break as well. Our production VMs usually use e1000, but we use VMXNET for some lab and Windows guests.

I am considering tweaking pfSense to see if I can get it to work properly.

Will I see any benefits from going to VMXNET3 in a simple in -> pfsense -> out configuration, or is VMXNET3 more for faster internal communication between VM's?
 
Zarathustra[H];1038379965 said:
I am considering tweaking pfSense to see if I can get it to work properly.

Will I see any benefits from going to VMXNET3 in a simple in -> pfsense -> out configuration, or is VMXNET3 more for faster internal communication between VM's?

You will only see a benefit going between VMs, or in a situation where your hardware NIC supports more features than the emulated e1000 NIC, like multiqueue or MSI/MSI-X.
 
The VMXNET3 drivers supplied in VMware tools with ESXi 5.0 are horribly broken when used with Solaris 11 Express and Solaris 11. You will get a continuous stream of error logs and very poor performance. Until revised drivers are shipped you should stick with e1000 if using Solaris Express 11 or Solaris 11.
 
Even prior to 5.0, I would see anomalous behavior. e.g. minifreezes, followed by spurts of data. gave up and went back to e1000, although what you describe sounds far worse :(
 
I'm using VMXNET3 on my 2008 servers. That way they have 10gig connectivity to one another in the event that I need to move a large file. Haven't had a problem thus far since switching from the E1000 driver a few months back.
 
Right, but we're saying that freebsd and/or opensolaris aren't good. If windows works for you, cool.
 
im using the latest open indiana with vmxnet 3 and dont see any issues so far... where should i be looking for these errors in what log files?
 
What I saw was very erratic performance. Spiky throughput, intervals for a good fraction of a second where nothing was flowing. It's by no means universal, but more than a few people have seen all kinds of odd behavior.
 
im using the latest open indiana with vmxnet 3 and dont see any issues so far... where should i be looking for these errors in what log files?

You'll find them on your console log. If you are having issues with it then you'll see hundreds or even thousands of error logs from the VMXNET3 driver.

Open Indiana is more closely related to later builds of Solaris 10 (OI151a/IIllumos), which did not seem to have the issues - or at least as many issues - with the VMXNET3 drivers from VMware.

While VMXNET3 may work well on Solaris 10 and/or OI, it is definitely broken on SE11 and Solaris 11.
 
You'll find them on your console log. If you are having issues with it then you'll see hundreds or even thousands of error logs from the VMXNET3 driver.

Open Indiana is more closely related to later builds of Solaris 10 (OI151a/IIllumos), which did not seem to have the issues - or at least as many issues - with the VMXNET3 drivers from VMware.

While VMXNET3 may work well on Solaris 10 and/or OI, it is definitely broken on SE11 and Solaris 11.

not sure what console log is but i looked around the log files on my OI install

getting a ton of these in /var/adm/messages

vmxnet3s:0: getcapab(0x200000) -> no


i dont like this. going to try the old nic type
 
Have you tried Flexible NIC ? It works only on 32bit OS - I use it on my OpenMediaVault VM and i got up to 100MB/s transfer. VMXNET3 works only in windows for me.
On FreeBSD/FreeNAS i get terrible data transfers. max is about 30-35MB/s on e1000 NIC
 
Flexible defaults (IIRC) to a pcnet NIC if you can't support the vmxnet way. If you're going to use an emulated NIC anyway, why not the e1000?
 
Have you tried Flexible NIC ? It works only on 32bit OS - I use it on my OpenMediaVault VM and i got up to 100MB/s transfer. VMXNET3 works only in windows for me.
On FreeBSD/FreeNAS i get terrible data transfers. max is about 30-35MB/s on e1000 NIC

I ran into the same 30-35 MB/s cieling with BSD guests using Xen hypervisor with either emulated e1000 or paravirtual drivers for which I had to specifically compile the whole thing from scratch. PCI passthrough speeds of an e1000e are normal though. Maybe it's a BSD thing.
 
I ran into the same 30-35 MB/s cieling with BSD guests using Xen hypervisor with either emulated e1000 or paravirtual drivers for which I had to specifically compile the whole thing from scratch. PCI passthrough speeds of an e1000e are normal though. Maybe it's a BSD thing.

Is that B(yte) or b(it)?
 
Back
Top