Windows Server 2012 on ESXi keep losing network connection

M

maw

Supreme [H]ardness
Joined
Sep 27, 2000
Messages
4,135
Hi all,

I was hoping someone could offer some help with this. I'm pretty new to ESXi, but I managed to it set up and install a Server 2012 VM. I thought everything was running smoothly, until I noticed that every couple of days the VM would simply lose network connection (the network icon in the taskbar shows it's disconnected). It will come back up if I run the network repair tool, or restart the VM. I get nothing in the event viewer except a DNS error saying it cannot resolve the address of the time server,which makes sense if it thinks it's disconnected from the network.

Just to be clear, I can always access the ESXi host server over the network just fine using the VMWare Client tool, and I can even access the Server 2012 VM from within the VMWare client tool console, so it's not the physical machine or ESXi that is losing connection, just the VM itself that is losing connection.

I'm just not sure where to start with troubleshooting this problem short of starting all over, or re-installing Server 2012 on it's own physical machine.

Has anyone heard of this issue? Is it something to do with VMWare, Server 2012 itself, or Server 2012 running in a VM?

The hardware specs of the machine are pretty ancient, but seems to work fine otherwise:
Supermicro H8SSL-I 939 mobo
AMD Opteron 165
4 GB DDR
WD 500GB HDD
2 built-in Broadcom Gigabit ethernet (only one is connected to the network)
The VM has its own static IP in the same subnet as the VMWare host machine (ESXi uses 192.168.0.11, Server 2012 VM uses 192.168.0.1)
 
Last edited:
I have experienced this problem as well. What I found to work for me is to install Vmxnet 2 or 3 adapter and remove the e1000.

If you running a DC I also found that shuffling your dns forwarders helps from IPV6 to IPV 4.

Its one of the weird bugs I have run across in 2012. Its full of fun hidden bugs. :D
 
Wrench00 said:
I have experienced this problem as well. What I found to work for me is to install Vmxnet 2 or 3 adapter and remove the e1000.

If you running a DC I also found that shuffling your dns forwarders helps from IPV6 to IPV 4.

Its one of the weird bugs I have run across in 2012. Its full of fun hidden bugs. :D
Click to expand...

I'm sorry, silly question. But how do I remove and install different adapters in ESXi?
 
maw said:
I'm sorry, silly question. But how do I remove and install different adapters in ESXi?
Click to expand...

Go to vsphere right click on the vm - edit settings - click add - Network adapter Choose VMXNET 3. (You need vmware tools installed) Windows will automatically detect and install the hardware for you.
Go back to settings of the VM and select the other adapter and click remove.
 
Wrench00 said:
Go to vsphere right click on the vm - edit settings - click add - Network adapter Choose VMXNET 3. (You need vmware tools installed) Windows will automatically detect and install the hardware for you.
Go back to settings of the VM and select the other adapter and click remove.
Click to expand...


OK, thanks. I'll try that.
 
OK, after two days it looks like changing over to the VMXNET 3 adatper works. Thanks!
 
Just wanted to post here for visibility, because we experienced the same issue, and changing the adapter worked for us as well. So, thanks.

VM initially was installed with default options which gave the machine an e1000E network adapter
  • network adapter listed in devmgr as Intel 82574L
  • subsequently suffered from continuous network drop out
  • RDP session would go blank & reconnect within ~15 seconds
  • Windows event viewer flooded with EventID 27 with sources e1iexpress & e1qexpress
  • removed e1000E adapter from within VMWare & added e1000 adapter
  • network adapter now listed in devmgr as Intel PRO/1000MT
  • no connection drop-outs since the switch

Very odd issue. Especially since we have another VM on the same physical server w/the same OS & the same (82574L) adapter installed that isn't experiencing these issues at all... We'd even tried updating the NIC drivers to the latest from Intel w/o success.
 
t-readyroc said:
Just wanted to post here for visibility, because we experienced the same issue, and changing the adapter worked for us as well. So, thanks.

VM initially was installed with default options which gave the machine an e1000E network adapter
  • network adapter listed in devmgr as Intel 82574L
  • subsequently suffered from continuous network drop out
  • RDP session would go blank & reconnect within ~15 seconds
  • Windows event viewer flooded with EventID 27 with sources e1iexpress & e1qexpress
  • removed e1000E adapter from within VMWare & added e1000 adapter
  • network adapter now listed in devmgr as Intel PRO/1000MT
  • no connection drop-outs since the switch

Very odd issue. Especially since we have another VM on the same physical server w/the same OS & the same (82574L) adapter installed that isn't experiencing these issues at all... We'd even tried updating the NIC drivers to the latest from Intel w/o success.
Click to expand...

I am not sure what the actual problem is either. Its puzzling to me. I think its a new driver model that Server2012 users since the network cards share a single processor I am thinking that vmware drivers for e1000e have an issue. Thats why its effecting only certain machines.
 
Wrench00 said:
I am not sure what the actual problem is either. Its puzzling to me. I think its a new driver model that Server2012 users since the network cards share a single processor I am thinking that vmware drivers for e1000e have an issue. Thats why its effecting only certain machines.
Click to expand...

I wonder if Win2012 is trying to use RSS or RDMA and the virtual E1000E adapter can't handle it.
 
I just wanted to say i am running esxi 5.5 and i have this problem with server 2012 also. I use vmxnet3 for everything and i am still having this problem. we originally used vmxnet3 because we have a 10g infrastructure. Obviously the provided fix isnt working for us. any other thoughts?
 
Has anybody tried 2012R2 and experienced the issue with a VMXNET3 adapter?
 
Running it currently on a few 2012/2012R2 boxes, no issues here with vmxnet3 and 5.1u1 Patch 2. I haven't rolled out Patch 3 yet, I am planning on going straight to 5.5 once the holidays are over, hopefully this isn't an issue with 5.5?
 
Wrench00 said:
I am not sure what the actual problem is either. Its puzzling to me. I think its a new driver model that Server2012 users since the network cards share a single processor I am thinking that vmware drivers for e1000e have an issue. Thats why its effecting only certain machines.
Click to expand...

Intel/Microsoft actually provides the driver, and since it's a dead card (the revision emulated), I wouldn't be surprised if a bug has slipped in...
 
Child of Wonder said:
I wonder if Win2012 is trying to use RSS or RDMA and the virtual E1000E adapter can't handle it.
Click to expand...

Well, there IS an issue we're investigating where if your hardware cards in the host offer offload, under very high load it'll start dropping some packets as we try to hand off ops to the network card, but that's VERY high load sustained for a long time. And requires specific hardware cards as well.
 
same issue here,

vm's loses network every few days. disable/enable on OS level re-instates connection

ESX 5.5
windows server 2012 R2
E1000 driver

I excluded the switch or physical esx nic as problem source.

i just reconfigured to VMXNET3 driver.
I ll post the result in a few days
 
I'm curious why people running Windows machines are using E1000 to begin with, especially on 2012 boxes?
 
QHalo said:
I'm curious why people running Windows machines are using E1000 to begin with, especially on 2012 boxes?
Click to expand...

Exactly! Always try and run Vmxnet3, then Vmxnet2, then E1000E/E1000 in that order unless you have a specific reason not too.
 
QHalo said:
I'm curious why people running Windows machines are using E1000 to begin with, especially on 2012 boxes?
Click to expand...

Forgot to change it on a template >_<

Now changing it on a dozen production vms is a pain (downtime).
 
Quick note on this from my recent testing:
The last few days I've ran into issues with 10Gb networks and e1000 > vmxnet3 adapters.

Interesting things I found:
1 - if you have a VM running the e1000 adapter and you push it over the 1.5Gbps limit, it makes the network stack barf and traffic shutters until it does a soft stack reset.

2 - Receiving data at 10Gbps appears to be easy but getting a VM with a VMXNet3 adapter to transmit 10Gbps is painful!

Both of these resulted in testing with a Linux and Windows VM so (OS independent)

Nick
 
Child of Wonder said:
I wonder if Win2012 is trying to use RSS or RDMA and the virtual E1000E adapter can't handle it.
Click to expand...

We don't, but it does do some offload.

QHalo said:
LOL. Well done sir. I left a damn ISO attached to mine. That sucks too.
Click to expand...

Had one of those too - only did three machines, and at least dropping a template is easy :D

nicholasfarmer said:
Quick note on this from my recent testing:
The last few days I've ran into issues with 10Gb networks and e1000 > vmxnet3 adapters.

Interesting things I found:
1 - if you have a VM running the e1000 adapter and you push it over the 1.5Gbps limit, it makes the network stack barf and traffic shutters until it does a soft stack reset.

2 - Receiving data at 10Gbps appears to be easy but getting a VM with a VMXNet3 adapter to transmit 10Gbps is painful!

Both of these resulted in testing with a Linux and Windows VM so (OS independent)

Nick
Click to expand...

Lemme do some more digging.
 
I have a very similar issue with our 2012 servers (ESXi5). The server's network (e1000) seemingly randomly develops a problem which can be resolved by Troubleshooting Problems.

We have tried using Vmxnet3 but the OS hangs when any configuration is made. The network card is recognised by Windows (as long as VMware tools are installed) and device manager indicates it is working ok.

I would appreciate any suggestions and will continue investigate/test.
 
You have a bigger problem if windows crashes when you switch to vmxnet.

that being said - get the latest driver from intel - it seems to help significantly.
 
I've been experiencing a similar problem with 2012 R2 on 5.1. Initially trying VMXNET3, then E1000e adapters both with about 4-8% packet loss. What was baffling to me was that my domain controller, also 2012 R2, didn't seem to have this problem. I deployed the DC probably a month before the other servers and set the firmware boot option to EFI instead of BIOS. I made a new VM with EFI and it seems to be functioning perrfectly, so far zero dropped packets. I'll have to redeploy as switching the setting on an existing VM prevented it from booting.

EDIT: I take this back, my problem seems to be with Stablebit Scanner, not with ESXi.
 
Last edited:
cryptz said:
I just wanted to say i am running esxi 5.5 and i have this problem with server 2012 also. I use vmxnet3 for everything and i am still having this problem. we originally used vmxnet3 because we have a 10g infrastructure. Obviously the provided fix isnt working for us. any other thoughts?
Click to expand...

Brahmzy said:
Has anybody tried 2012R2 and experienced the issue with a VMXNET3 adapter?
Click to expand...

We began deploying 2012R2 a few months back. Never experienced any problems with this on 2008R2 (we weren't using VMXNET3).

After a couple weeks we started seeing network connectivity just completely drop out of the 2012R2 VMs, just like everyone has seen here - accessible by the console, can vmotion, etc. Sometimes to get connectivity back we would have to completely shutdown the VM and vmotion it to another host.

We figured it was because we were using the old driver so we overhauled all of our 2012R2 boxes to use the VMXNET3 adapter. It seems to be better, but just had a NIC drop over the weekend. Fix = vmotion.

We're running ESX 5.1 build 1312873. Hardware is DL585G5s with Intel NC364T. Apparently the host side driver for that is e1000e. I don't believe that can be changed.

Any luck with fixes?
 
Good day. I just wanted to say "thanks"...I've been experiencing this precise issue in a recent new configuration of VMware and Server 2012. I had disabled all the TCP offload features and was at my wit's end.

Your excellent description of the issue helped me find this page and I joined the forum just to post a thanks to the OP and the responders.

I will know in a week if this fixed it for sure, but I have hopes that it will. Again...this matches my problem to a "T", so I'm hoping the solution holds.

Just curious...is the VMXNET available to clients before they have vmtools installed, or only after the vmtools is installed?

Cheers!
 
Last edited:
Okay...I've been using this solution for several weeks and it appears to have done the trick for me. I've not had an instance where I've had to restart the server or disable/enable the network cards since making this change.

Thanks again for the good advice. It took a while to find this solution on this forum (took come creative wording of the search in search engines), but this did the trick.

Cheers!
 
Hi All,
New to this forum.
I know this is an old thread, but I just got here by searching this issue. It seems I have a similar issue and would like some advise on how to resolve.
My setup, problem, and dilemma:
Dell R710 running ESXi 5.5, and two guests running 2012 R2. One guest is a AD/DC and DNS, and the other is just a file server. The R710 has 4 - GB Broadcom's and they are all cabled up. The 2012R2 AD/DC server has detected an Intel 82574L, and has no issues, but the File server (exact same OS) detects an Intel Pro 1000 MT. The one with the Pro 1000MT just drops connectivity every few days. I can still access the FS via VM workstation and with the vSphere Client, and just doing a "Troubleshoot Problems" on the adaptor fixes the issue. The DC guest has had no issues.
The Guest that sees the Intel Pro 1000MT has a E1000 as a adaptor in Vm, and the Intel 82574L is seen from an E1000E, hence why Windows is detecting two different adaptors.
My thought is to change the VM adaptor on the offending guest OS from the E1000 to a VMXNET3, or at least to an E1000E.
My dilemma is that the server is in Texas, and I am in Ohio, and I don't want to screw this up and have to board a plane to Texas. Too hot there right now lol
SO what is a good procedure to change (or add new and remove old) the E1000 to a VMXNET3 WITHOUT losing access to OS, VM, etc.
Please and Thank You!
 
As long as you have access to the host, you can fix it - you're going to lose network connectivity for a bit there since it will show up as a new network adapter to windows, and you'll have to configure that one, but the host will be unaffected. You can use the console to make the changes to the guest.
 
I haven't checked but I believe the e1000 adapter uses a native driver. Obviously VMXNET3 uses a VMware driver. I know the e1000 driver works a lot better in some older Linux distros though.
 
You need to switch out the E1000e adapter for the VMXNET3. I've never seen that issue actually fixed, and if I have issues with a server it's usually that. Here are several links all talking about the same issue, and they claim it was fixed twice.... Just switch everything over to VMXNET3 and you won't have to worry about it again.

https://kb.vmware.com/selfservice/m...nguage=en_US&cmd=displayKC&externalId=2109922

https://communities.vmware.com/thread/433792

https://kb.vmware.com/selfservice/m...nguage=en_US&cmd=displayKC&externalId=2072694



SNOWHNTR said:
SO what is a good procedure to change (or add new and remove old) the E1000 to a VMXNET3 WITHOUT losing access to OS, VM, etc.
Please and Thank You!
Click to expand...

You need to use the Console access from a VMware client. (Vsphere client, Web GUI, or VMWare workstation) If you have direct access to the console, you do NOT need the VM to have an internet connection for it to work. That's the advantage of using Virtual machines in that it's not a physical piece of hardware. You have full access to the BIOS and virtual hardware at any time, from anywhere.

So basically fire up the VSphere Client, and login to the host. If you have a static IP be sure to change it back to automatic or else Windows will complain when you try to add the new adapter. After that's done power down the VM. Then edit the VM from vsphere client, remove the old adapter. Create a new adapter, and you'll have to expand the advanced properties. Switch it from E1000e to VMXNET3. Set the Vlan if needed. Hit Okay and then power the server back on. Once it's powered up login to it. Go into the network adapter settings, and set your static IP again. After that you should be good to go.

The only gotchas I can think of is if for some reason you have a policy that doesn't cache domain credentials on your server. If so then you'll probably need a local account because if it loses internet connection, you won't be able to login. But given that the servers have already broken and you logged in, I'm going to assume that won't be an issue. The only other thing that could happen is if for some reason you go back to automatic and that server picks up a DHCP lease, that will usually end up getting published back to AD. The issue will sort itself out in time, but it will be annoying for a little bit because the static IP of the server will be wrong in DNS.
 
Last edited:
Thankfully I haven't seen this issue at all in ESXi 6.0 From base up through U3 so maybe it's fixed here.
 
You must log in or register to reply here.
Back
Top