Windows Server 2012 on ESXi keep losing network connection

Discussion in 'Virtualized Computing' started by maw, Apr 16, 2013.

  1. maw

    maw [H]ardness Supreme

    Messages:
    4,126
    Joined:
    Sep 27, 2000
    Hi all,

    I was hoping someone could offer some help with this. I'm pretty new to ESXi, but I managed to it set up and install a Server 2012 VM. I thought everything was running smoothly, until I noticed that every couple of days the VM would simply lose network connection (the network icon in the taskbar shows it's disconnected). It will come back up if I run the network repair tool, or restart the VM. I get nothing in the event viewer except a DNS error saying it cannot resolve the address of the time server,which makes sense if it thinks it's disconnected from the network.

    Just to be clear, I can always access the ESXi host server over the network just fine using the VMWare Client tool, and I can even access the Server 2012 VM from within the VMWare client tool console, so it's not the physical machine or ESXi that is losing connection, just the VM itself that is losing connection.

    I'm just not sure where to start with troubleshooting this problem short of starting all over, or re-installing Server 2012 on it's own physical machine.

    Has anyone heard of this issue? Is it something to do with VMWare, Server 2012 itself, or Server 2012 running in a VM?

    The hardware specs of the machine are pretty ancient, but seems to work fine otherwise:
    Supermicro H8SSL-I 939 mobo
    AMD Opteron 165
    4 GB DDR
    WD 500GB HDD
    2 built-in Broadcom Gigabit ethernet (only one is connected to the network)
    The VM has its own static IP in the same subnet as the VMWare host machine (ESXi uses 192.168.0.11, Server 2012 VM uses 192.168.0.1)
     
    Last edited: Apr 16, 2013
  2. Wrench00

    Wrench00 2[H]4U

    Messages:
    3,423
    Joined:
    Sep 30, 2003
    I have experienced this problem as well. What I found to work for me is to install Vmxnet 2 or 3 adapter and remove the e1000.

    If you running a DC I also found that shuffling your dns forwarders helps from IPV6 to IPV 4.

    Its one of the weird bugs I have run across in 2012. Its full of fun hidden bugs. :D
     
  3. maw

    maw [H]ardness Supreme

    Messages:
    4,126
    Joined:
    Sep 27, 2000
    I'm sorry, silly question. But how do I remove and install different adapters in ESXi?
     
  4. Wrench00

    Wrench00 2[H]4U

    Messages:
    3,423
    Joined:
    Sep 30, 2003
    Go to vsphere right click on the vm - edit settings - click add - Network adapter Choose VMXNET 3. (You need vmware tools installed) Windows will automatically detect and install the hardware for you.
    Go back to settings of the VM and select the other adapter and click remove.
     
  5. maw

    maw [H]ardness Supreme

    Messages:
    4,126
    Joined:
    Sep 27, 2000

    OK, thanks. I'll try that.
     
  6. maw

    maw [H]ardness Supreme

    Messages:
    4,126
    Joined:
    Sep 27, 2000
    OK, after two days it looks like changing over to the VMXNET 3 adatper works. Thanks!
     
  7. Wrench00

    Wrench00 2[H]4U

    Messages:
    3,423
    Joined:
    Sep 30, 2003
    Glad that I could help :D
     
  8. t-readyroc

    t-readyroc Gawd

    Messages:
    560
    Joined:
    May 15, 2002
    Just wanted to post here for visibility, because we experienced the same issue, and changing the adapter worked for us as well. So, thanks.

    VM initially was installed with default options which gave the machine an e1000E network adapter
    • network adapter listed in devmgr as Intel 82574L
    • subsequently suffered from continuous network drop out
    • RDP session would go blank & reconnect within ~15 seconds
    • Windows event viewer flooded with EventID 27 with sources e1iexpress & e1qexpress
    • removed e1000E adapter from within VMWare & added e1000 adapter
    • network adapter now listed in devmgr as Intel PRO/1000MT
    • no connection drop-outs since the switch

    Very odd issue. Especially since we have another VM on the same physical server w/the same OS & the same (82574L) adapter installed that isn't experiencing these issues at all... We'd even tried updating the NIC drivers to the latest from Intel w/o success.
     
  9. Wrench00

    Wrench00 2[H]4U

    Messages:
    3,423
    Joined:
    Sep 30, 2003
    I am not sure what the actual problem is either. Its puzzling to me. I think its a new driver model that Server2012 users since the network cards share a single processor I am thinking that vmware drivers for e1000e have an issue. Thats why its effecting only certain machines.
     
  10. Child of Wonder

    Child of Wonder 2[H]4U

    Messages:
    3,220
    Joined:
    May 22, 2006
    I wonder if Win2012 is trying to use RSS or RDMA and the virtual E1000E adapter can't handle it.
     
  11. cryptz

    cryptz n00bie

    Messages:
    5
    Joined:
    Nov 16, 2011
    I just wanted to say i am running esxi 5.5 and i have this problem with server 2012 also. I use vmxnet3 for everything and i am still having this problem. we originally used vmxnet3 because we have a 10g infrastructure. Obviously the provided fix isnt working for us. any other thoughts?
     
  12. Brahmzy

    Brahmzy [H]ardness Supreme

    Messages:
    4,338
    Joined:
    Sep 9, 2004
    Has anybody tried 2012R2 and experienced the issue with a VMXNET3 adapter?
     
  13. ND40oz

    ND40oz [H]ardForum Junkie

    Messages:
    11,060
    Joined:
    Jul 31, 2005
    Running it currently on a few 2012/2012R2 boxes, no issues here with vmxnet3 and 5.1u1 Patch 2. I haven't rolled out Patch 3 yet, I am planning on going straight to 5.5 once the holidays are over, hopefully this isn't an issue with 5.5?
     
  14. lopoetve

    lopoetve Imhotep

    Messages:
    28,724
    Joined:
    Oct 11, 2001
    Intel/Microsoft actually provides the driver, and since it's a dead card (the revision emulated), I wouldn't be surprised if a bug has slipped in...
     
  15. lopoetve

    lopoetve Imhotep

    Messages:
    28,724
    Joined:
    Oct 11, 2001
    Well, there IS an issue we're investigating where if your hardware cards in the host offer offload, under very high load it'll start dropping some packets as we try to hand off ops to the network card, but that's VERY high load sustained for a long time. And requires specific hardware cards as well.
     
  16. dasaint

    dasaint [H]ard|Gawd

    Messages:
    1,718
    Joined:
    Jun 1, 2002
  17. fxs

    fxs n00bie

    Messages:
    1
    Joined:
    Jan 9, 2014
    same issue here,

    vm's loses network every few days. disable/enable on OS level re-instates connection

    ESX 5.5
    windows server 2012 R2
    E1000 driver

    I excluded the switch or physical esx nic as problem source.

    i just reconfigured to VMXNET3 driver.
    I ll post the result in a few days
     
  18. QHalo

    QHalo 2[H]4U

    Messages:
    3,434
    Joined:
    Sep 30, 2002
    I'm curious why people running Windows machines are using E1000 to begin with, especially on 2012 boxes?
     
  19. peanuthead

    peanuthead [H]ardness Supreme

    Messages:
    4,205
    Joined:
    Feb 1, 2006
    Exactly! Always try and run Vmxnet3, then Vmxnet2, then E1000E/E1000 in that order unless you have a specific reason not too.
     
  20. lopoetve

    lopoetve Imhotep

    Messages:
    28,724
    Joined:
    Oct 11, 2001
    Forgot to change it on a template >_<

    Now changing it on a dozen production vms is a pain (downtime).
     
  21. nicholasfarmer

    nicholasfarmer Limp Gawd

    Messages:
    231
    Joined:
    Apr 19, 2007
    Quick note on this from my recent testing:
    The last few days I've ran into issues with 10Gb networks and e1000 > vmxnet3 adapters.

    Interesting things I found:
    1 - if you have a VM running the e1000 adapter and you push it over the 1.5Gbps limit, it makes the network stack barf and traffic shutters until it does a soft stack reset.

    2 - Receiving data at 10Gbps appears to be easy but getting a VM with a VMXNet3 adapter to transmit 10Gbps is painful!

    Both of these resulted in testing with a Linux and Windows VM so (OS independent)

    Nick
     
  22. QHalo

    QHalo 2[H]4U

    Messages:
    3,434
    Joined:
    Sep 30, 2002
    LOL. Well done sir. I left a damn ISO attached to mine. That sucks too.
     
  23. lopoetve

    lopoetve Imhotep

    Messages:
    28,724
    Joined:
    Oct 11, 2001
    We don't, but it does do some offload.

    Had one of those too - only did three machines, and at least dropping a template is easy :D

    Lemme do some more digging.
     
  24. rufik

    rufik Limp Gawd

    Messages:
    151
    Joined:
    Sep 16, 2013
  25. MWG8

    MWG8 n00bie

    Messages:
    1
    Joined:
    Feb 12, 2014
    I have a very similar issue with our 2012 servers (ESXi5). The server's network (e1000) seemingly randomly develops a problem which can be resolved by Troubleshooting Problems.

    We have tried using Vmxnet3 but the OS hangs when any configuration is made. The network card is recognised by Windows (as long as VMware tools are installed) and device manager indicates it is working ok.

    I would appreciate any suggestions and will continue investigate/test.
     
  26. lopoetve

    lopoetve Imhotep

    Messages:
    28,724
    Joined:
    Oct 11, 2001
    You have a bigger problem if windows crashes when you switch to vmxnet.

    that being said - get the latest driver from intel - it seems to help significantly.
     
  27. frankhuzzah

    frankhuzzah [H]Lite

    Messages:
    96
    Joined:
    Apr 18, 2007
    I've been experiencing a similar problem with 2012 R2 on 5.1. Initially trying VMXNET3, then E1000e adapters both with about 4-8% packet loss. What was baffling to me was that my domain controller, also 2012 R2, didn't seem to have this problem. I deployed the DC probably a month before the other servers and set the firmware boot option to EFI instead of BIOS. I made a new VM with EFI and it seems to be functioning perrfectly, so far zero dropped packets. I'll have to redeploy as switching the setting on an existing VM prevented it from booting.

    EDIT: I take this back, my problem seems to be with Stablebit Scanner, not with ESXi.
     
    Last edited: Feb 12, 2014
  28. TabernacST

    TabernacST n00bie

    Messages:
    1
    Joined:
    Apr 7, 2014
    We began deploying 2012R2 a few months back. Never experienced any problems with this on 2008R2 (we weren't using VMXNET3).

    After a couple weeks we started seeing network connectivity just completely drop out of the 2012R2 VMs, just like everyone has seen here - accessible by the console, can vmotion, etc. Sometimes to get connectivity back we would have to completely shutdown the VM and vmotion it to another host.

    We figured it was because we were using the old driver so we overhauled all of our 2012R2 boxes to use the VMXNET3 adapter. It seems to be better, but just had a NIC drop over the weekend. Fix = vmotion.

    We're running ESX 5.1 build 1312873. Hardware is DL585G5s with Intel NC364T. Apparently the host side driver for that is e1000e. I don't believe that can be changed.

    Any luck with fixes?
     
  29. Scott-NC

    Scott-NC n00bie

    Messages:
    2
    Joined:
    Aug 19, 2014
    Good day. I just wanted to say "thanks"...I've been experiencing this precise issue in a recent new configuration of VMware and Server 2012. I had disabled all the TCP offload features and was at my wit's end.

    Your excellent description of the issue helped me find this page and I joined the forum just to post a thanks to the OP and the responders.

    I will know in a week if this fixed it for sure, but I have hopes that it will. Again...this matches my problem to a "T", so I'm hoping the solution holds.

    Just curious...is the VMXNET available to clients before they have vmtools installed, or only after the vmtools is installed?

    Cheers!
     
    Last edited: Aug 19, 2014
  30. peanuthead

    peanuthead [H]ardness Supreme

    Messages:
    4,205
    Joined:
    Feb 1, 2006
    Only after vmtools is installed.
     
  31. Scott-NC

    Scott-NC n00bie

    Messages:
    2
    Joined:
    Aug 19, 2014
    Okay...I've been using this solution for several weeks and it appears to have done the trick for me. I've not had an instance where I've had to restart the server or disable/enable the network cards since making this change.

    Thanks again for the good advice. It took a while to find this solution on this forum (took come creative wording of the search in search engines), but this did the trick.

    Cheers!
     
  32. SNOWHNTR

    SNOWHNTR n00bie

    Messages:
    1
    Joined:
    Apr 27, 2017
    Hi All,
    New to this forum.
    I know this is an old thread, but I just got here by searching this issue. It seems I have a similar issue and would like some advise on how to resolve.
    My setup, problem, and dilemma:
    Dell R710 running ESXi 5.5, and two guests running 2012 R2. One guest is a AD/DC and DNS, and the other is just a file server. The R710 has 4 - GB Broadcom's and they are all cabled up. The 2012R2 AD/DC server has detected an Intel 82574L, and has no issues, but the File server (exact same OS) detects an Intel Pro 1000 MT. The one with the Pro 1000MT just drops connectivity every few days. I can still access the FS via VM workstation and with the vSphere Client, and just doing a "Troubleshoot Problems" on the adaptor fixes the issue. The DC guest has had no issues.
    The Guest that sees the Intel Pro 1000MT has a E1000 as a adaptor in Vm, and the Intel 82574L is seen from an E1000E, hence why Windows is detecting two different adaptors.
    My thought is to change the VM adaptor on the offending guest OS from the E1000 to a VMXNET3, or at least to an E1000E.
    My dilemma is that the server is in Texas, and I am in Ohio, and I don't want to screw this up and have to board a plane to Texas. Too hot there right now lol
    SO what is a good procedure to change (or add new and remove old) the E1000 to a VMXNET3 WITHOUT losing access to OS, VM, etc.

    Please and Thank You!
     
  33. lopoetve

    lopoetve Imhotep

    Messages:
    28,724
    Joined:
    Oct 11, 2001
    As long as you have access to the host, you can fix it - you're going to lose network connectivity for a bit there since it will show up as a new network adapter to windows, and you'll have to configure that one, but the host will be unaffected. You can use the console to make the changes to the guest.
     
  34. bigdogchris

    bigdogchris Wii was a Novelty

    Messages:
    16,509
    Joined:
    Feb 19, 2008
    I haven't checked but I believe the e1000 adapter uses a native driver. Obviously VMXNET3 uses a VMware driver. I know the e1000 driver works a lot better in some older Linux distros though.
     
  35. bman212121

    bman212121 Gawd

    Messages:
    991
    Joined:
    Aug 18, 2011
    You need to switch out the E1000e adapter for the VMXNET3. I've never seen that issue actually fixed, and if I have issues with a server it's usually that. Here are several links all talking about the same issue, and they claim it was fixed twice.... Just switch everything over to VMXNET3 and you won't have to worry about it again.

    https://kb.vmware.com/selfservice/m...nguage=en_US&cmd=displayKC&externalId=2109922

    https://communities.vmware.com/thread/433792

    https://kb.vmware.com/selfservice/m...nguage=en_US&cmd=displayKC&externalId=2072694



    You need to use the Console access from a VMware client. (Vsphere client, Web GUI, or VMWare workstation) If you have direct access to the console, you do NOT need the VM to have an internet connection for it to work. That's the advantage of using Virtual machines in that it's not a physical piece of hardware. You have full access to the BIOS and virtual hardware at any time, from anywhere.

    So basically fire up the VSphere Client, and login to the host. If you have a static IP be sure to change it back to automatic or else Windows will complain when you try to add the new adapter. After that's done power down the VM. Then edit the VM from vsphere client, remove the old adapter. Create a new adapter, and you'll have to expand the advanced properties. Switch it from E1000e to VMXNET3. Set the Vlan if needed. Hit Okay and then power the server back on. Once it's powered up login to it. Go into the network adapter settings, and set your static IP again. After that you should be good to go.

    The only gotchas I can think of is if for some reason you have a policy that doesn't cache domain credentials on your server. If so then you'll probably need a local account because if it loses internet connection, you won't be able to login. But given that the servers have already broken and you logged in, I'm going to assume that won't be an issue. The only other thing that could happen is if for some reason you go back to automatic and that server picks up a DHCP lease, that will usually end up getting published back to AD. The issue will sort itself out in time, but it will be annoying for a little bit because the static IP of the server will be wrong in DNS.
     
    Last edited: Apr 29, 2017
  36. Grimlaking

    Grimlaking [H]ard|Gawd

    Messages:
    1,597
    Joined:
    May 9, 2006
    Thankfully I haven't seen this issue at all in ESXi 6.0 From base up through U3 so maybe it's fixed here.