FreeBSD Based Guests in ESXi with VMXNET3?

Discussion in 'Virtualized Computing' started by Zarathustra[H], Apr 1, 2015.

  1. Zarathustra[H]

    Zarathustra[H] Pick your own.....you deserve it.

    Messages:
    23,177
    Joined:
    Oct 29, 2000
    Hey all,

    I stumbled across this strange behavior in another thread.

    At first I thought it was a problem with my setup, but MySongRanHills had the same issue.


    So the issue is this:

    VMXNet3 works fine for other guests. registering ~13Gbit/s as measured in iPerf between them, until I test with FreeNAS (9.3)

    I get ~21Gbit/s when Ubuntu is the server (iperf -s) and FreeNAS is the client (iperf -c) but only ~3.2Gbit/s when FreeNAS is the server and Ubuntu is the client.

    iperf output:

    Two Iperf tests with FreeNAS as a server (iperf -s) and Ubuntu Server as client (iperf -c):

    Code:
    ------------------------------------------------------------
    Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
    TCP window size: 85.0 KByte (default)
    ------------------------------------------------------------
    [  3] local xxx.xxx.xxx.xxx port 33485 connected with xxx.xxx.xxx.xxx port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.0 sec  3.79 GBytes  3.25 Gbits/sec
    
    ------------------------------------------------------------
    Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
    TCP window size: 85.0 KByte (default)
    ------------------------------------------------------------
    [  3] local xxx.xxx.xxx.xxx port 43405 connected with xxx.xxx.xxx.xxx port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.0 sec  3.69 GBytes  3.17 Gbits/sec
    Two Iperf tests with Ubuntu Server as a server (iperf -s) and FreeNAS as client (iperf -c):

    Code:
    ------------------------------------------------------------
    Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
    TCP window size: 32.5 KByte (default)
    ------------------------------------------------------------
    [  3] local xxx.xxx.xxx.xxx port 53941 connected with xxx.xxx.xxx.xxx port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.0 sec  24.9 GBytes  21.4 Gbits/sec
    
    ------------------------------------------------------------
    Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
    TCP window size: 32.5 KByte (default)
    ------------------------------------------------------------
    [  3] local xxx.xxx.xxx.xxx port 64555 connected with xxx.xxx.xxx.xxx port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.0 sec  25.5 GBytes  21.9 Gbits/sec

    I recently got a couple of Brocades to get better transfer speeds between FreeNAS and my workstation, but this issue is holding me back.

    Any ideas what might be going on, and how I can fix it?
     
  2. BrianAz

    BrianAz n00bie

    Messages:
    3
    Joined:
    Sep 16, 2010
    Sorry for posting on such an old thread... couldn't find anyone others with my exact issue.


    Thanks for your post!

    I was experiencing similar issues running a FreeNAS VM on top of ESXi 5.5. I eventually setup a second virtual switch and added another interface to my FreeNAS and Ubuntu VMs to create a storage network. I enabled Jumbo Frames on this second network. My speeds between Ubuntu VMs jumped from ~ 9 Gbits/sec to ~ 14 Gbits/sec. The best part though, is that my Ubuntu->FreeNAS iperf results went from ~ 3-4Gbits/sec to ~ 10 Gbits/sec.



    Code:
    Last login: Sun Jul  9 13:44:59 2017 from 192.168.30.91
    FreeBSD 10.3-STABLE (FreeNAS.amd64) #0 r295946+21897e6695f(HEAD): Thu Jun  8 20:02:12 UTC 2017
    
        FreeNAS (c) 2009-2016, The FreeNAS Development Team
        All rights reserved.
        FreeNAS is released under the modified BSD license.
    
        For more information, documentation, help or support, go here:
         http://freenas.org
    Welcome to FreeNAS
    [root@freenas] ~# iperf -s
    ------------------------------------------------------------
    Server listening on TCP port 5001
    TCP window size: 64.0 KByte (default)
    ------------------------------------------------------------
    [  4] local 192.168.40.3 port 5001 connected with 192.168.40.4 port 44688
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0-10.0 sec  12.1 GBytes  10.4 Gbits/sec
    ^C[root@freenas] ~# iperf -c 192.168.40.4
    ------------------------------------------------------------
    Client connecting to 192.168.40.4, TCP port 5001
    TCP window size: 35.0 KByte (default)
    ------------------------------------------------------------
    [  3] local 192.168.40.3 port 39684 connected with 192.168.40.4 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.0 sec  27.3 GBytes  23.5 Gbits/sec
    [root@freenas] ~#
    I am still perplexed with how FreeNASVM->UbuntuVM is 23Gbits/sec.. That's 10Gbits/sec faster than my UbuntuVM<->UbuntuVM results and 13Gbits/sec faster than UbuntuVM->FreeNASVM.
     
  3. lopoetve

    lopoetve Imhotep

    Messages:
    28,956
    Joined:
    Oct 11, 2001
    Are yours on the same host, or different?
     
  4. BrianAz

    BrianAz n00bie

    Messages:
    3
    Joined:
    Sep 16, 2010
    All same. Have only one ESXi server.
     
  5. lopoetve

    lopoetve Imhotep

    Messages:
    28,956
    Joined:
    Oct 11, 2001
    What are the results on the E1000 virtual cards instead? there have been issues with VMXNET on some of the BSD and oddball OSes (solaris).
     
  6. BrianAz

    BrianAz n00bie

    Messages:
    3
    Joined:
    Sep 16, 2010

    e1000 adapter in my FreeNAS VM. My Ubuntu iPerf test VMs always ran the VMXNET3. No Jumbo Frames:


    Code:
    iPerfTestClient (VMXNET 3) -> FreeNAS (e1000) - ~ 1.7Gbits/sec
    brian@iPerfTestClient:~$ iperf -c 192.168.30.101
    ------------------------------------------------------------
    Client connecting to 192.168.30.101, TCP port 5001
    TCP window size: 85.0 KByte (default)
    ------------------------------------------------------------
    [  3] local 192.168.30.239 port 46434 connected with 192.168.30.101 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.0 sec  2.02 GBytes  1.74 Gbits/sec
    
    
    FreeNAS (e1000) -> iPerfTestServer (VMXNET 3) - ~ 2.5 Gbits/sec
    [root@freenas ~]# iperf -c 192.168.30.244                                                                                           
    ------------------------------------------------------------                                                                       
    Client connecting to 192.168.30.244, TCP port 5001                                                                                 
    TCP window size: 32.5 KByte (default)                                                                                               
    ------------------------------------------------------------                                                                       
    [  6] local 192.168.30.101 port 15085 connected with 192.168.30.244 port 5001                                                       
    [ ID] Interval       Transfer     Bandwidth                                                                                         
    [  6]  0.0-10.0 sec  3.02 GBytes  2.59 Gbits/sec                                                                                   
    [root@freenas ~]# iperf -c 192.168.30.244                                                                                           
    ------------------------------------------------------------                                                                       
    Client connecting to 192.168.30.244, TCP port 5001                                                                                 
    TCP window size: 32.5 KByte (default)                                                                                               
    ------------------------------------------------------------                                                                       
    [  6] local 192.168.30.101 port 63030 connected with 192.168.30.244 port 5001                                                       
    [ ID] Interval       Transfer     Bandwidth                                                                                         
    [  6]  0.0-10.0 sec  2.91 GBytes  2.50 Gbits/sec                          
    In the end, I chalked it up to issues with FreeBSD and the VM NIC. Hoping that once FreeNAS 11 is stable enough to upgrade to this'll be resolved. I need to find time to setup a VM to test the RCs with.

    Thanks,
    Brian