40/10Gb Ethernet under $500 (noisy but fast) HowTo

[vxspiritxv]

Just thought I would put this out there for those looking for speed on the cheap, most items ebay of course.

Here is a list of compatible parts:
Ebay switch $400 search: Cisco N3K-C3064PQ-10GX
Ebay 40gbit network cards $43 search: 79DJ3 Mellanox ConnectX-3
My knowledge of Mellanox is limited, but I believe "ConnectX-3 Pro" and "ConnectX-3 VPI" cards are the ones ethernet compatible, rest of the ConnectX-3 cards are infiniband cards. More info below on setting up the Dell card.
Intel x520 (pcie 2.0) cards are about $45
H! Fiber SFP-10G-T-S SFP work, for 10Gbit copper connections $40
Ebay fiber SFP-10G-SR are currently under $7 !
Bit more rare on ebay, QSFP-H40G-CU3M (change the 3 with other lengths) , I paid $27 , these work in Mellanox.

Now for the Mellanox info, this is specific to ESXi 6.7, but likely works on all OSes.
You will need management tools: http://www.mellanox.com/page/management_tools
For me that's currently:
mft-4.13.3.6-10EM-650.0.0.4598673.x86_64.vib
nmst-4.13.3.6-1OEM.650.0.0.4598673.x86_64.vib
I don't know if its needed, but I had my esxi in community mode (driver signing)
CLI Install commands:
esxcli software vib install -v /vmfs/volumes/XXXXXXXXXXXXXXX/mft-4.13.3.6-10EM-650.0.0.4598673.x86_64.vib --no-sig-check
esxcli software vib install -v /vmfs/volumes/XXXXXXXXXXXXXXX/nmst-4.13.3.6-1OEM.650.0.0.4598673.x86_64.vib --no-sig-check

/opt/mellanox/bin/mlxconfig query

Device #1:
----------

Device type: ConnectX3
Device: mt4099_pciconf0

Configurations: Next Boot
SRIOV_EN True(1)
NUM_OF_VFS 4
LINK_TYPE_P1 N/A
LINK_TYPE_P2 N/A​

/opt/mellanox/bin/mlxfwmanager --query

Querying Mellanox devices firmware ...

Device #1:
----------

Device Type: ConnectX3
Part Number: 079DJ3
Description: ConnectX-3 VPI adapter; single-port QSFP; FDR IB (56Gb/s) and 40GbE;PCIe3.0 x8 8GT/s; Dell PowerEdge
PSID: DEL1100001019
PCI Device Name: mt4099_pci_cr0
Port1 MAC: e41d2db2bc81
Port2 MAC: e41d2db2bc82
Versions: Current Available
FW 2.32.5100 N/A
PXE 3.4.0306 N/A

Status: No matching image found
​

Not on FW 2.42? Might want to update that...
https://www.mellanox.com/page/firmware_table_dell?mtag=oem_firmware_download
fw-ConnectX3-rel-2_42_5000-079DJ3-FlexBoot-3.4.752.bin

/opt/mellanox/bin/flint -d mt4099_pciconf0 -i /vmfs/volumes/XXXXXXXXXXXXXXX/fw-ConnectX3-rel-2_42_5000-079DJ3-FlexBoot-3.4.752.bin burn
/opt/mellanox/bin/mlxconfig -d mt4099_pciconf0 reset
reboot
/opt/mellanox/bin/mlxconfig -d mt4099_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2
reboot

Now if you query it, it should look like this and work like any other ethernet card:
/opt/mellanox/bin/mlxconfig query

Device #1:
----------

Device type: ConnectX3
Device: mt4099_pciconf0

Configurations: Next Boot
SRIOV_EN False(0)
NUM_OF_VFS 8
LINK_TYPE_P1 ETH(2)
LINK_TYPE_P2 ETH(2)
LOG_BAR_SIZE 3
BOOT_PKEY_P1 0
BOOT_PKEY_P2 0
BOOT_OPTION_ROM_EN_P1 False(0)
BOOT_VLAN_EN_P1 False(0)
BOOT_RETRY_CNT_P1 0
LEGACY_BOOT_PROTOCOL_P1 None(0)
BOOT_VLAN_P1 1
BOOT_OPTION_ROM_EN_P2 False(0)
BOOT_VLAN_EN_P2 False(0)
BOOT_RETRY_CNT_P2 0
LEGACY_BOOT_PROTOCOL_P2 None(0)
BOOT_VLAN_P2 1
IP_VER_P1 IPv4(0)
IP_VER_P2 IPv4(0)
CQ_TIMESTAMP True(1)​

sh interface Ethernet1/49

Ethernet1/49 is up
admin state is up, Dedicated Interface
Hardware: 40000 Ethernet, address: 1c6a.xxxx.xxxx (bia 1c6a.xxxx.xxxx)
MTU 1500 bytes, BW 40000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is access
full-duplex, 40 Gb/s, media type is 40G
Beacon is turned off
Auto-Negotiation is turned on FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
admin fec state is auto, oper fec state is off
Last link flapped 00:12:05
Last clearing of "show interface" counters never
4 interface resets
Load-Interval #1: 30 seconds
30 seconds input rate 1992 bits/sec, 1 packets/sec
30 seconds output rate 5656 bits/sec, 1 packets/sec
input rate 1.99 Kbps, 1 pps; output rate 5.66 Kbps, 1 pps
Load-Interval #2: 5 minute (300 seconds)
300 seconds input rate 336 bits/sec, 0 packets/sec
300 seconds output rate 1312 bits/sec, 0 packets/sec
input rate 336 bps, 0 pps; output rate 1.31 Kbps, 0 pps
RX
72 unicast packets 3 multicast packets 15 broadcast packets
90 input packets 10207 bytes
0 jumbo packets 0 storm suppression packets
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
67 unicast packets 1151 multicast packets 6 broadcast packets
1224 output packets 128401 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause​