ESXi 5.5 Mellanox Drivers

4

402blownstroker

[H]ard|DCer of the Month - Nov. 2012
Joined
Jan 5, 2006
Messages
3,242
I am trying to install the Mellanox drivers on a ESXi 5.5 system and they are failing to install. I have a 483513-B21 card. The drivers I found are:

MLNX-OFED-ESX-1.9.10.0-10EM-510.0.0.799733.zip

When I run the following command to install them, I get the following:

Code: 
 esxcli software vib install -d /tmp/MLNX-OFED-ESX-1.9.10.0-10EM-510.0.0.799733.zip 
Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: Mellanox_bootbank_net-ib-addr_1.9.10.0-1OEM.510.0.0.799733, Mellanox_bootbank_net-ib-cm_1.9.10.0-1OEM.510.0.0.799733, Mellanox_bootbank_net-ib-core_1.9.10.0-1OEM.510.0.0.799733, Mellanox_bootbank_net-ib-mad_1.9.10.0-1OEM.510.0.0.799733, Mellanox_bootbank_net-ib-sa_1.9.10.0-1OEM.510.0.0.799733, Mellanox_bootbank_net-ib-umad_1.9.10.0-1OEM.510.0.0.799733, Mellanox_bootbank_net-mlx4-core_1.9.10.0-1OEM.510.0.0.799733, Mellanox_bootbank_net-mlx4-en_1.9.10.0-1OEM.510.0.0.799733, Mellanox_bootbank_net-mlx4-ib_1.9.10.0-1OEM.510.0.0.799733, Mellanox_bootbank_net-rdma-cm_1.9.10.0-1OEM.510.0.0.799733, Mellanox_bootbank_scsi-ib-iser_1.9.10.0-1OEM.510.0.0.799733
   VIBs Removed: VMware_bootbank_net-mlx4-core_1.9.7.0-1vmw.550.0.0.1331820, VMware_bootbank_net-mlx4-en_1.9.7.0-1vmw.550.0.0.1331820
   VIBs Skipped:

So it looks like they installed OK. I reboot the server, but the card is not showing up in the server/config/network adapter section. Is there any thing that needs to be done? Has anyone else got one of these cards to work with ESXi 5.5?

Running the following shows it is installed:

Code: 
~ # esxcli software vib list
Name                           Version                               Vendor    Acceptance Level  Install Date
-----------------------------  ------------------------------------  --------  ----------------  ------------
net-ib-addr                    1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
net-ib-cm                      1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
net-ib-core                    1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
net-ib-mad                     1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
net-ib-sa                      1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
net-ib-umad                    1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
net-mlx4-core                  1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
net-mlx4-en                    1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
net-mlx4-ib                    1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
net-rdma-cm                    1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
scsi-ib-iser                   1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
ata-pata-amd                   0.3.10-3vmw.550.0.0.1331820           VMware    VMwareCertified   2014-06-15  
ata-pata-atiixp                0.4.6-4vmw.550.0.0.1331820            VMware    VMwareCertified   2014-06-15  
ata-pata-cmd64x                0.2.5-3vmw.550.0.0.1331820            VMware    VMwareCertified   2014-06-15  
ata-pata-hpt3x2n               0.3.4-3vmw.550.0.0.1331820            VMware    VMwareCertified   2014-06-15  
ata-pata-pdc2027x              1.0-3vmw.550.0.0.1331820              VMware    VMwareCertified   2014-06-15
.....
 
Last edited:
402blownstroker said:
??? What do you mean? I am trying to setup the card to do IPoIB.
Click to expand...

The inbox drivers for it are for ethernet. You've replaced those with infiniband, which isn't inherently a networking protocol as much as it's a bus protocol that can happen to carry IP frames at times.

I don't know where to guide you from here - VMware technically doesn't support any form of IB.
 
lopoetve said:
The inbox drivers for it are for ethernet. You've replaced those with infiniband, which isn't inherently a networking protocol as much as it's a bus protocol that can happen to carry IP frames at times.

I don't know where to guide you from here - VMware technically doesn't support any form of IB.
Click to expand...

Are you just assuming he's got a 10Gbps ethernet card?

He HAS an IB card. And: http://kb.vmware.com/selfservice/mi...nguage=en_US&cmd=displayKC&externalId=2058261

Unfortunately I don't know how to solve his problem, though.
 
That OFED driver seems very Wrong!.

I use 1.8.4.2 and thats the latest that shows on mellanox's website for vSphere 5.5

http://www.mellanox.com/page/products_dyn?product_family=36&menu_section=34

I use IPoIB currently in my environment for vMotion, the question is what do you plan on using? do you have a Mellanox Switch that has openSM or do you plan on using Raphaels OpenSM driver from hypervisor.fr?

The OFED Driver i am providing provides drivers and the IPoIB Driver stuff for card you are trying to use. You will also need to make sure you are running a 2.7 Firmware its been the best for me.
 
dandragonrage said:
Are you just assuming he's got a 10Gbps ethernet card?

He HAS an IB card. And: http://kb.vmware.com/selfservice/mi...nguage=en_US&cmd=displayKC&externalId=2058261

Unfortunately I don't know how to solve his problem, though.
Click to expand...

Most of the cards go both ways, depending on driver installed :) They're fully supported using the inbox ethernet driver - his install copy/paste shows that it replaced the ethernet driver with the IB driver, so yes, I'm assuming he has one capable of being an ethernet card, since it was one before he made the change. :)

The PVSP program is ~not~ support - it's "someone made this work. Good on them, don't call us".
 
lopoetve said:
his install copy/paste shows that it replaced the ethernet driver with the IB driver
Click to expand...

While it may say: VIBs Removed: VMware_bootbank_net-mlx4-core_1.9.7.0-1vmw.550.0.0.1331820, VMware_bootbank_net-mlx4-en_1.9.7.0-1vmw.550.0.0.1331820

It also then says:
net-mlx4-core 1.9.10.0-1OEM.510.0.0.799733 Mellanox PartnerSupported 2014-07-18
net-mlx4-en 1.9.10.0-1OEM.510.0.0.799733 Mellanox PartnerSupported 2014-07-18
net-mlx4-ib 1.9.10.0-1OEM.510.0.0.799733 Mellanox PartnerSupported 2014-07-18
net-rdma-cm 1.9.10.0-1OEM.510.0.0.799733 Mellanox PartnerSupported 2014-07-18


So no, he didn't remove anything. It updated those components. Unless you're saying he replaced it with a version that somehow doesn't support what he's trying to do.
 
I am saying he updated the driver not the IB Stuff he wanted.. it should look like mine

~ # esxcli software vib list |grep -i Mellanox
net-ib-cm 1.8.2.4-1OEM.500.0.0.472560 Mellanox PartnerSupported 2014-06-27
net-ib-core 1.8.2.4-1OEM.500.0.0.472560 Mellanox PartnerSupported 2014-06-27
net-ib-ipoib 1.8.2.4-1OEM.500.0.0.472560 Mellanox PartnerSupported 2014-06-27
net-ib-mad 1.8.2.4-1OEM.500.0.0.472560 Mellanox PartnerSupported 2014-06-27
net-ib-sa 1.8.2.4-1OEM.500.0.0.472560 Mellanox PartnerSupported 2014-06-27
net-ib-umad 1.8.2.4-1OEM.500.0.0.472560 Mellanox PartnerSupported 2014-06-27
net-mlx4-core 1.8.2.4-1OEM.500.0.0.472560 Mellanox PartnerSupported 2014-06-27
net-mlx4-ib 1.8.2.4-1OEM.500.0.0.472560 Mellanox PartnerSupported 2014-06-27
scsi-ib-srp 1.8.2.4-1OEM.500.0.0.472560 Mellanox PartnerSupported 2014-06-27


Just install the Mellanox IB OFED Driver from the website i provided above.

This is the way to make it work Solid! especially with the older IB Cards hes got a CX4 Based DDR IB Card same as me. The OFED software on Mellanox's website contains a driver replacement, so what it will do is also rename the IB Adapters but give you Storage/Network Adapaters. From there its IPoIB that requires a subnet manager!
 
OK, I followed the steps at these two places, pretty much the same

http://www.bussink.ch/?p=1306

https://www.techidiots.net/notes/infiniband/setup-infiniband-in-esx

Everything installed fine not errors. When I got to the step of copying the partitions.conf to the /scratch/opensm directories, there where not any of the node and port directories there. So I booted into a Fedora LiveCD, installed the needed stuff( yum install infiniband-diags perftest qperf opensm ), and ran ibstat to get the node and port GUIs. Booted back into ESXi, created the corresponding node and port directories, and then copied the partitions.conf file into them. When I run ibstat on ESXi, nothing..... I even reboot after getting the partition directories populated and nothing.... Anyone have any ideas on what is happening here???!?!?!?
 
dasaint said:
That OFED driver seems very Wrong!.

I use 1.8.4.2 and thats the latest that shows on mellanox's website for vSphere 5.5

http://www.mellanox.com/page/products_dyn?product_family=36&menu_section=34

I use IPoIB currently in my environment for vMotion, the question is what do you plan on using? do you have a Mellanox Switch that has openSM or do you plan on using Raphaels OpenSM driver from hypervisor.fr?

The OFED Driver i am providing provides drivers and the IPoIB Driver stuff for card you are trying to use. You will also need to make sure you are running a 2.7 Firmware its been the best for me.
Click to expand...

I tried removing everything and installing the driver fro the above link and nothing is showing up in the network adapters section after reboot. The switch I have I do not think it has openSM.

Maybe I will just have to settle with using passthrough to a specific server rather than having the entire VM suite having access to it.
 
To make the gen 1 cards work (Ethernet) I had to disable passthrough in the BIOS. Maybe you could try that?
 
Check these Articles

Need to use Chrome to translate French to English
http://www.hypervisor.fr/?p=4662

Erik had some issues running vSAN with IPoIB
http://www.bussink.ch/?p=1410

I talked with Erik and Raphael many times about the idea of passing through the card they told me they had very NIL success on that. Just make sure your cards are also running the firmware and arent already passed through... if needed we could get on a webex or something and i can look around see if i can help ya out.

I would also run
lspci make sure its showing up in there.
Mine shows up like this
0000:06:00.0 Serial bus controller: Mellanox Technologies MT25418 [ConnectX VPI - 10GigE / IB DDR, PCIe 2.0 2.5GT/s] [vmhba_mlx4_0.2.1]

Shows the PCI port and the module being used.

in the network side i see the adapter as VMNIC_IB0 and VMNIC_IB1
the hba side i see vmhba_mlx4_0.2.1 - SCSI for both adapters.
 
dandragonrage said:
While it may say: VIBs Removed: VMware_bootbank_net-mlx4-core_1.9.7.0-1vmw.550.0.0.1331820, VMware_bootbank_net-mlx4-en_1.9.7.0-1vmw.550.0.0.1331820

It also then says:
net-mlx4-core 1.9.10.0-1OEM.510.0.0.799733 Mellanox PartnerSupported 2014-07-18
net-mlx4-en 1.9.10.0-1OEM.510.0.0.799733 Mellanox PartnerSupported 2014-07-18
net-mlx4-ib 1.9.10.0-1OEM.510.0.0.799733 Mellanox PartnerSupported 2014-07-18
net-rdma-cm 1.9.10.0-1OEM.510.0.0.799733 Mellanox PartnerSupported 2014-07-18


So no, he didn't remove anything. It updated those components. Unless you're saying he replaced it with a version that somehow doesn't support what he's trying to do.
Click to expand...

It supports what he's trying to do, but probably not with that precise package, at least not without additional pieces. The partnersupported ones are all the ones installed by that package (ESX doesn't ship with any, unless it's a custom iso or vendor-branded iso).
 
It looks like youve got the OEM drivers for ESX 5.1 not 5.5, i got my drivers from here: https://my.vmware.com/group/vmware/...DT-ESXI55-MELLANOX-MLX4_EN-1990&productId=353

402blownstroker said:
I am trying to install the Mellanox drivers on a ESXi 5.5 system and they are failing to install. I have a 483513-B21 card. The drivers I found are:

MLNX-OFED-ESX-1.9.10.0-10EM-510.0.0.799733.zip

When I run the following command to install them, I get the following:

Code: 
 esxcli software vib install -d /tmp/MLNX-OFED-ESX-1.9.10.0-10EM-510.0.0.799733.zip 
Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: Mellanox_bootbank_net-ib-addr_1.9.10.0-1OEM.510.0.0.799733, Mellanox_bootbank_net-ib-cm_1.9.10.0-1OEM.510.0.0.799733, Mellanox_bootbank_net-ib-core_1.9.10.0-1OEM.510.0.0.799733, Mellanox_bootbank_net-ib-mad_1.9.10.0-1OEM.510.0.0.799733, Mellanox_bootbank_net-ib-sa_1.9.10.0-1OEM.510.0.0.799733, Mellanox_bootbank_net-ib-umad_1.9.10.0-1OEM.510.0.0.799733, Mellanox_bootbank_net-mlx4-core_1.9.10.0-1OEM.510.0.0.799733, Mellanox_bootbank_net-mlx4-en_1.9.10.0-1OEM.510.0.0.799733, Mellanox_bootbank_net-mlx4-ib_1.9.10.0-1OEM.510.0.0.799733, Mellanox_bootbank_net-rdma-cm_1.9.10.0-1OEM.510.0.0.799733, Mellanox_bootbank_scsi-ib-iser_1.9.10.0-1OEM.510.0.0.799733
   VIBs Removed: VMware_bootbank_net-mlx4-core_1.9.7.0-1vmw.550.0.0.1331820, VMware_bootbank_net-mlx4-en_1.9.7.0-1vmw.550.0.0.1331820
   VIBs Skipped:

So it looks like they installed OK. I reboot the server, but the card is not showing up in the server/config/network adapter section. Is there any thing that needs to be done? Has anyone else got one of these cards to work with ESXi 5.5?

Running the following shows it is installed:

Code: 
~ # esxcli software vib list
Name                           Version                               Vendor    Acceptance Level  Install Date
-----------------------------  ------------------------------------  --------  ----------------  ------------
net-ib-addr                    1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
net-ib-cm                      1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
net-ib-core                    1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
net-ib-mad                     1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
net-ib-sa                      1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
net-ib-umad                    1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
net-mlx4-core                  1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
net-mlx4-en                    1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
net-mlx4-ib                    1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
net-rdma-cm                    1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
scsi-ib-iser                   1.9.10.0-1OEM.510.0.0.799733          Mellanox  PartnerSupported  2014-07-18  
ata-pata-amd                   0.3.10-3vmw.550.0.0.1331820           VMware    VMwareCertified   2014-06-15  
ata-pata-atiixp                0.4.6-4vmw.550.0.0.1331820            VMware    VMwareCertified   2014-06-15  
ata-pata-cmd64x                0.2.5-3vmw.550.0.0.1331820            VMware    VMwareCertified   2014-06-15  
ata-pata-hpt3x2n               0.3.4-3vmw.550.0.0.1331820            VMware    VMwareCertified   2014-06-15  
ata-pata-pdc2027x              1.0-3vmw.550.0.0.1331820              VMware    VMwareCertified   2014-06-15
.....
Click to expand...
 
moto316 said:
It looks like youve got the OEM drivers for ESX 5.1 not 5.5, i got my drivers from here: https://my.vmware.com/group/vmware/...DT-ESXI55-MELLANOX-MLX4_EN-1990&productId=353
Click to expand...

OK, I got a chance to play around a little more and installing that driver produces the same result. It shows up in lspci and I can make it as pass through. For some reason it does not show up in the network adapters section. Other than loading the driver, is there anything else that needs to be done? Prior to installing this version, I removed the other version I installed first.

Code: 
0000:02:00.0 Serial bus controller: Mellanox Technologies MT25208 [InfiniHost III Ex]
 
402blownstroker said:
OK, I got a chance to play around a little more and installing that driver produces the same result. It shows up in lspci and I can make it as pass through. For some reason it does not show up in the network adapters section. Other than loading the driver, is there anything else that needs to be done? Prior to installing this version, I removed the other version I installed first.

Code: 
0000:02:00.0 Serial bus controller: Mellanox Technologies MT25208 [InfiniHost III Ex]
Click to expand...

Ah you've got an infinihost card. I'm not positive if those will work as net adapters in esxi, my experience has been with the connectx-2 line of cards.
 
To 402BlownStroker - were you able to get the Infinihost III adapter to work? I have the same, although mine is LX not EX. ESX see's the device, I was able to install the driver, the opensm and the OFED software, but it doesn't appear as a network / storage device, ect. I do note that both you and I have the same output from lspci - the device is listed, but the associated module is not.

Any input from your experience and from anyone else would be greatly appreciated!
 
Nope. I ended up passing it through to the one important guest that needed it. Would have been nice if all the guests could use it, but not the case.
 
Thanks for the update....I ended up getting two connect-X cards, and all was well. Should have paid the little extra up front, I spend all day trying to get the infinihost cards working! Plugged the connect-X in, whala, worked like a champ.

Thanks again, and have a good one!
 
You must log in or register to reply here.
Back
Top