• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

ESX 4 & Infiniband

M

mike938

Weaksauce
Joined
Jan 22, 2007
Messages
97
Does anyone know if there is a way to get Infiniband working with ESX 4? There are drivers for mellanox cards for ESX 3.5 but nothing yet for 4 as far as I can see.
 
I don't have my hardware yet and was trying to avoid purchasing the cards if it wasn't going to work. The drivers installed fine in a VM running ESX (via Workstation) but I don't believe it actually loaded them at all since the card wasn't present. If nobody knows for sure I'm going to buy 1 card and try it on my desktop before the server hardware arrives to find out.
 
Just wondering which cards you're picking up and what your plans are for them? PCIe/PCIX? Do you do native IB on them, or planning on IP over IB?

I've been using the OpenFabrics stack and drivers here for IPoverIB, seeing 160-180MB/s sustained throughput backing up one server to the other. :)
 
I'm going to pick up the Mellanox InfiniHost III ex PCI-E dual 10Gb cards and i'm unsure if I'm going to do IP over IB or what as I've never used the OpenFabrics stack before so I'm unsure how it can be configured.
 
IP over IB doesn't use RDMA, which is why its so slow. The latency and bandwidth is not nearly as slow as gigabit, but its not a huge leap up either. There are some things that can use RDMA, but I believe they're mostly limited to the linux side of things. Native IB traffic can do RDMA, the linux drivers can do RDMA with their iSCSI over IB stack (iSER), FC over IB is possible, but I believe you need a switch/gateway to do it, not sure if you can do it all in software.

https://wiki.openfabrics.org/tiki-index.php?page=iSER

Another thing to note is on the dual 10Gb cards, the second ports are for failover, not increased bandwidth. I tried like hell to bring up both ports at the same time, it wouldn't work with direct connection, or a second switch.
 
When you say the second port is for failover, do you mean if they are connecting the same 2 hosts? I was planning on having a storage server (on linux if necessary and it will support easy, direct exporting of hardware raid volumes which openfiler did not like to do via iSCSI if arrays are expanded) with 1 dual port card and then 2 ESX hosts with dual port cards with 1 port on each going to the storage server and then if possible the 2nd ports on each to each other for vmotion traffic via IPoIB.
 
Ok that is something I haven't tried, but I have connected both ports to the same fabric and wasn't able to get both ports activated at the same time. I also attempted to connect both ports on both HCAs together directly and was only able to bring up the first connection. I guess its something you'll just have to attempt.

The ESX IB documentation I've read says it supports the OFED stack, but I can't find a definite list of the protocols for it.

You'll also need to run OpenSM (Subnet Manager) on one of the machines in order to bring the fabric up.
 
You must log in or register to reply here.
Back
Top