Infiniband connection between 2 windows servers

H

Hammer!

Limp Gawd
Joined
Nov 25, 2011
Messages
155
Hi, I would like to use infiniband to connect a standalone windows box to an esxi 5.0 host. The goal is to have a super fast connection between the two so that the esxi host can have fast access to the raid array in the standalone windows box.

Can someone recommend the best way to accomplish this? Ideally, I would like to use hardware pass through so that one of the vm guest in the esxi host can have direct access to the infiniband card as only that be guest needs access. However, I am open to ideas, especially since I'm not sure I can actually pass through my card to the vm guest; I'm currently running into issues and after some googling, find that it might not be possible to do.

I have two Mellanox connectx-2 cards and a cable - I will not be using a switch. Since, I won't have a switch, I'll still be using Ethernet to connect the esxi host to the rest of my network. The infiniband link is simple to have a fast connection to access the raid array on the standalone windows box. How do I setup the ip addresses of the machines so they can use both the Ethernet nics and infiniband at the same time?

Thanks in advance!
 
Simple....

Infiniband will have an IP address and a subnet... i.e. 172.16.1.1 / 255.255.255.0 and .2 respectively.

Gig nics will have a separate network to run on that the rest of your business runs on so if all the desktops and printers etc... are .. 192.168.1.1-254 / 24 bit subnet then use an IP address from that pool and specify a gateway for each server to your router or whatever.

so in summary...

Server1
InfBand --172.16.1.1 /24 or /30 since there are only two hosts on this network via direct connect cable. Do not use a gateway address.
Gigabit NIC - 192.168.1.1 /24 use a gateway that everyone else uses if you want.

Server2
InfBand --172.16.1.2 /24 or 30 do not use a gateway
Gigabit NIC -- 192.168.1.2/24 use a gateway that everyone else uses if you want.

Of course you shared NOTHING about your network topography, number of hosts, switching setup i.e. all layer 2 only or layer 3 switching, etc...

It can be very simple and there can be other ways that are more complex but will serve you better but it all depends on how much information you are willing to share to quantify the proper answers for your scenario.
 
Thanks. I network is pretty simple. I have the two servers I mentioned which are connected to an ethernet switch and other windows clients are also connected to that same switch. My network addresses are all 192.168.1.x

So, I should leave the existing nic ip addresses unchanged and for the two new infband cards, set IP address to 172.16.1.1 and 172.16.1.2? What should the subnet mask be? 255.255.255.0? And leave the gateway address blank?

What is /24 or /30?
 
Also, if there are two "links" connecting the two servers (nic via the ethernet switch and infband between the two), how do I make sure the data is passed using the fast infband connection?
 
Hammer! said:
Also, if there are two "links" connecting the two servers (nic via the ethernet switch and infband between the two), how do I make sure the data is passed using the fast infband connection?
Click to expand...

Well that depends on how you set up your maps and your software...

i.e. If you are going to use a CIFS (Windows Share) then you need to make sure that on server 2 you map the drives correctly.

Dont map anything to the network of the gig lan, rather make sure you are mapping drives that are on the infiniband network.

This is one of the simplest things to do but hardest to explain on a forum maybe someone else can expalin it better.
 
Either use the server's IPoIB IP when mapping or browsing the network share or put a entry in the client's hosts file that gives a name to the server's IPoIB IP.
 
Also, looking at the OP again, you can try passing the IB card through, or add it as a interface on a vSwitch and make it a new network segment.
 
After some trying, I don't think these card are hardware "passible" to a vm guest and so my only option is to add it to a vSwitch. Do I create a new vSwitch with just the infiniband card, or do I add it to my existing vSwitch with the existing ethernet NIC's?

How should I setup the IP address? My existing network is 192.168.1.x and the mask I use on all the PC;s in the network is 255.255.255.0. Thanks, Hammer!
 
As an update, I've setup the ip address on the infiniband card on the non-ESXi server to be 192.168.1.60. Both ESXi and Windows tells me their respective cards are detecting a link, but when I try to ping 192.168.1.60 from one of the VM's, I get Destination Host is unreachable.

Advice?
 
I suggest you follow tangoseal's example and do not assign the IBs IP addresses in the same 192.1681.X network which is the same as your ethernet network.

In this case, both your server's ethernet and IB NICs are on the same network. Windows is guessing that the .60 address is on the ethernet network and then errors out when it can't find it. Change both of your IB addresses to be something other than 192.168.1.X and then try pinging the new IP.
 
Hammer! said:
As an update, I've setup the ip address on the infiniband card on the non-ESXi server to be 192.168.1.60. Both ESXi and Windows tells me their respective cards are detecting a link, but when I try to ping 192.168.1.60 from one of the VM's, I get Destination Host is unreachable.

Advice?
Click to expand...


My guess would be your subnet mask isn't correct. If your Ethernet machines are using 192.168.1.x - 60 I'm assuming it's a /26 which means the subnet mask would need to be 255.255.255.192

I would also agree with putting the IB link on a separate network. Maybe Server A: 192.168.2.1 and Server B: 192.168.2.2 with a subnet mask of 255.255.255.252 on both cards. I'm not very familiar with the other ESXi stuff.
 
You definitely need them on a different subnet if you are going to have Ethernet and IB. I had a similar experience with passthrough not working, but I think it was before the update that fixed PSOD.

Try this:
IPoIB on Windows w/ IP: 192.168.2.2 255.255.255.0 (/24)

CX-2 added as uplink on new vSwitch.

Add vmxnet3 NIC on VMs w/ IP: 192.168.2.3 255.255.255.0 (/24)

Share something on VM and connect to it with \\192.168.2.3 from the Windows machine.
 
Last edited:
Thanks to all. If I used 192.168.2.1 and 192.168.2.2 on each server, respectively, what should my subnet mask be?
 
I think I need to rephrase my question. My main network is 192.168.1.1 and currently, I use 255.255.255.0 for all these devices.

Now that I am trying to add 192.168.2.1 and 192.168.2.2, what should my mask be for these two servers and do i need to change the mask on my other 192.168.1.x devices?

Thank you.
 
Hammer! said:
I think I need to rephrase my question. My main network is 192.168.1.1 and currently, I use 255.255.255.0 for all these devices.

Now that I am trying to add 192.168.2.1 and 192.168.2.2, what should my mask be for these two servers and do i need to change the mask on my other 192.168.1.x devices?

Thank you.
Click to expand...



The Infiniband links on each server should have a subnet mask of 255.255.255.252.


The GigE links can be 255.255.255.0
 
I just tried that, but the ping still does not work. Are there any other settings I need to be aware of?

Both the Esxi host and the Server tell me that there is a link so I think the hardware part is working...I've also installed OpenSM and that is running; although just on the standalone server and not on the ESXi host.

Thanks.
 
Also, in the ESXi host, under vSwitch Properties, the infinband card is reporting 192.168.2.2 in Observed IP ranges. This is the address I've given to the standalone server.

And in the vm guest that I would like to be able to access 192.168.2.2, I have 2 vNIC's, one with IP of 192.168.1.32 with Mask 255.255.255.0 and another vNIC with IP 192.168.2.1 with mask 255.255.255.252 as suggested. Do I need to set a Default Gateway? Or any DNS or WINS settings? Under the Infinband adapter settings, I have checked:

1) Client for MS Networks
2) QoS Packet Scheduler
3) File and Printer Sharing for MS Networks
4) Link-Layer Topology Discovery Mapper I/O Driver
5) Link-Layer Topology Discovery Responder
6) IP Version 6
7) IP Version 4

Appreciate all your help!
 
OpenSM needs to be on a single node only. Since the VM does not see the physical IB hardware, it will need to be the Windows box.

The IPoIB interface does not need a default gateway, dns, or wins. Just IP and subnetmask

Don't change your GbE interface settings.

As for trroubleshooting the connection first thing I would do is run vstat on the windows box to see if the connection came up alright; should say ACTIVE. I have never run OpenSM on a windows machine, but from my experience the SM not being configured correctly would be my first guess. On Linux, I would then run ibhosts and ibping, but I dont know the command or if it exists on windows. Section 1.5 of this technet article gives some more information on making sure opensm is running as a service.
 
OpenSM appears to be running properly.

I think the issue is that on both machines, the infiniband NIC's are appearing under "Unidentified Network"/Public Network. Could that be the problem?
 
Investigating further, I created a new guest vm and added a vNIC and set ip address to 192.168.2.3 and from this vm, I can ping 192.168.2.1 which is the vm guest I was working with above. However, I cannot ping the server connected externally via the infiniband card. So, does that imply my infiniband connection is not working? But as I mentioned, both on ESXi and on the connected server, the Infiniband NIC tells me it sees the connection and is running at 40Gbps on ESXi and 32Gbps on the standalone server.
 
Success! Now I can ping after setting up a new vm group in the vSwitch. I'm not sure why, but it now works. It could also be because I updated the firmware on the cards. I can't tell which since I did both at the same time. Thanks to everyone for their help!
 
what cards did you use? I was thinking of doing this between two servers for similar reasons, seems like you can get CX4 10gb IB cards for dirt cheap on ebay

will they all pass IP traffic?
 
You must log in or register to reply here.
Back
Top