• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Infiniband Network Problem/Question

S

senseloss

n00b
Joined
Mar 2, 2013
Messages
4
Excited by the prospect of a cheap >10Gbit home network I purchased a boat load of infiniband hardware on ebay. I purchased the following

  • Qlogic Silverstorm 9024 (300$)
  • 5x Mellanox MHGH28-XTC Connect-X VPI DDR 2 Port cards (75$/ea)
  • 5x SFF-8087 8M cables (DDR rated - 25$/ea)

5 infiniband connected computers @ 20Gbit for just 800$!

I want to have my wired lan all be on infiniband using IPoIB/iSCSI/RDMA/etc while still being able to access devices on my ethernet/wireless networks via the router.

My Problems:

#1) My Qlogic Silverstorm 9024 is detecting my Mellanox cards as being 4x SDR instead of 4x DDR. I've tried various driver versions, no matter what I do, the link is negotiated as SDR. When I try to force DDR in the switch it says the card is not capable of DDR and downgrades to a SDR connection.

-> ismChassisSetSpeed
Cable 1 link speed is fixed 2.5 (SDR only device)
Cable 2 link speed is fixed 2.5 (SDR only device)

Does this mean that the cable is at fault? I'm wondering if I got screwed and sold a SDR cable listed as DDR.

#2) I need to figure out some way to easily bridge my ethernet adapter to my infiniband network. So when I put my fileserver online on the IB network wireless/wired ethernet clients can contact it (or vice versa, IB clients may need to contact wired/wireless). I incorrectly assumed this would be easy. I setup a bridge added my lan port to the bridge then went to add the IB interface and it failed. Turns out that brctl doesn't support IB devices. I read online about a new 'eipoib' driver that will support such functionality, but has not been released yet. Does anyone have any ideas how I might bridge 2 physical adapters on the same subnet (10.0.0.x/24) without using linux bridging? Keep in mind that the router will also be providing WAN/NET access so it will have a third adapter to supply internet connectivity to the entire network.

My current bridging fall back plan involves using iptables to default forward any ib0 traffic to tap0 (which will be a br0 member). But it's messy, and I'm not even sure it will work like I want.

Thank you in advance for any help


...


For anyone who wants to know for curiousity:

I'm making an all-in-one nas/router solution. The router will be providing internet connectivity to the wireless/wired/IB networks, routing data between the IB and wired/wireless networks, and providing NAS services. The NAS will be a 5 drive 3TB raid 5 running on an Adaptec 5805. I'm not really planning on doing anything too fancy with it... Just using it to learn the tech and to get > 1Gbit connectivity to the NAS from wired machines.
 
Last edited:
And you are sure you are using
9024-FC24-ST1-DDR or
9024-CU24-ST2-DDR

SKU´s? (Should be on a sticker on the top side of the IB-switch)
 
allban said:
And you are sure you are using
9024-FC24-ST1-DDR or
9024-CU24-ST2-DDR

SKU´s? (Should be on a sticker on the top side of the IB-switch)
Click to expand...


PtNumEncLen: cd PartNum: 200708-000-12
ModelEncLen: cd ModelNum: 9024-CU24-ST2
VersionEncLen:c5 Version: 1.0.0
MfgEncLen: d6 Manufacturer: InfiniCon Systems Inc.
ProdNmEncLen: d3 ProductName: InfinIO 9024 Switch

I haven't updated the firmware from the one on the system. I'm concerned that updating it could introduce new licensing measures that will lock me out of the unit. Do you know if they ever changed the licensing mechanisms in the switch firmwares? (Am I safe to update?)

Firmware Version: 4.1.1.1.11


Regarding my #2 problem it seems OFED version 2.0 is due out which contains eipoib that will allow me to create a bridgeable interface. I'll just wait for that to come out.
 
Last edited:
As an eBay Associate, HardForum may earn from qualifying purchases.
As your cards are dual-port, maybe you can get some more cables and hook the second port up and add those to the links? I don't actually use IB myself nor know very much about it, but I wouldn't have thought there would be much difference between bonding 4 channels over 1 link and bonding 8 channels over 2 links. That should give you 20Gbps...
 
bootc said:
As your cards are dual-port, maybe you can get some more cables and hook the second port up and add those to the links? I don't actually use IB myself nor know very much about it, but I wouldn't have thought there would be much difference between bonding 4 channels over 1 link and bonding 8 channels over 2 links. That should give you 20Gbps...
Click to expand...

I was going to expand to more than 5 boxes eventually. I wanted to experiment with RDMA and the new GPUDirect thing (if it's a driver level function and doesnt require a newer IB card).

...

Speed results if anyone is curious:

** Datagram mode w/ 2044 MTU, TXQUEUELEN 10000 and default Centos 6.3 memory/buffer settings

MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.15 () port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 65536 65536 10.00 7661.14

** Connected mode w/ 65520 MTU, TXQUEUELEN 10000 and default Centos 6.3 memory/buffer settings

MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.15 () port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 65536 65536 10.00 7859.30

Reading from SSD to 1x HDD I'm getting the following results:

Getting 900Mbyte/s from apache memory to system memory once memory is full it goes down to 50Mbyte/s for the rest of the transfer, 175Mbyte/s average on a 3.5G file using FTP, and about 50Mbyte/s over Samba.

Then I setup a ramcache and ran the same tests.

FTP ramdisk -> ramdisk
3366977536 bytes received in 3.43 secs (981857.98 Kbytes/sec)

Samba 3500Mbyte transfer ramdisk -> ramdisk 66Mbyte/s
real 0m53.175s
user 0m0.097s
sys 0m15.813s

Apache Ramdisk -> Ramdisk

100%[============================================================================================================>] 3,366,977,536 935M/s in 3.4s

...

After making the following changes to sysctl, updating to samba version 4 I'm now hitting samba performance equaling FTP, Apache and the rest (900+Mbyte/s ramdisk to ramdisk).

net.core.wmem_max = 64582912
net.core.rmem_max = 64582912
net.ipv4.tcp_rmem = 128400 1283800 64582912
net.ipv4.tcp_wmem = 128400 1283800 64582912
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_sack = 1
net.core.netdev_max_backlog = 50000
 
Last edited:
You must log in or register to reply here.
Back
Top