Fibre Channel Question

S

Skud

Gawd
Joined
Sep 4, 2002
Messages
587
So, I have a ZFS storage server and will be using it to serve a hyper-v failover cluster. I originally had thought I was going to do 1Gb iSCSI, but I think I'm going to want something faster.

10Gb is too expensive.
1Gb iSCSI is going to be okay, but I'd like better
2 x 1Gb iSCSI either via LACP or MPIO is better, but still has downfalls in that you only get the speed one-way.

I'm looking at some cheap 4Gb fibre channel HBAs, but NOT using a SAN switch. I see you can buy "crossover" fibre cables and I assume this is for linking two HBAs together like you would two regular ethernet NICs. As far as I can tell, this would be the same as a FC-AL implementation, but with only two nodes. Correct?

I'd like a dual port 4Gb HBA in the storage server and a single port in each hyper-v cluster node connecting back to each port in the storage server.

Thoughts?

Thanks!!
Riley
 
I'm going to answer my own question....

It looks like there are three FC topologies:

Fibre Channel Arbitrated Loop (FC-AL)
Fibre Channel Switched Fabric
Fibre channel Point to Point

FC-Al is like token ring where the transmit from one HBA goes into the receive of another. This can go on for up to 127 devices ending back into the receive port of the first HBA. This forms a ring. Only one device talks at a time and devices joining/leaving the loop cause a "LIP" (loop initialization primitive) that temporarily takes down the ring.

FC Switched Fabric is just like a hub and spoke Ethernet implementation with a switch in the middle

FC Point to Point is what I want.. It's just simply connecting two FC devices together by reversing the send and receive pair on one HBA.

So, it looks like it's possible. I'm going to find some cheap(ish) 4gb FC HBAs and see if I can get it working...

Riley
 
This sounds really interesting if it can be done relatively inexpensively as 10gigE is just too expensive now
 
If you find some affordable 4Gb HBA's would you mind posting them here? I'm also interested in a similar setup...
 
If you are going to do away with the SAN switch and go Point-to-Point anyway why not just use 10Gbe? Its faster than fiber channel, it uses native TCP/IP that you probably already understand, all of the "built in" interfaces to you SAN already understand it (Samba, NFS, iSCSI), you'll have a broader group of people who know enough to support you, and if you don't need a switch its WAY less expensive. Its only the switch that makes 10Gbe too expensive...

You can either use 10Gbase-T NICs (single-port Intel are ~$400 new, ~$150 used) connected using Cat6 or dual-port SFP+ based NICs like the X520-DA2 or SuperMicro AOC-STGN-i2S. For the SFP+ based NICs, connect them back-to-back using a "direct attach copper" cable, which is basically twinax cable with passive SFP+ connectors on both ends. Make up a new subnet and assign static IP addresses in that subnet to both ends. In most cases, that's all it takes (though you may need to do some parameter tuning to get the performance you want).
 
PigLover said:
If you are going to go Point-to-Point anyway why not just use 10Gbe? Its faster than fiber channel, it uses native TCP/IP that you probably already understand, all of the "build in" interfaces to you SAN already understand it (Samba, NFS, iSCSI), you'll have a broader group of people who know enough to support you, and if you don't need a switch its WAY less expensive. Its only the switch that makes 10Gbe expensive...

You can either use 10Gbase-T NICs (single-port Intel are ~$400 new, ~$150 used) connected using Cat6 or dual-port SFP+ based NICs like the X520-DA2 or SuperMicro AOC-STGN-i2S. For the SFP+ based NICs, connect them back-to-back using a "direct attach copper" cable, which is basically twinax cable with passive SFP+ connectors on both ends.
Click to expand...

A 10Gb NIC for $150? I want one! If I can get a $150 10Gb NIC I'll go with 10Gb Ethernet in a heartbeat..

A brief perusal of Ebay shows...

10Gb dual port NICs are ~$750 each and that's for CX4 units - which would be okay, but CX4 is a bit "off" standard now.. Single port cards are ~$350..

Dual port 4Gb PCIe FC cards are around $300 and single port cards are about half that.

Riley
 
PigLover said:
If you are going to do away with the SAN switch and go Point-to-Point anyway why not just use 10Gbe? Its faster than fiber channel, it uses native TCP/IP that you probably already understand, all of the "built in" interfaces to you SAN already understand it (Samba, NFS, iSCSI), you'll have a broader group of people who know enough to support you, and if you don't need a switch its WAY less expensive. Its only the switch that makes 10Gbe too expensive...

You can either use 10Gbase-T NICs (single-port Intel are ~$400 new, ~$150 used) connected using Cat6 or dual-port SFP+ based NICs like the X520-DA2 or SuperMicro AOC-STGN-i2S. For the SFP+ based NICs, connect them back-to-back using a "direct attach copper" cable, which is basically twinax cable with passive SFP+ connectors on both ends. Make up a new subnet and assign static IP addresses in that subnet to both ends. In most cases, that's all it takes (though you may need to do some parameter tuning to get the performance you want).
Click to expand...

I can't seem to find $150 10gigE cards either, any links?
 
Skud said:
A 10Gb NIC for $150? I want one! If I can get a $150 10Gb NIC I'll go with 10Gb Ethernet in a heartbeat..

A brief perusal of Ebay shows...

10Gb dual port NICs are ~$750 each and that's for CX4 units - which would be okay, but CX4 is a bit "off" standard now.. Single port cards are ~$350..

Dual port 4Gb PCIe FC cards are around $300 and single port cards are about half that.

Riley
Click to expand...

Shop victoriously...within the last 6 weeks I scored two 10Gbase-T NICs for $125/each...Look often. The deals are out there. They don't last.
 
i know this might sound silly but what about inifiniband, could that be used as well?
 
how much overhead? aren't there cheap 8gb infiniband cards? I've been wanting to do 10gbe also. But Switches hurt my head in terms of cost. Though I might pare down to two hosts this summer so I'd need four 10gbe nics. One for each host, and two in the SAN.

Now I'm also thinking about virtualizing my SAN and doing VT-d passthrough of the SAS controller. This would allow me to drop down to only two 10gbe nics... which makes things more soft effective. Just means that if/when I utilize the third ESXi host it'd have to run over 1gbe copper.
 
Last edited:
Garuda888 said:
how much overhead? aren't there cheap 8gb infiniband cards? I've been wanting to do 10gbe also. But Switches hurt my head in terms of cost. Though I might pare down to two hosts this summer so I'd need four 10gbe nics. One for each host, and two in the SAN.

Now I'm also thinking about virtualizing my SAN and doing VT-d passthrough of the SAS controller. This would allow me to drop down to only two 10gbe nics... which makes things more soft effective. Just means that if/when I utilize the third ESXi host it'd have to run over 1gbe copper.
Click to expand...

Yes, there are cheap infiniband cards. Problem is, they are cheap because nobody wants them...and nobody wants them because they lack the TCP Offload Engine (TOE) that is found on their more expensive brothers and almost all modern Ethernet NICS (both 1G and 10G).

For cheap(ish) 10Gbe, search "XR997" on ebay. Right now they are +/- $199, through from time to time they drop as low as $150 buy-it-now. Like I said above, I recently scored two @ $125 on an ebay "make offer" purchase. This is probably the best solution for the OPs problem. One note: this card has a screaming little fan on it. Not suitable for living spaces.
 
Last edited:
Are there any lightpeak/thunderbolt networking options that might work?
 
I tried the infiniband stuff - wasn't able to get much over 200-300MByte/sec with them (using NFS/SMB) - need a TOE engine. the newer mellanox connectX has offloading, but it costs as much as GigE stuff.
(And the 200-300 was with ~100 processor usage on a i7-920 client <-> x3440 server)

Going to the dell xr997 and making no other changes I was able to get 500-700MByte/sec (write/read) real world file transfer speed (not iPerf) (was to a local 16Gb ram drive - local disk wasn't fast enough to sustain that kind of speed)
 
Hi,

Switches are definitely pricey; however, if you just want 10GbE from Point to Point (e.g., crossover), the Intel AT2 10GbE cards are ~$500 and a cat 6 cable is $10.

Speeds are good between two fast systems, 500-800MiB/s.

<--- 150 9308814.8 kbytes to download
`...myfile_backup.bkf' at 2246574080 (23%) 590.73M/s eta:12s [Receivi
`...myfile_backup.bkf' at 3256772828 (34%) 592.81M/s eta:10s [Receivi
`...myfile_backup.bkf' at 4469835240 (46%) 625.35M/s eta:8s [Receivin
`...myfile_backup.bkf' at 5704774828 (59%) 647.48M/s eta:6s [Receivin
`...myfile_backup.bkf' at 6906118144 (72%) 658.93M/s eta:4s [Receivin
`...myfile_backup.bkf' at 7968391168 (83%) 655.25M/s eta:2s [Receivin
<--- 226-File successfully transferred
<--- 226 13.462 seconds (measured here), 675.30 Mbytes per second
---- Got EOF on data connection
---- Closing data socket
9532226323 bytes transferred in 13 seconds (674.93M/s)
 
Last edited:
Alight i'm gonna be the retard and ask. Even though i see its recommended against infiniband i'm thinking about trying it out anyways. However with those cards since there is 2 ports on them, would it be possible or even such a thing exists to convert from that 10gb infiniband to standard 1gb ethernet via like SFP? not QSFP or SFP+ just SFP? reason is the server i want to try it out on only has 1 PCie slot available and i need another port regardless and was thinking i could use one port to connect to another server and use the other to connect into my main data network.
 
moose517 said:
Alight i'm gonna be the retard and ask. Even though i see its recommended against infiniband i'm thinking about trying it out anyways. However with those cards since there is 2 ports on them, would it be possible or even such a thing exists to convert from that 10gb infiniband to standard 1gb ethernet via like SFP? not QSFP or SFP+ just SFP? reason is the server i want to try it out on only has 1 PCie slot available and i need another port regardless and was thinking i could use one port to connect to another server and use the other to connect into my main data network.
Click to expand...

I'm not an expert but I'm pretty sure this could be done with *nix, its basically like what a router does. I'm not sure how but it can probably be done.

I think the problem with the infiniband is high CPU usage due to lack of TCP offload engine so if you don't have high end hardware might not be worth it.
 
would be going into a pair of servers that use dual xeons so CPU won't be a problem.
 
You cannot convert infiniband to ethernet. You could bridge connections between an ethernet HBA and it, but you would be obviously limited to encapsulated ethernet traffic and not actual infiniband. The transceivers in your example only convert the medium used to transmit data and do not change the payload at all and are therefore possible.
 
You must log in or register to reply here.
Back
Top