NAS connected to desktop using Ethernet crossover with PCIe 10 Gbps?

Discussion in 'SSDs & Data Storage' started by SomeGuy133, Sep 17, 2015.

  1. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    I wanted to know if it is possible to connect my desktop (Win 7 Pro) to my NAS running with Server 2008 R2 Datacenter using 2 10Gb E cards using crossover. I would still want to use the motherboards Gb E to connect to the main network. Is it possible to have two networks connected at the same time?

    I really want to get a connection faster then 1 Gb E to my main computer from the NAS. As far as I know I can't use usb 3.0 because no bridges exist for that yet like they do for USB 2.0.

    There are fairly cheap PCIe cards for 10 Gb E but no decent priced routers. I can get a 10 Gb E card for 240 and even if it isn't the best card it has to be way faster then Gb E. I could get away with 2 cards making total cost around 500 bucks verse 1000-2000 with a router. Is that possible to have two separate networks?

    Is there a better way I don't know about?


    http://ccm.net/faq/6340-connecting-two-pcs-using-a-crossover-ethernet-cable

    http://www.newegg.com/Product/Product.aspx?Item=N82E16833114122
     
  2. Blue Fox

    Blue Fox [H]ardForum Junkie

    Messages:
    11,685
    Joined:
    Jun 9, 2004
    Yes, easily done.
     
  3. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    The NAS would show up on two networks on the desktop. How do I tell it to only use the crossover 10 Gbps connection?

    Also is there a more cost effective easier method for higher bandwidth then Gb E or is this the best solution?

    I am only really concerned with the one computer. As far as I know eSATA/USB 3.0 are not an option unless i am missing something.
     
  4. Blue Fox

    Blue Fox [H]ardForum Junkie

    Messages:
    11,685
    Joined:
    Jun 9, 2004
    You'd specify the address for your shares to be the one corresponding to the appropriate network. There are cheaper ways, however they're more complicated and probably not what you want.
     
  5. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    could you elaborate the cheaper ways. I would be interested to at least hear them.
     
  6. Blue Fox

    Blue Fox [H]ardForum Junkie

    Messages:
    11,685
    Joined:
    Jun 9, 2004
    FC, InfiniBand, bonded ethernet, etc.
     
  7. rsq

    rsq Limp Gawd

    Messages:
    246
    Joined:
    Jan 11, 2010
    Beware: Bonded Ethernet will not help single stream transfers.

    My advice: go for 10gig ethernet, but stick to SFP+ cards. Either go direct attach cable, or OM3 fibre.
     
  8. halfelite

    halfelite n00b

    Messages:
    31
    Joined:
    Feb 17, 2009
    You are going way to expensive already.

    Buy two mellanox cards like this http://www.ebay.com/itm/Lot-of-2-Me...-MNPA19-XTR-/301689994552?hash=item463e1ffd38

    then a single sfp cable. Can find on ebay for about 15 dollars.

    Total cost 75$ This is what I did to start to transfer between two of my NAS units it ran perfectly.

    I then went ahead and bought a CRS226-24G-2S+IN microtik switch can be found for around 200$ it has two SFP+ ports so no need to direct connect between boxes anymore
     
  9. cbf123

    cbf123 n00b

    Messages:
    50
    Joined:
    Dec 12, 2013
    Not entirely true. You can do round-robin where packets alternate between the bonded links. In the general case this can be problematic (especially for net-booting) but in this case it should work okay.
     
  10. zaniix

    zaniix Gawd

    Messages:
    895
    Joined:
    Nov 4, 2011
    You are nitpicking, this is not a good solution for a NAS hookup for a single user.


    Hmm makes me wonder if there is a way to use Thunderbolt in this scenario.
     
  11. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    infiniband looks expensive compare dto 10 Gb E at least from the things that i have found. Bonded ethernet is a band-aid. And what is FC?

    @halfelite That router according to a review at amazon only does 25 Mbps or is that the Gb E and not the SFP+? Can you elaborate your set up and explain the performance?
     
  12. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
  13. halfelite

    halfelite n00b

    Messages:
    31
    Joined:
    Feb 17, 2009
    Dont confuse the switching capabilities with the routing capabilities.. Routing capabilities are a slow performer. Switching capabilities are fine. But there are even cheaper basic switches. The microtik is a big step up in SOHO from what you can usually get for 200$ If you use it as a basic switch without enabling to many L3 options you can get 9gigabit/s between to the two sfp ports.

    And yes a 3 meter cable should be fine i run one with mine. It was the IBM 3m Passive 10Gb SFP+ Direct Attach Copper DAC Cable 90Y9428 - 90Y9430
     
  14. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    what are these L3 options? :/ I dont do a lot of advance networking. Most I do is port forwarding and blocking and stuff like that. Just a humble geek in and over his head in a new area.
     
  15. ToddW2

    ToddW2 2[H]4U

    Messages:
    4,019
    Joined:
    Nov 8, 2004
    You can go a lot further than 1M if you don't use a DAC, but it raises cost slightly.
     
  16. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    DAC is direct attached copper right?
     
  17. -Dragon-

    -Dragon- 2[H]4U

    Messages:
    2,318
    Joined:
    Apr 6, 2007
    Just FYI you don't need a crossover cable since gigabit ethernet and above all support auto MDI-X (auto crossover). I did just what you're talking about a while back using the Intel X540-T2 (dual port) cards (before I got the 8 port netgear 10Gbit switch) and it works amazing. Don't know anything about the card you linked and I realize there are cheaper options out there but the X540 cards I know are great, used iperf bidirectional and was able to sustain 20Gbit bi-directional (40 total) for as long as I cared to (it actually pushed my 4.4GHz i7-2600k to like 80% CPU usage from just the network transfers, and that was with jumbo packets). The X540-T1 (single port) cards are around $330. If you do decide to go that route be careful what network cables you use, 10Gbit ethernet is much much much more sensitive to cabling than 1Gbit. Had issues at times where the network would crawl practically to a stop, found out I was getting massive packet loss across the Cable Matters Cat6 cable I was using, probably not Cat6 connectors, replaced them with Monoprice Cat6a with shielded connectors and haven't had a problem since.

    As to if it's worth it or not, for me I'll always remember that night I was moving steam games around from my local SSD to the iSCSI drive on my server, and after copying a 10GB game to the server at 400+MB/sec I realized I actually didn't want it there I still wanted it on the SSD so I moved it back. And since I was using multipath iSCSI, and the file copy was still in the servers RAM, and because my desktop had plenty of RAM to cache a 10GB file copy to, I was barely able to see the file transfer window open long enough to see that the copy speed was pegged at 2GB/sec. Totally worth it.
     
  18. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    as far as i knew all Ethernet specs allowed crossover by default. I remember it working when i was a kid using 10/100Mbps Ethernet.
     
  19. halfelite

    halfelite n00b

    Messages:
    31
    Joined:
    Feb 17, 2009

    The options are when you get into NAT translation, firewall settings just to name a few but if you set this up correctly your slow down would be on the WAN side if you are using it as a router not the switching side from computer to computer
     
  20. Aluminum

    Aluminum Gawd

    Messages:
    671
    Joined:
    Sep 18, 2015
    So I had an account here a decade ago or something but mostly only lurk, but just can't ignore this one since OP was actually considering some bizzare card on newegg for way too much money.
    I mean hell, not even intel cards and thinking of spending $500 or more, I have to save him! :eek:

    First, add this name to your repertoire:
    Mellanox

    Second, remove this term from future consideration for anything besides gigabit to the desktop unless you are an enterprise with a massive investment in dead-end legacy cable:
    RJ45 - its dead Jim. (still overpriced nics/switches, significantly extra power required per port, upper limit of cable spec etc)

    Then go to fleabay and become enlightened :D


    For under $60, you can have a 10Gbe peer-to-peer link up to 15' or so:
    <$20 ConnectX-2 SFP+
    ~$20 SFP+ DAC copper cable 1-5 meters

    For up to a few hundred meters you can get SFP+ transceivers as low as $16, premade cables surprisingly cheap. Doctors agree, fiber is good for you.


    For ~$100 you can have a 40/32Gbit IB peer-to-peer link:
    ~$40 ConnectX-2 VPI QSFP (ignore EN versions)
    ~$20 QSFP DAC copper cable 1-5 meters (might have to settle for slower chinese vendors or pay a bit more)
    36 port switches are dropping under $400 if you want to expand later

    QSFP 'active optical cables' tend to be fairly expensive but sometimes you can get lucky, I grabbed a few 100' for $30 each last year yay me ;)


    For <$400 you can have a 56Gbit FDR / 40GbE peer-to-peer link:
    <$200 ConnectX-3 VPI (ignore EN SFP+ versions, but EN QSFP are ok if you want only want ethernet)
    ~$20 QSFP DAC copper twinax cable 1-5 meters (might have to settle for slower chinese vendors or pay a bit more)

    Used FDR switches are starting to show up lately, there is an 18 port for under a grand right now (retail is closer to 10k) I'm currently fan-modding a tabletop switch.
    FDR optical cables are still rarely cheap though :(


    Mellanox cards are well supported by Win/Lin/VMware/etc, you can even get them to work on pfsense with a little elbow grease and they are usually years behind on freebsd releases which is usually years behind on hardware support. The X-2 and later cards also support some interesting advanced things like RDMA, this can make certain things really fly like SMB3 shares.
     
  21. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    Well that's good. I am only using this for switching. I use modem/router for WAN.

    FYI the star tech for 240 per card and was cheapest on newegg. I never buy no named or used unless recommended by someone hence why I am going with the cards other posters posted about from ebay. I can do this for like 100 bucks now total :)
     
  22. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    I got a couple questions.

    So look at the picture below from dropbox and humor me here and tell me if this works or not.

    https://www.dropbox.com/s/2vo35al0yskwt2u/Network - New Page.png?dl=0

    If that 10Gb E cheap o switch is set up as a pure switch will this work? As in will this run at Gb E and 10 Gb E speeds because this is purely switching.

    This would allow multiple Gb E devices to get full Gb E speeds instead of just my desktop getting 10Gb E crossover and everyone else stuck to a single shared Gb E right?

    The guy who has used this can you elaborate on this and tell me if I am missing anything and if this will work at full speeds because if the NAS can feed all the Gb Es at Gb E speeds that is so much better then the massive congestion that could happen being limited to a single Gb E port. Imagine 2 computer back up while 2 people are streaming Plex at 1080P -_-

    Will everyone also be able to access internet at full speeds? Will this switch be able to handle several back ups and plex viewings while allowing people to access the internet at 120/25Mbps?


    Also Aluminum can you elaborate with examples of products because I have no idea on what to look for or what my options really are. You have some interesting ideas.

    Also my whole house is wired with Cat 5e or Cat6 already so being able to use that and not rewire the whole house would be nice So does that leave IB out?
     
  23. cbf123

    cbf123 n00b

    Messages:
    50
    Joined:
    Dec 12, 2013
    Below 1gig it requires a special crossover cable. 1gig and up it uses the regular cable and both ends automatically figure it out.
     
  24. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    yea i wasn't referring to the wiring but maybe he was. Thanks for the info. I thought you still needed to do cross over :p

    In that post i was talking about settings and such. Thats what i thought he was talking about.
     
  25. ST3F

    ST3F Limp Gawd

    Messages:
    181
    Joined:
    Oct 19, 2011
    I own a mikrotik with 2x SFP+ ; works like a charm with
    • HP NC552SFP 614203-B21 (Emulex OCe11102-N)
    • HP NC522SFP 468349-001
    • Dell 10GbE SFP+ PCIe / Broadcom 5711 Network Card NIC KJYD8
    • Dell 10GB PCI-E Xf Sr Server Adapter Mfr P/N E15729 / Intel expx9501afxsr
    • Intel 520 XFP
    and SFP+ from finistar ;)

    Next switch will be the Netgear GS728TXS with 4x 10 GbE and flow control
     
  26. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    So sorry to bug you guys on this again but whats a good cable for SFP+ that is long. I think my original idea needs to change as in a DAC works for the NAS but DAC won't work for PC

    are these any good with a mono price cable?

    http://www.ebay.com/itm/Arista-SFP-...439a69&pid=100010&rk=5&rkt=24&sd=181515173439

    monoprice cable
    http://www.monoprice.com/product?c_id=102&cp_id=10237&cs_id=1023704&p_id=10328&seq=1&format=2

    or is a certain brand/model a must use? I don't am unfamiliar to this stuff so not sure if certain items are good and certain ones are junk....no reviews on this type of stuff :/

    Thanks
     
    Last edited: Dec 12, 2015
  27. Netwerkz101

    Netwerkz101 Gawd

    Messages:
    663
    Joined:
    Jul 11, 2010
    I may have missed a post ..but I did not see the cards you actually went with.

    (Remainder of post is guessing/assuming Mellanox ConnectX)

    When you say "DAC won'twork for PC" ... is your Win 7 Pro 32Bit?
    I notice 2008R2 drivers on their site but only 64bit.

    To find cables, I went here:
    http://www.mellanox.com/page/cables

    MC3309124-007 (7m) is the longest I saw direct from Mellanox.
     
  28. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    these will be the cards i will be using.
    http://www.ebay.com/itm/Lot-of-2-Me...-/301689994552?hash=item463e1ffd38&rmvSB=true

    DAC won't work because of range/reach/distance. I need a longer cable which is why I was looking at the transceiver and monoprice fiber cable.

    DAC will work for the Server but my desktop will be too far and I need a longer 10 Gb E cable so I was going to do fiber with a transceiver...or at least i think that's what i need to do.
     
  29. Netwerkz101

    Netwerkz101 Gawd

    Messages:
    663
    Joined:
    Jul 11, 2010
    Understood now. Can't help as I only have experience with DAC at this point. You seem to be on track though.
    In reading up on Mellanox's module: MFM1T02A-SR, it looks very similar in specs to what you picked out.
     
  30. rsq

    rsq Limp Gawd

    Messages:
    246
    Joined:
    Jan 11, 2010
    IMO 10 GBe belongs on fiber.

    You can order 2 SFP+ modules from fiberstore and an OM3 patch cable in the length you need. You can use up to 300M of OM3 cable with the standard short reach SFP+ modules.

    The 2 modules will set you back about 30 bucks. the cable another 20 or so. It;s even cost competitive with copper. It's also much easier to connect hardware of different brands together if you go fiber.

    Fiber links are much more reliable, and cards will require much less power for the drivers. Also much easier to reach acceptable distances.

    Test for the skeptics who swear by DAC: run iperf test, tie knot in cable, run iperf test again. Observe bandwidth.

    Just my 2c...
     
  31. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    no offense but can you elaborate a little more...I literally only know whats in this thread and a little more with googling and talking to a friend that does IT stuff but has limited experience with 10GbE.

    Can you provide a source to this example you are using? I only need to know how to connect those mellanox to this mikrotik switch and I need a cable that is like 60 feet. What do I need? My friend thinks that those transceivers and monoprice cable will suffice and your confusing me. Does what I have picked out work or not? Or is there a much better similarly priced option?

    http://www.amazon.com/dp/B00KVF7S40...TF8&colid=2EKNW43BVT70W&coliid=I2NLHBGWS21DIL
     
  32. xx0m3gaxx

    xx0m3gaxx [H]Lite

    Messages:
    70
    Joined:
    Oct 24, 2013
    What nas/software are you using? sorry if I missed the post on it. if you are using Freenas please note I do not think the mellanox cards will work with it. I ordered them without looking a while back and FN does not seem to pick them up :(
     
  33. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    Using SnapRAID on windows. I would assume they are supported....Bad assumption?
     
  34. xx0m3gaxx

    xx0m3gaxx [H]Lite

    Messages:
    70
    Joined:
    Oct 24, 2013

    Tested on windows 7 for a quick test. windows update does not seem to pick them up at all. not sure if windows server will or not. honestly having odd luck with these cards and I got them back when they were 90 for the set of 2 from the same guy. would seem the lack of support is the reason the cards are this cheap. looking at the listing the seller knows its hard to get support as they say make sure your os will support the card and they do not supply drivers for them.
     
  35. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    interesting thanks for the tip. Are there other cheap cards that support Windows 7 and/or 2008 R2
     
  36. xx0m3gaxx

    xx0m3gaxx [H]Lite

    Messages:
    70
    Joined:
    Oct 24, 2013
    What I tend to do is find a card on ebay and look it up on google and find reviews and see what people say. at one point it was new so people wanted to know about it just like gfx cards and cpus a lot of times there are videos as well lol
     
  37. olavgg

    olavgg Limp Gawd

    Messages:
    232
    Joined:
    Oct 27, 2010
    I have both fiber and passive DAC's. There is no big difference when it comes to performance.
     
  38. olavgg

    olavgg Limp Gawd

    Messages:
    232
    Joined:
    Oct 27, 2010
    I have tested Chelsio S310E and Mellanox ConnectX-2 EN on Linux and FreeBSD. Holy cow the performance is awesome! Seriously hitting 10Gbps when transferring files from my NAS is just as exciting as when I got my first SSD or if you're old enough..Voodoo2 :D

    10GBase-T(RJ45) is dead! You want SFP+, less power, less latency and much cheaper! The only thing I'm missing now is a 12 port SFP+ switch that is not noisy. Currently I've built my own 8 port switch by bridging 4 dual port mellanox card on FreeBSD. A small performance hit, but I still get on average around 8Gbps
     
  39. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    I had a 3dfx card back in the day :D

    So no one has answered yet...does that fiber cable and transceiver that my friend found me work for SFP+?

    Getting a card that works I can handle I just need to know if I got the right cabling.