Looking for >gigabit between 2 PCs, I'm just at the 10m range of sfp+ DAC

Discussion in 'Networking & Security' started by arnemetis, Sep 15, 2018.

  1. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,349
    Joined:
    Aug 2, 2004
    Hello,

    I'm interested in running something between my server and my main pc to accelerate network transfer speed. I often move large files and obviously gigabit works, but I'd like something faster. The raid 6 arrays and my ssd are capable of 370MB/sec or more read and writes for large files. 10g ethernet was my original thought, but it's very expensive (and noisy) to get a switch at this point. A lot of my searching has landed me looking at sfp+ DAC, but that has a maximum length of 10m and taking the most direct path (back of pc to ceiling, straight line over server, straight shot down) I'm looking at 33'-4" (9.95m.) I think that is cutting things way to close, and it means the cable would be right against the drop ceiling track, and really cutting things too close for my liking.

    So I'm looking for some advice on how to best proceed with getting this connection going. I'll present a few options below, because there's a lot of different information on what cards to get as well, I'm sure there's a more up to date suggestion around. I've been looking at new stuff, but I'm not opposed to considering used equipment on ebay etc.

    1) Buy two 10gbit ethernet pci express cards for each pc, and run a 30' cat 6a cable between server and patch panel, and a 7' cat 6a cable between patch panel and my pc. Looks like I can pick up the cards for $100 each and the cables will run me about 20, only finding cat 6 keystone jacks though maybe that's fine?

    2) Inspired by this fiber keystone jack I saw at monoprice https://www.monoprice.com/product?c_id=104&cp_id=10426&cs_id=1042606&p_id=2875&seq=1&format=2 maybe I should run my own cables between to almost do a simulated dac? This would also have the nice benefit of opening the opportunity to do a switch later if I wanted, although I do not forsee this being something I ever do. Monoprice sells various color fiber pretty affordably (I think?) but the colors are for different types and I'm a bit lost there. I would need two sfp+ pci express cards, a 10m run of fiber, the keystone jack, a 3m run of fiber, and two of the sfp+ termination modules so I can connect the fiber to the cards. These seem pretty expensive, but I probably just have no idea what I'm looking at. Also doesn't have to be monoprice, was just a reference I was using.

    3) I could raise up the server and go the dac route anyway. I would have to buy / fabricate some kind of support for it, and I'd still be cutting it close. I'd also be making a new hole in the ceiling directly over my pc. Not my top choice but cost wise this may make the most sense. I'd need two cards and the cable.

    Thank you for reading, I am open to comments and suggestions, including methods that I have not explored here.
     
  2. IdiotInCharge

    IdiotInCharge Not the Idiot YOU are Looking for

    Messages:
    8,110
    Joined:
    Jun 13, 2003
    Used SFP+ cards, new 10Gbit transceivers, and appropriate fiber are pretty cheap- I almost went that route, but decided not to because 'consumer' and SOHO 10Gbit is going copper with 10Gbase-T.

    Still, you can just grab two cards and run CAT6a and roll with it. Any consumer 10Gbit card you buy is going to be some implementation of the Aquantia AQ-107, which I'm using one of right now in my server. My desktop has the same chipset built in.
     
  3. tedych

    tedych Limp Gawd

    Messages:
    139
    Joined:
    Jan 18, 2013
    I had the same dillema before.
    I ended up getting Mellanox ConnectX-2 SFP+ adapters. Up to 10m you can run copper and for longer you could get fiber for slightly more and be done with it.
    I use my cards with Windows Server 2016 and I get 9.5gbps on iperf tests and 200+MB/s on real copy to a HDD while at the same time I could use another HDD or SSD for another transfer.
    10g LAN cards would be much more expensive and I don't expect this to change anytime soon. 10g switches are also a no-go and for 2-3 machines it's much cheaper to equp every machine with one or two cards and implement something like token-ring or just peer-to-peer (when using 2 cards on each machine for 3 machines).
     
  4. Vengance_01

    Vengance_01 [H]ardness Supreme

    Messages:
    5,555
    Joined:
    Dec 23, 2001
    Just do 2 cards and a crossover cable and when small switches get cheaper add it in.
     
  5. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,349
    Joined:
    Aug 2, 2004
    Thanks, I'm going to attempt to make a shopping list here but I am not sure I am selecting the right transceivers, and I am especially lost when it comes to selecting the right cable.
    1) Start with two of the sfp+ cards tedych mentioned below for $30 each
    2) Get two transceivers for $16 each - https://www.fs.com/products/11552.html
    3a) Get a 10m cable for $5.70 and a 3m cable for $3.30 here - https://www.fs.com/products/41728.html (+$8.50 shipping) However they do not sell keystone jacks, but I can get one on ebay for $7.50 shipped - https://www.ebay.com/itm/Fiber-Opti...397878?hash=item4d2e028136:g:CDkAAOSwk5FU0qWO
    3b) Alternatively I could just get a 15m cable for $7.50 (+$8.50 shipping)
    So total for the two cable & keystone route is $117, total for single cable and no keystone route is $108. For such a small difference, assuming they will all work together ( the transceivers, cables, and keystone jack) I'd go the two cable route.

    Going the ethernet route, the cards end up being cheaper than the newegg results at 80 a pop, picking up a 35' cat 6a and a 7' cat 6a at amazon gets me to a total of $177 this way.

    Thanks for the reply! Looks like from ebay I can get those cards shipping from the US for $30 a pop, a little cheaper if I want to get from from China. As I mentioned I'm really at the limit of 10m so I want to look at the options for longer runs via a keystone jack, but there's a whole lot of options. I ran down some options above in my reply to IdiotInCharge, what do you think of that?

    Thanks, I just might, but unless I'm way off base it looks like the sfp+ router is significantly cheaper right now.


    So at first glance, the sfp+ card method looks to be a winner, unless I suck at selecting fiber products?
     
  6. mkrohn

    mkrohn 2[H]4U

    Messages:
    2,323
    Joined:
    Apr 30, 2012
    I made the mistake of looking at 40gb you can get dual 40gb cards for like $100. Now the thoughts of taking a retired mining mobo and popping some of these in to create a high powered switch. I don't think I'd connect more then 3 machines yet.
     
  7. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,349
    Joined:
    Aug 2, 2004
    Yeah that's way past the capabilities of my machines, no need for me to even look at that level! Do you have any experience with picking fiber cables, could you comment on my 'shopping list' I made above? In particular the transceivers, cables, and the keystone jack? Really unsure if I've selected the right combination of parts.
     
  8. cyclone3d

    cyclone3d [H]ardForum Junkie

    Messages:
    12,754
    Joined:
    Aug 16, 2004
  9. tedych

    tedych Limp Gawd

    Messages:
    139
    Joined:
    Jan 18, 2013
    I got my gear from a local guy, I didn't want the delay with purchasing from abroad/evay etc.
    Here is an example of 10m ready-to-use fiber cable but the length can be several tens of kilometers :) . Two cards and you are done. The cable is not very cheap though. I'm using twinax (copper) cable because my span is below 10m.
     
  10. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,349
    Joined:
    Aug 2, 2004
    Oh cool, that makes this path even cheaper. Problem is, I dint know what kind of transceiver is in there (if it even matters.) So if I picked up two of these, what kind of cable and keystone jack would I need? Would the ones I posted above be appropriate?
     
  11. mkrohn

    mkrohn 2[H]4U

    Messages:
    2,323
    Joined:
    Apr 30, 2012
    One of the biggest points to doing things around here is going above what you need :)
    Aren't those the ones people are ditching because they use a ton more power than like Mellanox and some of the others?
     
  12. IdiotInCharge

    IdiotInCharge Not the Idiot YOU are Looking for

    Messages:
    8,110
    Joined:
    Jun 13, 2003
    Probably true, but it matters very little for desktops, and fiber at 10Gbit uses less power than copper; making copper SFP+ 10Gbase-T transceivers is actually an issue, as the SFP+ standard doesn't seem to easily support the power requirements and there are questions as to whether it ever will.

    Microtik and I think Aquantia themselves are working to make that happen. Biggest problem is that the industry is moving away from SFP+ in general, and toward QSFP and better for 40Gbps+.

    Which is why this stuff is cheap on ebay- but consumer/SOHO grade stuff is just coming with 10Gbase-T.
     
  13. sinisterDei

    sinisterDei Gawd

    Messages:
    822
    Joined:
    Dec 1, 2004
    Inexpensive 10 GbE cards + a crossover cable, no need for a switch. Should be fine and no need to mess with fiber.
     
  14. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,349
    Joined:
    Aug 2, 2004
    Can you give me an example of cheap 10gbit ethernet cards? Per my second reply in the thread, it looks like going the ethernet route will run me $177 while my first stab at fiber runs me $117. If I go the secondary route of getting sfp+ cards with transceivers in them already I can knock it off a bit more and get under $100. No one has replied with specifics about the type of cables that would be appropriate for me, so I may be way off base with that part though.
     
  15. sinisterDei

    sinisterDei Gawd

    Messages:
    822
    Joined:
    Dec 1, 2004
    The Aquantia cards mentioned before seem fine, though I've never touched them personally.

    I'd probably try and find a pair of X540 or X540-T2 cards second hand or by eBay or something. There's one listing on eBay for $48 which is super cheap. I've seen them occasionally on FS/FT here and on AT and STH as well. I'm pretty partial to Intel NICs though, and that preference would cost a bit of money.
     
  16. Keljian

    Keljian Limp Gawd

    Messages:
    291
    Joined:
    Nov 7, 2006
    Look seriously at one of these:
    https://mikrotik.com/product/CSS326-24G-2SplusRM

    Amazon have them and it would potentially negate you having a second switch on the network

    They are fanless

    Add either copper or fibre SFP+ (cheap lamps, see above)
    Add Mellanox connectx-2 (or chelsio t420-so-cr)
     
  17. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,349
    Joined:
    Aug 2, 2004
    Thanks Keljian. I don't think that switch brings me any benefit with only two sfp+ ports, it's no different that directly connecting the two pcs with sfp+ and then keeping the current switch I have in place. None of the other devices on the network would require access to faster than gigabit data, I probably shouldn't have even brought up a switch lol.

    I also should have never mentioned that I was near 10m, let's pretend I never said that. I need a 10m cable, a keystone jack, and a 3m cable. I think due to the total length and it not being direct connect copper is out of the picture. I don't know what you mean by lamps, that's the first I'm seeing that mentioned here. The fiber cables come in multimode or single mode, there are various ratings such as OM1/2/3/4, there's lc duplex and sc simplex (I think I need lc dusplex with multimode?) I read that single mode may be a better way to go, but it has issues with short runs such as my 3m? Please could you offer up an example of a transceiver, fiber cables, and keystone combo that would be appropriate to connect over these lengths to one of the common cards you mentioned?
     
  18. Keljian

    Keljian Limp Gawd

    Messages:
    291
    Joined:
    Nov 7, 2006
    you mentioned server - so I assumed the server served other things on the network. This being the case, that switch would allow the server to communicate with other things on the network (albeit at gigabit speed) without a separate network port.. all good. I was in a similar situation and a Mikrotik CRS226, which is an older model, negated the need for extra gear.

    lamps = transceivers (it's like coders = programmers)

    Re cables LC-LC OM3 or OM4 fibre is what you're looking for. OM4 is only marginally more expensive and usually comes in Aqua. At 10gbps OM4 is likely to work well over 400 meters, OM3 will do 300 meters comfortably ( https://en.wikipedia.org/wiki/Multi-mode_optical_fiber ). At the distances you're talking, either would be more than ample

    https://www.ebay.com/itm/3-meters-L...683955&hash=item2ab225641b:g:DjIAAOSwsddbcbUw Would work for 3 meters
    https://www.ebay.com/itm/10-Meter-L...873762?hash=item4b47d7d2e2:g:k68AAOSwNRdX9Tmw Would work for 10 meters

    Transceivers can be a bit finicky depending on card - Chelsio or Mellanox can use any transceivers, intel must have intel transceivers (for the most part)
    https://www.ebay.com/itm/SFP-SR-10G...983962&hash=item237921ea83:g:PuMAAOSwux5YUX0L These would work for mellanox or chelsio

    https://www.ebay.com/itm/Intel-E10G...887111&hash=item2a6a5b817c:g:Mz8AAOSwjXRXYmfi These would work for intel

    In my experience - having done what you're planning to do - short runs of 1 or 2 or 3 meters are a non issue for LC-LC MM.

    https://www.ebay.com/itm/Fiber-Opti...386340987?epid=1762575706&hash=item2126e3247b would work as a keystone
     
    Last edited: Sep 17, 2018
    IdiotInCharge and arnemetis like this.
  19. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,349
    Joined:
    Aug 2, 2004
    This is awesome, exactly what I was looking for! I still don't understand everything fully, but it's great to finally have a working set of components that I can look at, and learn some more. Makes the shopping list easier too!

    Yeah I guess my 'server' isn't serving as much locally as many others might be. I work from home, and 90% of local traffic will be between my pc and it, so that's why I'm looking at options for more speed. I have an existing cat 6 infrastructure that the server feeds the local network and gigabit internet from. Having a dedicated line between my pc and the server will help ensure the internet facing nic isn't saturated with lan traffic. It hasn't been a problem yet, just something I'm thinking about.

    Anyway I'm still working so I should get back to that, but I really appreciate your reply!
     
  20. Keljian

    Keljian Limp Gawd

    Messages:
    291
    Joined:
    Nov 7, 2006
    In my case, I was streaming from the server (8 disk Raidz2/freenas), using it as a router (pfsense), using tvheadend to accept two streaming feeds from a HDhomerun, and moving big video files around for editing - all at the same time. This would saturate a 1gbit link so watching videos on the network on other boxes was just asking a bit too much. The move to 10gbit made it a lot easier as it all "just worked". I've since scaled back as I don't do video editing, nor do I use the server as a router.

    Should also mention, optics are lower latency than cable.
     
    Last edited: Sep 17, 2018
  21. Spartacus09

    Spartacus09 Limp Gawd

    Messages:
    437
    Joined:
    Apr 21, 2018
    Thats only accurate at long distances, pretty sure DAC copper at short distances (<3m) is faster and has lower latency.
     
  22. Keljian

    Keljian Limp Gawd

    Messages:
    291
    Joined:
    Nov 7, 2006
    Nope - Not in my experience. Having tested both.

    I tested with and without virtualisation, with cable and with fibre, and consistently my latency figures were an order of magnitude lower with fibre
     
    Last edited: Sep 17, 2018
  23. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,349
    Joined:
    Aug 2, 2004
    Well alright it looks like I may have a shopping list ready, at long last! Here's what I'm looking at now:
    $31.00 Lot of two Mellanox MNPA19-XTR cards - https://www.ebay.com/itm/LOT-OF-2-M...-ETHERNET-NETWORK-INTERFACE-CARD/282092353923
    $30.00 Two SFP+SR 10GE 850nm MMF transceivers - https://www.ebay.com/itm/SFP-SR-10G...r-for-Ubiquiti-UniFi-US-48-US-24/152356121219
    $9.50 Three pack (guess I'll have spares) keyustone jacks for Multimode LC Duplex - https://www.ebay.com/itm/Fiber-Opti...386340987?epid=1762575706&hash=item2126e3247b
    $7.12 3m LC-LC Duplex 50/125 Multimode Aqua OM3 patch cable - https://www.ebay.com/itm/3m-LC-LC-D...10-Gb-Fiber-Patch-Cable-Aqua-om3/141442609722
    $15.95 10m LC-LC Duplex 50/125 Multimode Aqua OM3 patch cable - https://www.ebay.com/itm/10-Meter-L...imode-PVC-Fiber-Optic-Cable-Aqua/323327873762
    $93.57 Total
    Seems like a winning combo to me. I swapped out for a different 3m cable than Keljian suggested so I don't have to wait until November for it to arrive (lol,) and found a cheap lot of 2 Mellanox cards. I'll probably pull the trigger tomorrow morning on this unless I somehow managed to screw this up and someone finds an erroneous component!
     
  24. Spartacus09

    Spartacus09 Limp Gawd

    Messages:
    437
    Joined:
    Apr 21, 2018
    To reduce failure points and potential signal loss, I recommend doing straight point to point vs. adding the keystones.
    I know its convenient and makes it look pretty, but fiber is alot more temperamental than copper.
     
  25. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,349
    Joined:
    Aug 2, 2004
    You really think so? It's going to be a pain in the ass to run this in the ceiling now that I've got everything finished, so I'd really prefer I only do this once. I could pick up a 15m cable here for $13.50 shipped https://www.ebay.com/itm/NEW-AQUA-1...044290?hash=item44275ae482:g:3F8AAOSwgsVbk9jG or if that seller looks fishy I could get it from a bigger store for $17.95 https://www.ebay.com/itm/15-Meter-L...117306?hash=item4b140fa6fa:g:k68AAOSwNRdX9Tmw
    Either way would save me a chunk of change I suppose, and I could always just run it through an open keystone. Actually I could probably modify a keystone blank with a slot for these to run through once I get the ends through. Hmm this may be a valid point, I suppose I'll sleep on it. As an aside, it brings the cost down to $74.50, this came down pretty low.
     
    Last edited: Sep 17, 2018
  26. Keljian

    Keljian Limp Gawd

    Messages:
    291
    Joined:
    Nov 7, 2006
    One thing I will add to all of this - buy at least one extra fibre cable. Just in case you accidentally break it. (unlikely, but possible)
     
  27. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,349
    Joined:
    Aug 2, 2004
    Buying extra cables puts me firmly in the one run, no keystone camp then lol. Doubling down on the cheaper 15m run gets me a total of $87.25, still stomping the costs of ethernet into the ground. Suppose I'd run them at the same time, and just run one down from the ceiling, keeping the second one for expansion or if I somehow break it.
     
  28. IdiotInCharge

    IdiotInCharge Not the Idiot YOU are Looking for

    Messages:
    8,110
    Joined:
    Jun 13, 2003
    We've used fiber couplers without a problem at gigabit speeds; only concern would be going to 10Gbit.
     
  29. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,349
    Joined:
    Aug 2, 2004
    Alright that does it then. I was thinking this morning I would go with the single cable run anyway, so I've gone ahead and ordered two 15m cables (one is a spare) two transceivers and the two Mellanox cards. Hopefully when I report back I won't have any issues!
     
  30. /dev/null

    /dev/null [H]ardForum Junkie

    Messages:
    14,256
    Joined:
    Mar 31, 2001
    I've done the same setup before....eventually I wanted to expand & picked up a Mikrotik 16 port 10Gbit switch. Long as you stick with SFP+ transceivers (and not DAC) and cheap mmf it should work great. ConnectX-3 cards are significantly better (feature/speed wise) and those are sub $30 on sale now, so I'd go for those over connectx-2 if you can.
     
  31. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,349
    Joined:
    Aug 2, 2004
    Oh well, purchase was made this morning. I didn't go for DAC because realistically I'm beyond the 10m limit. I can't imagine what features would be better, and speed wise my limiting factor will be my raid arrays which do about 370MB/sec so unless the x-2 cards do less than 5gbit I won't be able to use the speed anyway. Since I picked up a lot of two for $31, the cost increase is pretty significant comparatively. What features do you speak of?
     
  32. /dev/null

    /dev/null [H]ardForum Junkie

    Messages:
    14,256
    Joined:
    Mar 31, 2001
    Better real world throughput & pci-e 3.0 connectivity vs pci-e 2.0 on connectx-2. I think more hardware offloading features as well...IIRC Connectx-3 is also still officially supported & Connectx-2 is EOL.
     
  33. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,349
    Joined:
    Aug 2, 2004
    Ahh ok, thank you for the clarification. I was planning on putting the card on the server's pci-e 2.0 x4 (x8 physical) slot anyway. Like I said I don't have anything that comes close to saturating a 10gbit link. As the years pass and other pcs may want to get faster than gigabit, I may consider upgrading these cards. All in all, sucks that 2.5 / 5gbit switches never came about, that was my original plan.
     
  34. tedych

    tedych Limp Gawd

    Messages:
    139
    Joined:
    Jan 18, 2013
    ConnectX-2 are able to do 9.5gbps with the iperf test. They work out of the box in Windows Server 2016 and Win10 as well.
    Also as arnemetis said, they are most likely to be put in 2.0 slot that is fed by the chipset and most low to mid range chipsets at least in AM4 are 2.0. We'd like to use all the CPU 3.0 lanes for the graphics card :) . My cards are also in 2.0 slots. Their (slots') theoretical limit is 16gbps for x4 anyway I think.
    Of course if anyone finds similarly priced ConnectX-3 cards, they should get that.
    At least my connectX-2 cards have some little bugs. If one machine is rebooted sometimes the connection is not recovered after the machine boots. It seems some bug in the firmware. In one end the connection says "cable unplugged". A simple scheduled script checks the condition of the other machine and the state of the connection and disables/enables the connection which resolves the issue.
     
    IdiotInCharge likes this.
  35. Zepher

    Zepher [H]ipster Replacement

    Messages:
    16,693
    Joined:
    Sep 29, 2001
    what happens when you use a cat5e cable between a pair of 10g cards? will it work at reduced speeds but higher than gigabit? or not work at all?
     
  36. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,349
    Joined:
    Aug 2, 2004
    Dang sounds like I might be in for some fun then. The server runs 24x7 but I shut my pc down every night. Oh well, a little enable/disable action won't kill me. To hell with the high speed ports for the gpu, that's reserved for the raid cards! lol
     
  37. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,349
    Joined:
    Aug 2, 2004
    It might work, it might not. I didn't end up going that route as I went with fiber, but if it's anything like the move from 100 to gigabit short runs can work. I have some old cat 5 patch cables that operate at full gigabit speeds, in fact when my fios was installed they use cat 3 patch cable to run my 150/150 service and it worked.
     
  38. Zepher

    Zepher [H]ipster Replacement

    Messages:
    16,693
    Joined:
    Sep 29, 2001
    I was just wondering since my download machine is upstairs and my plex server is downstairs in the garage, with a 100'+ cable running in the walls between them.
     
  39. arnemetis

    arnemetis 2[H]4U

    Messages:
    2,349
    Joined:
    Aug 2, 2004
    You could get one of the multi mode capable cards (such as the Aquantia linked in the first reply of the thread) and hope, but really I wouldn't do it unless you were prepared to replace the run with cat6a if it didn't work. If you aren't willing to do the cable run, that's $160 just to test.
     
    Zepher likes this.
  40. Keljian

    Keljian Limp Gawd

    Messages:
    291
    Joined:
    Nov 7, 2006
    Sample size of 1 - but I never had any issues with my connectx-2 until I overheated it (don't put the heatsink/card in a place no airflow can reach!) no connect/disconnect issues etc..