...the 10-25GbE rabbit hole... moving fast storage off the main system.

Okay, so I'm basically mirroring a reddit post I did, to get your feedback.

https://www.reddit.com/r/HomeNetwor..._1025gbe_rabbit_hole_moving_fast_storage_off/

Post body:

Please bear with me... I'm exploring the idea of offloading my 'faster' storage (last benchmark peaked at 2684.999 MB/s read) to an 'older' system (with more available expansion slots) and using higher speed network interfaces (10-25GB) to connect that system and my main system.

--> to follow up on that, these are FusionIO SSDs, they interface via PCIE slots, not NVME/M.2/SATA.

The oldest pcie interface I have is PCIE 2.0 PCIE 3.0 , the most capable system (using that interface) being a dual processor xeon system. Oodles of expansion slots, ram, blargh. So as far as direct to motherboard interfacing, PCIE 2.0 is the limiting factor. (other things such as CPU aside)

I've... 'looked' at 10/25GbE adapters... briefly, am aware they take a specific not ethernet cable...

I guess I'm just looking for opinions on moving/offloading high speed interfaces and fast storage arrays to another system and using a very fast network interface/speed to utilize them with minimal loss in performance 'compared to keeping it in my main system'...

Did that make sense?

Edit: I was wrong, the motherboard/server I have has PCIE3.0. It's from a ASUS ESC4000 G2S. So no PCIE 2.0 limitation.

Click to expand...

What I've gathered so far from the responses:
- It's possible, given the system that would host the storage drives has an asston of cores and pcie slots.
-since cable lengths are short, it's easier.
- NICs... not really specified, but...cable / SFP+ ? (so this takes a NIC + transceiver)?
--And since this would be a point to point network...well no router needed.
- OS and software on the host device side really matters.

tl;dr storage thought experiment: having PCIE based SSDs used for server acceleration, how might I connect a host system to a client system and have the fastest possible network connection so that it's 'as if' it's a local drive...

...because my motherboard is lame and not having 9 pcie slots.

I'm scared. this isn't genmay.

don't hurt me
or do.
lay into me.

I fully acknowledge that the implementation of this thought experiment, both initial and ongoing costs, would far outweigh the benefits of just staying with what I have currently.
But I wanna learn something... without paying money and then stumbling and learning.

SFP+, you can get Chelsio T6225 25Gbe cards cheap...where i got mine (They are in the US)

https://www.ebay.ca/itm/356713709231

I run one of these from my main desktop, over fiber, to another on direct into my pfsense box.

From my pfsense box, I then have some Chelsio 40Gb cards, direct links to my TrueNAS server for a 40Gb link between them using a DAC cable (1meter long)

Christ, I wish the secondary pciex16 slot on my main computer's motherboard wasn't limited to x4...

You mention both DAC and fiber... What cost difference does it make if it's <2m?

and if I'm understanding...even with the nic you linked, would I still need a tranceiver?

DAC is fine if you can get one for the distance, and then you do not need transceivers as they are built into the DAC. Personally I have been using the 10GTek cables off amazon and no issues for 3 years now with 10Gb/40Gb DACs

For PCIe speeds, remember, PCIe slots are rated in GB/s , the network cards are Gb/s

So even if your PCIe x4 is Gen 3 - that is 4GB/s bandwidth = could technically handle a single saturated 40Gb Network link (assuming the I/O on both ends could keep up and file size matters- many small files = slower transfer speeds due to I/O seek vs larger files)

MrGuvernment said:

DAC is fine if you can get one for the distance, and then you do not need transceivers as they are built into the DAC. Personally I have been using the 10GTek cables off amazon and no issues for 3 years now with 10Gb/40Gb DACs

For PCIe speeds, remember, PCIe slots are rated in GB/s , the network cards are Gb/s

So even if your PCIe x4 is Gen 3 - that is 4GB/s bandwidth = could technically handle a single saturated 40Gb Network link (assuming the I/O on both ends could keep up and file size matters- many small files = slower transfer speeds due to I/O seek vs larger files)

View attachment 811677

Click to expand...

And now, if my understanding is right, it's the NEWER system that's the limiting factor - as the only extra long (x16) expansion slot is operating at x4...
how the tables turn...

But ultimately unless I go off the deep end, it'll be fine.

Simplify and do a DAS box instead if only one comouter accessing? Just a thought. Otherwise just 10gb with cheapest DACs. That way could expand later with a switch to other computers.

10Gb has become pretty cheap, and I would be extremely surprised if you were actually doing anything that would be limited by 10Gb. You can "benchmark" some very fast NVMe/PCIe SSDs and get speeds faster than 10Gb, but in those situations the "benchmark" is almost certainly reflective of the speeds from the DRAM and/or SLC cache on the SSD. Any kind of relevant sustained file transfer is going to blow through that DRAM and/or SLC cache and then slow down considerably (well below 10Gb) for the majority of the duration of the file transfer.

If it's all stuff that is in the same room or nearby rooms, then get a cheap 10Gb SFP+ switch (literally a switch with nothing but SFP+ ports) and use DAC cables for everything, to avoid the whole mess of expensive SFP+ transceivers that may or may not be compatible with your equipment.

Right now I have my main computer, my backup computer, and all three of my file servers connected to an 8 port 10Gb SFP+ switch using Intel X520 (PCIe 2.0) and X710 (PCIe 3.0) cards. I also have a couple 2.5GbE switches with 10Gb SFP+ ports also connected to the 10Gb SFP+ switch for devices that are farther away or where 2.5GbE is fast enough, including 2.5GbE runs to my WiFi 7 Ubiquiti AP and my pfSense router. It works great, and I believe that the total cost of the 10Gb SFP+ switch, the 2.5GbE switches, all of the Intel 10Gb cards, and all of the DAC cables was maybe $300-$400. An amazing bargain compared to what was possible even a handful of years ago.

I run this "OpenSpeedTest" server on my main file server. It is an HTML5-based speed test, and it allows me to do speed tests at various locations on my LAN to test for potential bottlenecks. It's very handy.

When you say 'not ethernet cable', you're mistaken. Ethernet runs on many media: thicknet, thinnet, twisted pair, DAC, fiber, actual ether (wifi is wireless ethernet), and if you want to be exotic you can probably use ir or free air lasers.

You're looking at 10g/25g ethernet.

10G can run on rj45 if you want. Spec says cat6a for 100m in dense conduit, maybe 55m on cat5e in dense conduit... It's worth trying whatever cable is in your wall; my cat5 from ~ 2003 works at 10g just fine for the two runs where it makes sense (and I haven't tried the other runs cause I don't have enough ports).

Cheap enterprise 10g equipment off ebay tends not to do 2.5g/5g, so don't expect that to work. I use intel x540 10g-baseT cards happily. x550 should also work, x710 does 2.5/5g I think.

For short runs, dac is cheaper and more power efficient. x520 is good. I think dac goes to two meters, if it's close, but not that close, preterminated fiber plus transceivers might still be cheaper than 10g-base-t cards/if you might want a switch. Avoid 10g-base-t transceivers in sfp+ ports unless there's no other option.

Lots of enterprise stuff is going to be dual port, depending on what you're doing that might get you to 20gbps with aggregation, or you might not be able to aggregate.

Word on the street is 40gb ethernet is a evolutionary dead end, but it might be cheaper than 25gbps, so it's worth a search or two.

Seems unlikely that anything more than 10G will work over twisted pair, but the latest twisted pair 10g stuff is supposed to be a lot more power efficient, so who knows if they'll figure something wacky out.

toast0 said:

When you say 'not ethernet cable', you're mistaken. Ethernet runs on many media: thicknet, thinnet, twisted pair, DAC, fiber, actual ether (wifi is wireless ethernet), and if you want to be exotic you can probably use ir or free air lasers.

You're looking at 10g/25g ethernet.

10G can run on rj45 if you want. Spec says cat6a for 100m in dense conduit, maybe 55m on cat5e in dense conduit... It's worth trying whatever cable is in your wall; my cat5 from ~ 2003 works at 10g just fine for the two runs where it makes sense (and I haven't tried the other runs cause I don't have enough ports).

Cheap enterprise 10g equipment off ebay tends not to do 2.5g/5g, so don't expect that to work. I use intel x540 10g-baseT cards happily. x550 should also work, x710 does 2.5/5g I think.

For short runs, dac is cheaper and more power efficient. x520 is good. I think dac goes to two meters, if it's close, but not that close, preterminated fiber plus transceivers might still be cheaper than 10g-base-t cards/if you might want a switch. Avoid 10g-base-t transceivers in sfp+ ports unless there's no other option.

Lots of enterprise stuff is going to be dual port, depending on what you're doing that might get you to 20gbps with aggregation, or you might not be able to aggregate.

Word on the street is 40gb ethernet is a evolutionary dead end, but it might be cheaper than 25gbps, so it's worth a search or two.

Seems unlikely that anything more than 10G will work over twisted pair, but the latest twisted pair 10g stuff is supposed to be a lot more power efficient, so who knows if they'll figure something wacky out.

Click to expand...

Another thing to keep in mind is 40gb+ cards will most likely use a lot of power and need extra cooling as well.

D-EJ915 said:

Another thing to keep in mind is 40gb+ cards will most likely use a lot of power and need extra cooling as well.

Click to expand...

This, as most of these cards are expected to be installed in U servers with high airflow.

For me I just put a 240m fan in my pfsense that points at all my PCIe cards, could do the same in a desktop, or try to mount a smaller fan on the heatsink to keep airflowing.

avatardelta said:

And now, if my understanding is right, it's the NEWER system that's the limiting factor - as the only extra long (x16) expansion slot is operating at x4...
how the tables turn...

But ultimately unless I go off the deep end, it'll be fine.

Click to expand...

as noted a x4 slot, even if PCIe Gen 3, you wont likely saturate.

toast0 said:

Word on the street is 40gb ethernet is a evolutionary dead end, but it might be cheaper than 25gbps, so it's worth a search or two.

Click to expand...

40Gb is pretty dead in the datacenter, but wasn't an evolutionary dead end. 40Gb is basically 4x 10Gb aggregated. You can even split it into 4x 10Gb on most cards and switches with a big messy adapter. It's just disfavored vs. newer, faster versions since it uses 4 pairs of fiber. It's an infrastructure headache, and if something normal that uses a single pair is close in price the single pair wins. There are multiple types of 100Gb, one of which is 4x25Gb and another is 2x50Gb. 200Gb and 400Gb also use more than one line. In other words using multiple lines isn't dead.

I wouldn't want to pull 40Gb around my house. The connectors are too big and I pull pre-terminated fiber up into first floor wall plates from my unfinished basement. That's annoying enough. Terminating raw fiber also takes more skill and gear than slapping an RJ-45 on a twisted pair cable. I used to be good at that, but now I'm rusty so it takes me a couple minutes. I'm currently on 10Gb and don't see much reason to upgrade to 25Gb+. I mostly went 10Gb because I wanted to play with some particular features on NVidia/Mellanox Connect-X and Solarflare NICs. That was supposed to be some continuing education to go get a new job after my then current employer sold out to private equity. Then I got a new job with a public corp. and forgot about it. Too busy with cloud and AI stuff.