Single Port 25gbe NIC's with 4x PCIe Gen3?

[Zarathustra[H]]

Hey everyone,

I'm in a bit of a bind, trying to ghetto-mod some extra network capacity into a mini-ITX system and only have a 4x m.2 slot available to me.

I'm considering using a m.2 to 4x PCIe slot riserand creatively mounting it somehow.

Ideally I'd want 25gbe bandwidth out of this solution.

Does anyone know if there are any single slot NIC's that work at full speed on 4x PCIe? Gen 3 or even Gen 4 would do the trick.

I've seen a few dual port ones with 8x slots, but in my experience just running these with half the number of PCIe lanes and only using one port doesnt work well, at
least judging by the old Intel AT2 adapters I've tried this on.

Worse comes to worse, I'll just do 10gbe, but I'd really like more bandwidth on this solution, as I intend to run NVME storage remotely, and want to maximize transfer speeds.

Appreciate any suggestions!

--Matt

 

[Zarathustra[H]]

...or does anyone have any experience with any brand of 25Gbit adapter, even dual port ones, working well and attaining full speeds when plugged into an x4 gen3/4 port, despite being an x8 card, if only one of the ports were in use?

Just because my decade old Intel adapters didn't do this, doesn't mean that no new ones will.

 

[Eulogy]

You're issue is going to be forcing an x8 interface into a x4 slot. I'd expect that to never work well - doubly so if you think you're actually able to saturate the interface in use. I can't think of any reputable 25g NIC that'll work for this application. What's in your current PCIe slot? Can you get a different (ATX?) motherboard to do this correctly?

 

[Sniper|3d-R|]

Short answer, no. You'll not see anything close to 25GB with PCIe 3.0 4x. You'll need PCIe 6.0 for the speeds you'll want to achieve with 4x.

https://mygraphicscard.com/pci-express-3-0-vs-4-0/
https://en.wikipedia.org/wiki/PCI_Express#History_and_revisions
https://www.techspot.com/news/41212-pci-express-30-specification-released.html

I was one of the first humans to test & validate 25GB Intel NIC's (XXV710) in a Linux KVM environment when I was in that dept.

 

[Eulogy]

Short answer, no. You'll not see anything close to 25GB with PCIe 3.0 4x. You'll need PCIe 6.0 for the speeds you'll want to achieve with 4x.

https://mygraphicscard.com/pci-express-3-0-vs-4-0/
https://en.wikipedia.org/wiki/PCI_Express#History_and_revisions
https://www.techspot.com/news/41212-pci-express-30-specification-released.html

I was one of the first humans to test & validate 25GB Intel NIC's (XXV710) in a Linux KVM environment when I was in that dept.

Mind your units PCIe 3.0 is 3.94GB/s, aka, ~32Gbps. Plenty for a single 25Gbps NIC, spec-wise.

 

[Sniper|3d-R|]

Mind your units PCIe 3.0 is 3.94GB/s, aka, ~32Gbps. Plenty for a single 25Gbps NIC, spec-wise.

Opps, you're 100% correct. Not sure what I was thinking..

 

[Zepher]

Opps, you're 100% correct. Not sure what I was thinking..

Maybe you were thinking in gigabytes and not gigabits.

 

[Zarathustra[H]]

Short answer, no. You'll not see anything close to 25GB with PCIe 3.0 4x. You'll need PCIe 6.0 for the speeds you'll want to achieve with 4x.

https://mygraphicscard.com/pci-express-3-0-vs-4-0/
https://en.wikipedia.org/wiki/PCI_Express#History_and_revisions
https://www.techspot.com/news/41212-pci-express-30-specification-released.html

I was one of the first humans to test & validate 25GB Intel NIC's (XXV710) in a Linux KVM environment when I was in that dept.

Interesting.

I trust your hands on experience, but I wonder why this is....

Gen 3 is rated at 985 MB/s per lane. (yes, the PCIe spec is in bytes for some reason, never understood why) Multiply by 4, and then by 8 (to convert to bits) and that gives us 31520 Mbit/s, or 30.7812 Gbit/s which ought to be enough, even allowing for 23% overhead.

Maybe 25Gb Ethernet has a lot of overhead?

In theory it should be enough bandwidth.

The interesting part is that most dual port SFP28 adapters out there are 8x Gen3, so you'd think that just having one port would require half the lanes, at 4x gen3. Or maybe those 8x adapters are blocking in bandwidth and can't support both ports going full out at the same time?

Unfortunately however, bandwidth rarely just adds up that way. Sadly it is highly based on the individual chip/adapters layout and how it uses each lane. Consumer GPU's usually allow you to just total up the bandwidth from all the available lanes and use it, but experience has taught me that enterprise NIC's don't always work that way.

I'd appreciate your thoughts from your hands on experience.

 

[SamirD]

I think you're up against, is it better to have 100% of 10Gb or only a percentage of 25Gb--you see the issue.

The other thing I think about is overall pcie lanes since systems with only 4x will also have limited lanes to max out an nvme and the pcie.