In search for a motherboard that can do 8 by x8 lanes pcie

bigdano

n00b
Joined
Jun 14, 2014
Messages
28
All,



I am desperately trying to find any motherboards out there that have the following specs:



  • Intel OR AMD (but preferably intel)
  • Single or dual processor
  • 7 or 8 PCI Express slots that are 16x sized. Spacing/layout of the slots does not matter.
  • Each PCIE slot should be able to run at 8x at the same time (Meaning, 64 lanes)
  • PLX solutions to the above are NOT ok. I need the speed natively.
This would be going into a server, so weird motherboard sizes are OK



Its been very hard to find this information because so many motherboard manufacturers seem to stretch the truth by using PLX or other memes.

I know the desire may be to ask "why do you need this" but please be assured that we do.



Thanks
 
Last edited:

extide

2[H]4U
Joined
Dec 19, 2008
Messages
3,494
Yeah getting more than 7 slots might be tricky as that's the sorta standard amount. Look through the SuperMicro page though, you might find something. Also would suggest going with EPYC. You get 128 lanes with 1 OR 2 cpu's, so unless you need the CPU horsepower (or additional memory capacity) a single CPU would be fine. You can also get enough lanes to run 7 lanes at x8 natively with a Threadripper system. (Threadripper has 64 lanes, 4 of which go to the chipset, and 60 available)

Out of pure curiosity, what are you guys doing with this? Sounds interesting.
 

cyclone3d

[H]F Junkie
Joined
Aug 16, 2004
Messages
13,505
Yeah getting more than 7 slots might be tricky as that's the sorta standard amount. Look through the SuperMicro page though, you might find something. Also would suggest going with EPYC. You get 128 lanes with 1 OR 2 cpu's, so unless you need the CPU horsepower (or additional memory capacity) a single CPU would be fine. You can also get enough lanes to run 7 lanes at x8 natively with a Threadripper system. (Threadripper has 64 lanes, 4 of which go to the chipset, and 60 available)

Out of pure curiosity, what are you guys doing with this? Sounds interesting.
I was looking stuff up and what I ran across said that you can get 256 available PCIe lanes with a dual socket EPYC system. Not that there are any boards available for purchase that makes use of that many.. but it would be sweet if there was.

Edit: Have a look at this site.. It looks like something may be available with 10x PCIE x16 slots... and that they will all run in x8 when fully populated.

This is with a dual XEON setup.
https://www.servethehome.com/deeple...0-ti-single-root-deep-learning-server-part-1/

I also saw mention somewhere else that Supermicro has the ability to make different configurations. Guessing that it wouldn't be very cost effective to have a one-off machine made though. Probably wouldn't hurt giving them a call though to see what they can do. I didn't see any pre-made boards on their site that came close to what you are looking for.
 
Last edited:

Lt.Gamer

n00b
Joined
Nov 18, 2007
Messages
9
Just to be that guy again, what cards are you looking at populating the board with? You say you need full length (16x physical) slots?

Depending on what it is if you are only interested in a 8x bus and what you are planning on doing you could get very getto with your solution and take a page out of the mining rig community and get risers to go from a 16x down to an 8x physical, it's not like you care about the additional pins anyway. I haven't seen any but I'd be surprised if they didn't exist as a spare in a Dell/HP/Supermicro/etc. somewhere. In which case you could cannibalise a board from an actual server, e.g. a Dell R720/R730/R740 which has your 7 PCI slots each running at >= 8x and gives you all the nice server features (e.g. OBM) because unlike things like the Asus workstation series it was actually designed to be a server.
 

emmjawsX

Weaksauce
Joined
Nov 21, 2015
Messages
86
You can also get enough lanes to run 7 lanes at x8 natively with a Threadripper system. (Threadripper has 64 lanes, 4 of which go to the chipset, and 60 available)
Is there a Threadripper 'board that can actually do that? All the boards seem to use 12 lanes for m2 and split the rest into 2x16 + 2x8 (or 3x16 for the taichi matx).
 

AXm77

Limp Gawd
Joined
Jun 4, 2012
Messages
154
SupermicroX10DRX - dual E5 v3/4 Xeons, 10x PCI-E x8.
 
Last edited:

cyclone3d

[H]F Junkie
Joined
Aug 16, 2004
Messages
13,505
SupermicroX10DRX - dual E5 v3/4 Xeons, 10x PCI-E x8.
There ya go.. and it doesn't use any PLX chips. But you would need to open up the end of the slots in order to fit x16 cards in there.
 

Dan_D

Extremely [H]
Joined
Feb 9, 2002
Messages
56,205
All,



I am desperately trying to find any motherboards out there that have the following specs:



  • Intel OR AMD (but preferably intel)
  • Single or dual processor
  • 7 or 8 PCI Express slots that are 16x sized. Spacing/layout of the slots does not matter.
  • Each PCIE slot should be able to run at 8x at the same time (Meaning, 64 lanes)
  • PLX solutions to the above are NOT ok. I need the speed natively.
This would be going into a server, so weird motherboard sizes are OK



Its been very hard to find this information because so many motherboard manufacturers seem to stretch the truth by using PLX or other memes.

I know the desire may be to ask "why do you need this" but please be assured that we do.



Thanks
Again, you'll need an Epyc or Xeon based system and there are some decent recommendations such as Supermicro X10DRX. I know you have no desire to explain the reason for needing this. That requirement is academic as you won't see PLX chips in use in systems like this. With Xeon and Epyc, it won't be necessary because of the number of PCIe lanes available to those platforms. The only reasons I can imagine your requirements making sense are precisely the ones that would highlight the issues created by a PLX solution, but again it's a moot point. You would only see PLX chips on motherboards that are incapable of running two PCIe x16 slots in an x16/x16 configuration. That pretty much rules out anything you would be using.

I'll simply say that using something from Supermicro is probably your best bet. When you get into specialized applications and hardware requirements like this, you can run into all sorts of weird issues building the hardware yourself. Supermicro has a proven track record in these circles and as much as I like companies like GIGABYTE and ASUS, this isn't their realm of expertise. Where they shine is in the gaming enthusiast market and that prosumer crossover realm between gaming PC's and more traditional, high powered workstations.
 

bwang

Gawd
Joined
Aug 6, 2011
Messages
979
The Supermicro 8/16 GPU systems are sometimes built with switches, presumably to be able to get "x16" on all 8 slots - a pair of Xeons only provides 80 lanes. I don't think there is a pre-built SM system without switches - they assume if you need that many physical x16 slots, you would like the additional load balancing provided by the PLX chips and don't mind the added latency (which is true of almost all compute applications).

The OP would really benefit from providing some details about the application. Does the 'no PLX' requirement mean 'we need native x8 bandwidth' or 'we need native latencies'? I can think of some applications (for example, a financial application implemented using FPGA dev boards that are x16) that would fall into the latter category, but the majority of needs would fall into the first category.
 
Top