In search for a motherboard that can do 8 by x8 lanes pcie

Discussion in 'Motherboards' started by bigdano, Feb 2, 2019.

  1. bigdano

    bigdano n00b

    Messages:
    28
    Joined:
    Jun 14, 2014
    All,



    I am desperately trying to find any motherboards out there that have the following specs:



    • Intel OR AMD (but preferably intel)
    • Single or dual processor
    • 7 or 8 PCI Express slots that are 16x sized. Spacing/layout of the slots does not matter.
    • Each PCIE slot should be able to run at 8x at the same time (Meaning, 64 lanes)
    • PLX solutions to the above are NOT ok. I need the speed natively.
    This would be going into a server, so weird motherboard sizes are OK



    Its been very hard to find this information because so many motherboard manufacturers seem to stretch the truth by using PLX or other memes.

    I know the desire may be to ask "why do you need this" but please be assured that we do.



    Thanks
     
    Last edited: Feb 2, 2019
  2. cyclone3d

    cyclone3d [H]ardForum Junkie

    Messages:
    12,844
    Joined:
    Aug 16, 2004
    If Intel, you would need a dual socket system.

    AMD EPYC supports up to 128 PCIe lanes per socket.

    I'm not finding anything with more than 7 PCIe slots.. but here it is anyway - you would have to open up the end of the x8 slots if you want to use x16 cards in them.. but it would work.
    https://b2b.gigabyte.com/Server-Motherboard/MZ31-AR0-rev-1x#ov
     
  3. extide

    extide 2[H]4U

    Messages:
    3,265
    Joined:
    Dec 19, 2008
    Yeah getting more than 7 slots might be tricky as that's the sorta standard amount. Look through the SuperMicro page though, you might find something. Also would suggest going with EPYC. You get 128 lanes with 1 OR 2 cpu's, so unless you need the CPU horsepower (or additional memory capacity) a single CPU would be fine. You can also get enough lanes to run 7 lanes at x8 natively with a Threadripper system. (Threadripper has 64 lanes, 4 of which go to the chipset, and 60 available)

    Out of pure curiosity, what are you guys doing with this? Sounds interesting.
     
  4. cyclone3d

    cyclone3d [H]ardForum Junkie

    Messages:
    12,844
    Joined:
    Aug 16, 2004
    I was looking stuff up and what I ran across said that you can get 256 available PCIe lanes with a dual socket EPYC system. Not that there are any boards available for purchase that makes use of that many.. but it would be sweet if there was.

    Edit: Have a look at this site.. It looks like something may be available with 10x PCIE x16 slots... and that they will all run in x8 when fully populated.

    This is with a dual XEON setup.
    https://www.servethehome.com/deeple...0-ti-single-root-deep-learning-server-part-1/

    I also saw mention somewhere else that Supermicro has the ability to make different configurations. Guessing that it wouldn't be very cost effective to have a one-off machine made though. Probably wouldn't hurt giving them a call though to see what they can do. I didn't see any pre-made boards on their site that came close to what you are looking for.
     
    Last edited: Feb 3, 2019
  5. Lt.Gamer

    Lt.Gamer n00b

    Messages:
    9
    Joined:
    Nov 18, 2007
    Just to be that guy again, what cards are you looking at populating the board with? You say you need full length (16x physical) slots?

    Depending on what it is if you are only interested in a 8x bus and what you are planning on doing you could get very getto with your solution and take a page out of the mining rig community and get risers to go from a 16x down to an 8x physical, it's not like you care about the additional pins anyway. I haven't seen any but I'd be surprised if they didn't exist as a spare in a Dell/HP/Supermicro/etc. somewhere. In which case you could cannibalise a board from an actual server, e.g. a Dell R720/R730/R740 which has your 7 PCI slots each running at >= 8x and gives you all the nice server features (e.g. OBM) because unlike things like the Asus workstation series it was actually designed to be a server.
     
  6. emmjawsX

    emmjawsX [H]Lite

    Messages:
    86
    Joined:
    Nov 21, 2015
    Is there a Threadripper 'board that can actually do that? All the boards seem to use 12 lanes for m2 and split the rest into 2x16 + 2x8 (or 3x16 for the taichi matx).
     
  7. AXm77

    AXm77 Limp Gawd

    Messages:
    128
    Joined:
    Jun 4, 2012
    SupermicroX10DRX - dual E5 v3/4 Xeons, 10x PCI-E x8.
     
    Last edited: Feb 3, 2019
  8. cyclone3d

    cyclone3d [H]ardForum Junkie

    Messages:
    12,844
    Joined:
    Aug 16, 2004
    There ya go.. and it doesn't use any PLX chips. But you would need to open up the end of the slots in order to fit x16 cards in there.
     
  9. Dan_D

    Dan_D [H]ardOCP Motherboard Editor

    Messages:
    53,125
    Joined:
    Feb 9, 2002
    Again, you'll need an Epyc or Xeon based system and there are some decent recommendations such as Supermicro X10DRX. I know you have no desire to explain the reason for needing this. That requirement is academic as you won't see PLX chips in use in systems like this. With Xeon and Epyc, it won't be necessary because of the number of PCIe lanes available to those platforms. The only reasons I can imagine your requirements making sense are precisely the ones that would highlight the issues created by a PLX solution, but again it's a moot point. You would only see PLX chips on motherboards that are incapable of running two PCIe x16 slots in an x16/x16 configuration. That pretty much rules out anything you would be using.

    I'll simply say that using something from Supermicro is probably your best bet. When you get into specialized applications and hardware requirements like this, you can run into all sorts of weird issues building the hardware yourself. Supermicro has a proven track record in these circles and as much as I like companies like GIGABYTE and ASUS, this isn't their realm of expertise. Where they shine is in the gaming enthusiast market and that prosumer crossover realm between gaming PC's and more traditional, high powered workstations.
     
    extide likes this.
  10. bwang

    bwang Gawd

    Messages:
    961
    Joined:
    Aug 6, 2011
    The Supermicro 8/16 GPU systems are sometimes built with switches, presumably to be able to get "x16" on all 8 slots - a pair of Xeons only provides 80 lanes. I don't think there is a pre-built SM system without switches - they assume if you need that many physical x16 slots, you would like the additional load balancing provided by the PLX chips and don't mind the added latency (which is true of almost all compute applications).

    The OP would really benefit from providing some details about the application. Does the 'no PLX' requirement mean 'we need native x8 bandwidth' or 'we need native latencies'? I can think of some applications (for example, a financial application implemented using FPGA dev boards that are x16) that would fall into the latter category, but the majority of needs would fall into the first category.