Are there any X570 mobos that supports 16x PCIe 4.0 (GPU) + 4x PCIe 4.0 (NVMe) + 8x PCIe 3.0 (50Gbps NIC)?

pclausen

Gawd
Joined
Jan 30, 2008
Messages
697
My understanding is that the AM4 socket is limited to the following:

1 16x PCIe 4.0 direct wired to AM4 (typically main PCIe 16x slot for GPU or split to 8x + 8x or 8x + 4x + 4x across 2 or 3 PCIe slots)
1 4x PCIe 4.0 direct wired to AM4 (typically M.2 slot for NVMe storage device)
1 4x PCIe 4.0 PCH (LAN, USB, SATA, etc)

I have the following that I want to plug into a X570 mobo:

RTX 3080 / 6900 XT (depending on which I can get my hands on first)
Mellanox MCX4121A dual 25 Gpbs NIC (8x PCIe 3.0)
Sabrent Rocket Q4 NVMe 4.0

I'm thinking that the best compromise would be to run those 3 devices as either:

GPU 8x PCIe 4.0 in PCIe 16x slot 1
NIC 8x PCIe 3.0 in PCIe 16x slot 2 (or 3)
NVMe 4x PCIe 4.0 in M.2 slot

Or: (not sure this will work since that effectively leaves no PCIe lanes for the PCH chipset)

GPU 16x PCIe 4.0 in PCIe 16x slot 1
NIC 4x PCIe 3.0 in PCIe 16x slot 2 (or 3)
NVMe 4x PCIe 4.0 in M.2 slot

Granted 4.0 is effectively twice as fast as 3.0, so maybe I won't take a performance hit by running the GPU @ 8x (assuming a PCIe 4.0 capable GPU of course).

PCIe 3.0 x 4 = 31.52 Gbps, so as long as I run just one port on the NIC, I'll be fine, but if I want to LAG them both together for 50 Gbps, that becomes a limiting factor.

Do most X570 mobo support either of these configs? (I'm looking mainly at the Asus ROG/TUF offerings without WiFi)

I'm coming from Intel where I have been spoiled with 40 PCIe 3.0 lanes for about as far back as I can remember, so being limited to 24 lanes will take some getting used to. :)

Thanks!
 
Pretty much every X570 board is setup that way. If you use the first 2 PCIe slots you would limit them to 2 8x PCIe 4.0 slots. The CPU feeds those and the first m.2 slot all at PCIe 4.0. There are 24 lanes on the CPU and the remaining 4 lanes go to the chipset at PCIe 4 speed. Most boards are setup so that you can get the other slots, not the first 2 connected to the CPU at PCIe 4.0 through the chipset. So you can pop that NIC in a 4x slot and get proper PCIe 4.0 speeds through the chipset.
 
All boards X570 boards are:
24 lanes from CPU. 4 for NVMe (AMD requirement), and 4 for southbridge (platform requirement). This leaves 16 from the CPU, and a bunch more breakouts from the chipset. The cheaper boards, like my X570 TUF, put all 16 of the remaining CPU lanes into the main PCIe slot. The other big PCIe slot is given 4 lanes from the chipset (meaning, the CPU has 4 lanes to the chipset, and the chipset has a hub/switch that has ~20 lanes, of which 4 are allocated to that other PCIe slot).

Some of the pricier boards will instead split that main 16 lane allocation from the CPU, into two x8 slots (using switch chips, so the main slot is x16/x8, and the other slot is x8).

IIRC, in my original research around the X570 launch, anything higher than a ASUS X570 TUF or ASRock Steel Legend/Extreme4 in pricing should have the latter layout. I'm somewhat interested, too, so I'll be back later with more findings.
 
Alright, I found the board I was thinking of. The ASUS Pro WS X570 ACE. ~$350+, so not worth it, outside of the configuration curiosity:

3 x16 slots. First two share the x16 from the CPU (so if only one slot is used, the first slot will get x16. If two slots are used, they each get x8). 3rd slot gets x8 from the chipset (obviously, chipset only has x4 to the CPU, but at PCIe 4.0 speeds, it's still more than enough to cover the 50GbE NIC, with enough headroom to spare for the onboard LAN, SATA, USB [though some USB ports are driven directly from the CPU and don't use the southbridge's bandwidth], etc).

Of course, for your utilization, the best board would be one that only has x16 from the CPU to the main slot, and x8 from the chipset to the second slot, but such a board doesn't exist, AFAIK.
 
Yep, I did not find a X570 with x8 to the chipset either.

I was looking at the ROG Crosshair VIII Formula and found this chart:

FormularX570PCIe.PNG


Unfortunately the ASUS manual does not include a block diagram, but from the above table, it would appear that one would run dual x8 cards and also dual NVMe's where the 2nd one would have to pass through the PCH.

I found this which I assume is what all X570 motherboards adheres to.

FormularX570PCIe2.PNG


For my use case, I think the most optimal would be to run the GPU in slot 1 and the NIC in slot 2 to give each x8 lanes directly to the CPU. I could then run dual NVMe's where one would go directly to the CPU and the other one would go through the chipset.

My NIC supports RDMA which I have already deployed between a couple of servers on my network which provides amazingly fast access to data without the CPU having to do any heavy lifting. I have large spinner arrays (48 drive RAID 60) as well as NVMe striped volumes using 4 of those Sabrent Rocket Q4 NVMe 4.0 drives. I want to extend access to that data from my new X570 build, hopefully using RDMA, but I'm not sure if that is supported on a non server OS like Win10, but it will be fun to see how fast I can go.
 
Yep, I did not find a X570 with x8 to the chipset either.

I was looking at the ROG Crosshair VIII Formula and found this chart:

View attachment 292133

Unfortunately the ASUS manual does not include a block diagram, but from the above table, it would appear that one would run dual x8 cards and also dual NVMe's where the 2nd one would have to pass through the PCH.

I found this which I assume is what all X570 motherboards adheres to.

View attachment 292134

For my use case, I think the most optimal would be to run the GPU in slot 1 and the NIC in slot 2 to give each x8 lanes directly to the CPU. I could then run dual NVMe's where one would go directly to the CPU and the other one would go through the chipset.

My NIC supports RDMA which I have already deployed between a couple of servers on my network which provides amazingly fast access to data without the CPU having to do any heavy lifting. I have large spinner arrays (48 drive RAID 60) as well as NVMe striped volumes using 4 of those Sabrent Rocket Q4 NVMe 4.0 drives. I want to extend access to that data from my new X570 build, hopefully using RDMA, but I'm not sure if that is supported on a non server OS like Win10, but it will be fun to see how fast I can go.
Ever figure this out? Even though the chipset is only x4.. it's pcie 4.0, so it's the same speed as pcie 3.0 x8. If you have a pcie 3.0 x8 connection to the chipset you won't be losing speed (unless it's overloaded with other things). That said, your plan to just use x8/x8 split is probably the safest and most easily supported. If you get a pcie 4.0 GPU you won't really notice the difference between x8 and x16.
 
This lack of cpu lanes was true back with the first gen Ryzen Vs Threadripper with GEN3 lanes. I got the TR 8 core 1900X and skipped the 8 core 1700X. 96GB of quad channel 3200C14, six GEN4 NVme drives on two Asus quad M.2 cards, two M.2drives on a DIMM2 card, one gen3 X16 gpu, and a 10GB lan card all running on GEN3 cpu lanes. The new TR has 72 GEN4 cpu lanes. In a few months we should see ZEN3 TR. Build a Formula 1 work station.
 
Alright, I found the board I was thinking of. The ASUS Pro WS X570 ACE. ~$350+, so not worth it, outside of the configuration curiosity:

3 x16 slots. First two share the x16 from the CPU (so if only one slot is used, the first slot will get x16. If two slots are used, they each get x8). 3rd slot gets x8 from the chipset (obviously, chipset only has x4 to the CPU, but at PCIe 4.0 speeds, it's still more than enough to cover the 50GbE NIC, with enough headroom to spare for the onboard LAN, SATA, USB [though some USB ports are driven directly from the CPU and don't use the southbridge's bandwidth], etc).

Of course, for your utilization, the best board would be one that only has x16 from the CPU to the main slot, and x8 from the chipset to the second slot, but such a board doesn't exist, AFAIK.
48 cpu lanes to the PCIe sockets is called Threadripper.
 
48 cpu lanes to the PCIe sockets is called Threadripper.
Definitely. It costs quite a bit more, however. I do wish the old 8 and 16 core Threadrippers were still around, but I understand that AMD isn't going for that, anymore.
 
I finally scored a 3090 yesterday, so I'll now be able to continue this build (in sig). With PCIe 4.0 GPU and NVMe, I suspect that being limited to 28 lanes won't bottleneck me. I'll post my findings once I get everything up and running.
 
Definitely. It costs quite a bit more, however. I do wish the old 8 and 16 core Threadrippers were still around, but I understand that AMD isn't going for that, anymore.
They still make the 2950X for that reason. Or so they say.
 
Apologize for the thread necro but I thought I'd chime in on this topic because I was concerned about the express lane configs on X570 initially as well. Long story short I wanted to add more fast storage so I added another nvme to my array and figured whatever the result of running the 6900xt at 4X I would just live with it. Imagine my surprise at seeing gaming and benchmark performance suffer no measurable penalties. Granted nothing runs like a fresh install but I can't find any negatives to doing so.
 
That won’t take anything from the GPU; it’s sharing the DMI link back to the CPU via the chipset.
 
The graphics card runs at 4x now not 8x. I believe the OP was concerned with limiting the cards bus width to 8 lanes.
 
The graphics card runs at 4x now not 8x. I believe the OP was concerned with limiting the cards bus width to 8 lanes.
That shouldn’t be. You should have x16 or x8/x8 for the first two slots, and a single direct connected x4 NVMe slot. All of those are direct attached to the CPU. No motherboard I’m aware of splits them farther than that for the lanes from the CPU- the remaining slots all come from the chipset (which has its own lanes). So unless you moved your GPU to the last slot in the system, it’s still getting x8 at worst- you’re just putting more on the chipset and it’s link back to the CPU

Check CPU-Z and the like- they’ll show it at x8, unless you have some extremely oddball board or one with a PLX chip.

What board is it, and what all have you installed? We’ll look at the block diagram.
 
Yeah. The Meg ace is pretty normal; no PLX chips or anything odd. First slot is either x16 or x8/x8, and the first NVMe drive pulls from the CPU. The third slot and all additional NVMe slots are tied to the chipset. So you didn’t touch the GPU, you just added more to the chipset links. (Zen has 24 lanes, 4 for the first NVMe, 4 for the chipset, the rest for the first slot (or two)). You’re sharing that x4 from the chipset back to the CPU, which is fine for almost everything. :)
 
Oh I see. That’s PCIE 4.0 @x16 4.0 (meaning you haven’t downclocked it to 3.0 for something like a riser cable) for GPU-Z. It’s confusing but you’re at x16.
 
Back
Top