PCIE Bifurcation

Joined
Nov 20, 2017
Messages
58
Just got the mail back from the support:

"Dear sir,
After check, it [ASRock X570D4I-2T] can support x16, x8x8, x8x4x4, x4x4x4x4. J"

Cheers!
I still want to know if the Oculink can be broken out to 4x PCIe or if it's just SATA.

Edit: Only one M.2 makes me sad
 
Last edited:

sdfewfe

Weaksauce
Joined
Dec 16, 2015
Messages
92
It should still show up in lspci, does bandwidth in the LnkCap section reduce for the root port?

Also I have good news:

My PLX x16->x8x8x8x8 board is finally in a state were I am willing to sell it. Its not perfect yet and I will make a larger one with more spacing in the next weeks.

View attachment 219006

https://peine-braun.net/shop/index.php?route=product/product&path=66&product_id=86

Please be aware that its super time consuming to test and assemble, also the PCBs and Parts are a lot more expensive compared to my bifurcation boards.
Hence the price.
How difficult would it be to make something like this but in M.2 format (x4 -> x4x4 or x4->x4x4x4x4 for instance)?
 

DaHans

n00b
Joined
May 21, 2015
Messages
38
I checked the Idea for the ncase riser and I was thinking about getting it for a Silverstone CS280 too.

That case has just two PCIe expansion slots but your idea should work in the same way as in the Ncase M1 right?

So I can add a Low profile Sata expansion card and a single Slot GPU.

ChronoDog
JokinenK
I also have news concerning the low profile x8x8 Splitter for the NCase M1:

It all seems to just barely fit together: I will make an x8x8 version first, and if it works the x8x4x4 with M.2 will follow on demand.
It will be a tad bit more expensive as the angled slots are €15 a piece without tax alone and the cradle mount slots are a pain to solder.

View attachment 197126

marcosscriven
This could also be an option four your case?!?
 

C_Payne

Limp Gawd
Joined
Jan 6, 2017
Messages
133
I checked the Idea for the ncase riser and I was thinking about getting it for a Silverstone CS280 too.
jup should work without issue.

How difficult would it be to make something like this but in M.2 format (x4 -> x4x4 or x4->x4x4x4x4 for instance)?
How do you imagine the physical layout? What outputs (2x M.2 or a cable?)

With a plx chip it will be nearly impossible, for various reasons, but there is smaller asmedia chips that could be used.
 

sdfewfe

Weaksauce
Joined
Dec 16, 2015
Messages
92
How do you imagine the physical layout? What outputs (2x M.2 or a cable?)

With a plx chip it will be nearly impossible, for various reasons, but there is smaller asmedia chips that could be used.
Either could work. I think two M.2 slots stacked vertically could work for some front M.2 slots in combination with a small enough CPU cooler. E.g. on my Gigabyte Z370N and Noctua L9i, the space directly above the front M.2 is empty. Two or four M.2 slots horizontally *might* work for some backside M.2 slots (maybe would have to use larger-than-standard mainboard standoffs). Or multiple M.2 slots in any layout to be used with riser cables.

Out of curiosity, what would the issue be with a PLX chip? Just layouting the PCB? And how are the Asmedia chips different?
 

C_Payne

Limp Gawd
Joined
Jan 6, 2017
Messages
133
Out of curiosity, what would the issue be with a PLX chip? Just layouting the PCB? And how are the Asmedia chips different?
Physical placement of all necessary components and the temperature of the PLX chip.
Need a smaller device, with less power consumption. (less power rails as well!)
 

Mosquito

n00b
Joined
Feb 9, 2020
Messages
5
Never did get the nvidia 760's working correctly. I think it's a combination of drivers maybe, and the C_Payne dual slot riser I was using. I switched to my Vega 56 Nano's and was having issues with the same setup using the riser card, but then switched to an ameri-rack riser and everything's working great now. This same ameri-rack riser didn't play any nicer with the 760's though, so maybe the 56's just don't like that riser card or something. Really strange.

Windows would lock up and freeze every minute or two most of the time. A few times right after a fresh windows install, or uninstalling all nvidia drivers I was able to run folding at home for like an hour with no issues, and use Davinci Resolve to transcode some video with out any issues, but then after that something would happen and it'd start locking up again. Just odd, but I'll go forward with the 56 Nano's and Ameri-rack as that combo seems to be working. Too bad, as the C_Payne riser card would have been nicer, as it was dual slot instead of the hackery I have to do with the single slot spaced ameri-rack. Now just need to see if I can track down a decently priced Silverstone 800w SFX PSU. The 700w version I've got would have been fine with the 760s, but I need to undervolt the 56's a fair bit to get them to play nice under load with 700w. That extra 100w would be nice to have :D
 

C_Payne

Limp Gawd
Joined
Jan 6, 2017
Messages
133
Hey mosquito!

May I ask what cable you were using in combination with which riser I sold you? What was the exact issue with my riser? You can also PM me!

Always trying to fix issues.
 

Mosquito

n00b
Joined
Feb 9, 2020
Messages
5
Hey mosquito!

May I ask what cable you were using in combination with which riser I sold you? What was the exact issue with my riser? You can also PM me!

Always trying to fix issues.
Thanks! I'm not sure I have enough posts yet to PM lol (or I'm just not seeing the options to).

It's the "PCIe Bifurcation - PCI Express bifurcated Riser x8x8 dual GPU dual Slot right" riser (bought through eBay in Dec).

I was using your riser with both a short and long Li-Heat riser cable (both were tested working with a single GPU).
For clarity, I was only using one at a time, not both, I just tried both a short and a long riser since I had both to try troubleshooting with.

With the nVidia 760's I'm not sure the issues were related to the riser, as I wasn't able to get those happy on either your riser or the ameri-rack riser.
But with both risers, I could get them to "work" and get video, and both detected by windows, it just locked up froze, and unfroze relatively frequently (same results with CentOS as well)

With the Vega 56's I didn't test as extensively, so I don't remember exactly what the scenarios were.
I think I wasn't able to get any video from either GPU when using your riser card, but was if I pulled the GPU out of the upper (closer to the motherboard) slot.
(Same BIOS settings for splitting PICe to x8 x8)
And if I switched to just x16 I'd get video out of one card, and the other card wouldn't get detected (which I would mostly expect)
Both cards work individually , and together with the ameri-rack riser.
Maybe this is a power draw issue? I'm not sure what to do to troubleshoot that, though it is slightly a PITA as the whole system needs to be watercooled, since everything has waterblocks :D

(Sorry for the long post...)
The_Modsquito_In_Win_A1_Mod_72.jpg
GTX 760s with short Li-Heat and C_Payne riser
 
Last edited:

C_Payne

Limp Gawd
Joined
Jan 6, 2017
Messages
133
Mosquito

I can ship you a replacement card, although I don't have any of the right turn ones left I think. Maybe it's an issue with the card. I don't really see why they shouldn't work.

One thing to try is to reduce link speed to gen2 or gen1, this way we can see if signal integrity is an issue. Haven't personally tested li-heat cables, but a cable that is covered is a bit suspicious.

You can contact me at risers@peine-braun.net
 

Mosquito

n00b
Joined
Feb 9, 2020
Messages
5
My understanding is that Li-Heat was the OEM for CoolerMaster, Lian-Li and Fractal Design PCIe riser cables, though I haven't checked or heard in a while if that still holds true. I've used 5 or 6 of these Li-Heat riser cables in the past without issue, and I use one of them to get the Ameri-rack riser card to work for my setup (since it's single-slot spaced), and it works in that capacity. I know ProClockers has reviewed li-heat risers favorably in the past. I might try out one of those LinkUP cables to see if that helps, as if nothing else having another cable configuration to test out in my buildout wouldn't hurt either.

Strangely, last night when I tried booting the setup again with the riser cable and the riser card, it did boot in to windows and task manager did see both cards, but that's all the further I got before I shut it down, since I was running them with no water loop. I'll have to set up another loop and give it a go sometime this week. Will fiddle with the link speed and see if that impacts anything on it as well.

I appreciate your willingness to help out
 

OFaceSIG

2[H]4U
Joined
Aug 31, 2009
Messages
2,390
I'm planning on rebuilding my FreeNAS with two NVME drives in a two or four way card in the primary slot. I hope to find a compatible motherboard relatively easy. Then run 10Gbps networking as I know 1Gbps will no longer be sufficient.
 

speedst3r

n00b
Joined
Jan 2, 2013
Messages
6
My cpaine riser arrived today, I'm using a PCIe extension cable from ADT-Link, all working well! :)
(E3C246D2I in a Node 804 case)

20200408_082856.jpg


One card installed so far, x4 connection!

Code:
$ sudo lspci -vv -s 02:00.0 | grep -e 02: -e LnkSta:
02:00.0 PCI bridge: PLX Technology, Inc. PEX 8632 32-lane, 12-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev bb) (prog-if 00 [Normal decode])
                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
 

garbanzo

n00b
Joined
Apr 3, 2020
Messages
1
I can report another success using a c_payne x8x8 riser on an ASRock B450-ITX/ac.
Any BIOS version above 3.20 didnt work with bifurcation but I got version 3.76 from asrock support which works fine and I don't need to patch my kernel to pass through one of the GPUs to a VM (There is a bug in AGESA versions <1.0.0.4 B).
Apparently there was an issue with the later AGESA versions where they do not support bifurcation on Pinnacle/Summit Ridge. I can upload this for anyone who wants it.

I originally tried using a bifurcation riser from taobao here but got no output on any card through the riser, even with only one slot populated. As far as I can tell it's a dud.
All the pcie pins are routed correctly so I can only assume it's the clock buffer or a signal integrity issue but I lack any equipment to test it further.
 

Meepcmx

n00b
Joined
Aug 20, 2019
Messages
3
Success with a MaxCloudOn 4x4x4x4x in Asrock Taichi X399 / TR2950x

It's a little more 'server' than the tiny builds here, but there's a lot to like. Given the amazing downstream slot placement flexibility, it may be of interest to some tiny build peeps.

Details on my blog, here;
https://mediaserver8.blogspot.com/2020/04/mediaserver-83-bifurcation-edition.html

DSC_0442.JPG


By the way, thanks to everyone contributing here. I learned so much, and it's really the only spot on the interweb that's talking about this stuff!
 

updawg

Gawd
Joined
Mar 20, 2010
Messages
609
Hi I have an asrock z87e-itx with the custom bios provided from a friendly forum member! I am looking to get a riser to allow me to use two cards. I have a matx rack mount case with 4 half height slots. I'd like to be able to use the remaining slots and mount it to the case either by a threaded standoff or just some double-sided velcro tape. Does any one have a recommendation on which bifurcation riser that I should purchase? Thanks in advanced!

 

C_Payne

Limp Gawd
Joined
Jan 6, 2017
Messages
133
Also I have worked on a quite a few new things! Check them out!

First of all gen4 is spreading and I have been working hard to find solutions for gen4 systems.
I have a gen4 signal booster ready. It is neither cheap nor easy to use though, but can be configured very flexible with individual gains for each channel.
While this board may not be the most useful on its own I can make custom bifurcation designs with gen4 redrivers on board now!
IMAG3885.jpg

Next are Slimline SAS 8i and Oculink Adaptors. They at least on my configuration work at gen4 speeds. (Gigabyte x570 ITX -> Adaptor -> 3M 50cm Cable -> Adaptor -> Radeon RX5700)
Very flexible custom configurations seem possible!
IMAG3896.jpg IMAG3822.jpg IMAG3830.jpg IMAG3798.jpg

Next comes the NCase M1 x8x4x4 Adaptor: Can be used like the x8x8 one, but the low profile card gets only x4 and the other x4 go to an M.2 Slot on the backside.
Please Note that the final product will have an right angled x8 Slot only, since its a pain to rip out the pins like on this prototype. Use an x8->x16 cable!
IMAG3890.jpg IMAG3891.jpg

Also I have the PLX x16->x8x8x8x8 with 3-Slot spacing.
IMAG3705.jpg IMAG3704.jpg

I will add everything to my shop soon.
Also I want everyone to be encouraged to ask for customized designs! ;) Everthing seems possible :)
This is not only related to PCIe, but all sorts of custom PCBs.
 

g_operator

n00b
Joined
Apr 30, 2020
Messages
2
Also I have worked on a quite a few new things! Check them out!

First of all gen4 is spreading and I have been working hard to find solutions for gen4 systems.
I have a gen4 signal booster ready. It is neither cheap nor easy to use though, but can be configured very flexible with individual gains for each channel.
While this board may not be the most useful on its own I can make custom bifurcation designs with gen4 redrivers on board now!
View attachment 241534

Next are Slimline SAS 8i and Oculink Adaptors. They at least on my configuration work at gen4 speeds. (Gigabyte x570 ITX -> Adaptor -> 3M 50cm Cable -> Adaptor -> Radeon RX5700)
Very flexible custom configurations seem possible!
View attachment 241535View attachment 241537View attachment 241536View attachment 241538

Next comes the NCase M1 x8x4x4 Adaptor: Can be used like the x8x8 one, but the low profile card gets only x4 and the other x4 go to an M.2 Slot on the backside.
Please Note that the final product will have an right angled x8 Slot only, since its a pain to rip out the pins like on this prototype. Use an x8->x16 cable!
View attachment 241540View attachment 241539

Also I have the PLX x16->x8x8x8x8 with 3-Slot spacing.
View attachment 241544View attachment 241546

I will add everything to my shop soon.
Also I want everyone to be encouraged to ask for customized designs! ;) Everthing seems possible :)
This is not only related to PCIe, but all sorts of custom PCBs.
nice work C_Payne and very interesting thread in general

I am currently setting up a home nas on my own using my old asrock z87e itx, to get my 970pro nvmes up and running. So, I got in contact with ameri-rack and asrock, who haven't replied so far.

Furthermore I followed the bifurcation guide from the other forum, however there is no such IIO section (where you can choose x4x4) in any of the available bios roms from asrock through AMIBCP.
Then I saw you post updawg, which gave me hope. How did you get the bios 2.50B version to enable bifurcation?

Also, if I get the PCI E bifurcation enabled, would the ASROCK Quad M.2 (https://www.asrock.com/mb/spec/product.asp?Model=ULTRA QUAD M.2 CARD) work?
Meaning, do I have enough pci-e lanes? I am happy to try and let you know.

Many thanks in advance.
 
Joined
Nov 20, 2017
Messages
58
Next are Slimline SAS 8i and Oculink Adaptors. They at least on my configuration work at gen4 speeds. (Gigabyte x570 ITX -> Adaptor -> 3M 50cm Cable -> Adaptor -> Radeon RX5700)
Very flexible custom configurations seem possible!
View attachment 241535View attachment 241537View attachment 241536View attachment 241538
Not that I can/should afford it now, but this basically means you have a working oculink -> 4x adapter. So theoretically I could get that Asrock x570 server board and break to oculink out to a 4x slot like I've dreamed? Again, not that I should, but maybe after Covid stuff settles a little bit, how much would a oculink -> 4x adapter be in a side form factor like this be? (https://ae01.alicdn.com/kf/H3ea259b500f0466cb38176a9f6fa3e88X.jpg Just the right side, where the slot is inline with the PCB?)
 

sdfewfe

Weaksauce
Joined
Dec 16, 2015
Messages
92
Oh wow, this is amazing! I don't have any use case personally for any of your current adapters, but I am absolutely fascinated by your work.

Just out of curiosity, I've made a few basic PCBs myself for hobby projects, at the level of Arduino shields with I2C, SPI, and similar stuff. How much harder than that is high-frequency design with PCIe? Is it something that one could feasibly learn as an ambitious hobbyist, or should I expect to need at least an undergrad EE degree to even think about tinkering with something like this?

Still itching for an M.2 slot multiplicator solution... ;)
 

C_Payne

Limp Gawd
Joined
Jan 6, 2017
Messages
133
Also, if I get the PCI E bifurcation enabled, would the ASROCK Quad M.2 (https://www.asrock.com/mb/spec/product.asp?Model=ULTRA QUAD M.2 CARD) work?
Meaning, do I have enough pci-e lanes? I am happy to try and let you know.
Intel mainstream can only support x8x4x4, which means you could only use 3, but that should work.

So theoretically I could get that Asrock x570 server board and break to oculink out to a 4x slot like I've dreamed?
Sure.
I have an oculink x4 to slot adaptor. Check picture, but it's an early development prototype and has one signal missing (a cable present signal) so it may not work.
A new design is €200 as with most of my custom PCIe boards due to PCB manufacturing cost.
IMG-20190813-WA0008.jpeg

How much harder than that is high-frequency design with PCIe?
Well I have read quite a lot, and there is lots of caveats involved. It's possible for the ambitious hobbyist like I consider myself, but involves many, many hours of learning, and some very frustrating moments.
 

g_operator

n00b
Joined
Apr 30, 2020
Messages
2
Intel mainstream can only support x8x4x4, which means you could only use 3, but that should work.



Sure.
I have an oculink x4 to slot adaptor. Check picture, but it's an early development prototype and has one signal missing (a cable present signal) so it may not work.
A new design is €200 as with most of my custom PCIe boards due to PCB manufacturing cost.
View attachment 242302


Well I have read quite a lot, and there is lots of caveats involved. It's possible for the ambitious hobbyist like I consider myself, but involves many, many hours of learning, and some very frustrating moments.
Thanks C_Payne, if I get the bifurcation enabled, I will give it a try!
 

speedst3r

n00b
Joined
Jan 2, 2013
Messages
6
Added a 1660Ti to my server (Plex transcoding/Folding@Home). Working a treat.

Code:
$ sudo lspci -tv
-[0000:00]-+-00.0  Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers
           +-01.0-[01]--+-00.0  NVIDIA Corporation Device 2182
           |            +-00.1  NVIDIA Corporation Device 1aeb
           |            +-00.2  NVIDIA Corporation Device 1aec
           |            \-00.3  NVIDIA Corporation Device 1aed
           +-01.2-[02-05]----00.0-[03-05]--+-04.0-[04]----00.0  LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]
           |                               \-05.0-[05]----00.0  LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]
Devices 0 and 2 on bus 1 are on the bifurcated x16 slot.

Code:
$ for slot in 01:00.0 02:00.0; do sudo lspci -vvs $slot | grep -e :00\.0  -e LnkSta: ; done
01:00.0 VGA compatible controller: NVIDIA Corporation Device 2182 (rev a1) (prog-if 00 [VGA controller])
                LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
02:00.0 PCI bridge: PLX Technology, Inc. PEX 8632 32-lane, 12-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev bb) (prog-if 00 [Normal decode])
                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
 

sdfewfe

Weaksauce
Joined
Dec 16, 2015
Messages
92
Well I have read quite a lot, and there is lots of caveats involved. It's possible for the ambitious hobbyist like I consider myself, but involves many, many hours of learning, and some very frustrating moments.
Do you have any pointers for where to begin? Purely out of curiosity about how it all works, at this point.
 

AP2

[H]ard|Gawd
Joined
Jun 25, 2007
Messages
1,456
so ive finally decided to give bifurcation a try and have the cpayne card already. just have a few questions hoping someone can help answer.

is there a performance hit if I split the lanes and run 2 2080tis with nvlink?
what riser cable is works with the cpayne cards?
does pcie bifurcation work on the Asus x570 impact board (mini dtx)?
 
Last edited:

dffvb

n00b
Joined
Jun 1, 2020
Messages
1
Hello everybody, I am looking for a splitter, and I am not sure, whether it does not exist, or if I havent looked well enough. What I am searching for is like a T-Adapter for ITX, which gives me two half hight slots. Meaning: for example a Node 304 I put the adapter in and could put in a single slot GPU and a single slot 10G network card, both half hight obiviously. Does this exist? Thanks for a short answer :)
 

C_Payne

Limp Gawd
Joined
Jan 6, 2017
Messages
133
Hello everybody, I am looking for a splitter, and I am not sure, whether it does not exist, or if I havent looked well enough. What I am searching for is like a T-Adapter for ITX, which gives me two half hight slots. Meaning: for example a Node 304 I put the adapter in and could put in a single slot GPU and a single slot 10G network card, both half hight obiviously. Does this exist? Thanks for a short answer :)
You can use this one in combinatiin with a short cable for the second card.
https://peine-braun.net/shop/index.php?route=product/product&path=65_59&product_id=77

I also have an x8x4x4 version coming up with an added M.2 Slot on the back.
 
  • Like
Reactions: dffvb
like this
Joined
Nov 20, 2017
Messages
58
C_Payne you may want to make a note on your "x8x8 left turn 1-Slot rev10" riser that the lower slot will be viewed as the primary. Even though my Gigabyte x570 itx motherboard has the setting to choose which is the primary GPU it only offers "Slot 1" even when bifurcation is enabled. (I have opened a ticket with Gigabyte support).

I am running/planning to run linux with GPU pass thru, where my primary (edit: linux/host) card is in the upper slot (watercooled single slot). It's a little annoying at the moment that my bios boots off the GPU that isn't connected to a real monitor. But your riser is awesome and much appreciated, nice of you to add the sata plug, although my version doesn't have that.
 
Joined
Nov 20, 2017
Messages
58
so ive finally decided to give bifurcation a try and have the cpayne card already. just have a few questions hoping someone can help answer.

is there a performance hit if I split the lanes and run 2 2080tis with nvlink?
what riser cable is works with the cpayne cards?
does pcie bifurcation work on the Asus x570 impact board (mini dtx)?
From everything I can tell online.... No, the asus impact board does not. And historically Asus refuses to support bifurcation on their itx (and now dtx) boards. When asked on social media or at conventions they have either given me the run around or flat out ignore the question. Asus may have good quality hardware, but their warranty and customer support have lost me.

Also itx boards don't have SLI certification, so to get SLI to work you'll have to get your modding hat out and do some research online, because it will not work out of the box.

Asrock and Gigabyte seem to regularly support bifurcation.
 

C_Payne

Limp Gawd
Joined
Jan 6, 2017
Messages
133
C_Payne you may want to make a note on your "x8x8 left turn 1-Slot rev10" riser that the lower slot will be viewed as the primary. Even though my Gigabyte x570 itx motherboard has the setting to choose which is the primary GPU it only offers "Slot 1" even when bifurcation is enabled. (I have opened a ticket with Gigabyte support).

I am running/planning to run linux with GPU pass thru, where my primary (edit: linux/host) card is in the upper slot (watercooled single slot). It's a little annoying at the moment that my bios boots off the GPU that isn't connected to a real monitor. But your riser is awesome and much appreciated, nice of you to add the sata plug, although my version doesn't have that.
Try disabling CSM, UEFI only did fix this for me. I had outputs on all connected GPUs, and some BIOSes well even let you chose an exact card and output as primary.
 
Joined
Nov 20, 2017
Messages
58
Try disabling CSM, UEFI only did fix this for me. I had outputs on all connected GPUs, and some BIOSes well even let you chose an exact card and output as primary.
Let me rephrase this, I have to have a dummy plug plugged into that card, therefore the bios sees it as a gpu with a display. The bios setting to select the card, just offers "Slot 1" there is nothing to change. However I will try those other settings though, when I refill the loop this weekend. What motherboards did you have success with?
 
Top