PCIE Bifurcation

Discussion in 'Small Form Factor Systems' started by jb1, Jul 28, 2015.

  1. C_Payne

    C_Payne [H]Lite

    Messages:
    119
    Joined:
    Jan 6, 2017
    I have also in the past weeks developed some new Risers.

    I can provide any combination of x8, x4 and M.2, custom dimensions and mounting holes etc.

    This for example is meant to extend a mini ITX Board to fit into an mATX case with one x8 GPU and two additional x4 slots.


    ITX.jpg

    IMAG1846.jpg


    This is all the PCBs I have around:

    IMAG1935.jpg

    I am working on a small webshop to sell them in the future.

    Also Asrock has been nice enough to enable x4x4x4x4 on their x99e itx/ac if anyone needs that bios please let me know.
     
  2. Neapolitan6th

    Neapolitan6th [H]ard|Gawd

    Messages:
    1,059
    Joined:
    Nov 18, 2016
    That's awesome. I enjoyed watching everyone in the comments on that one freak out because they'd never heard of PCIE bifurcation before.

    I'm curious, with PCIE 4.0 on the somewhat near horizon, do you think your crop of bifurcated risers will be forward compatible with the new standard?

    2 x8 4.0 lanes is quite a lot of bandwidth to play with.
     
  3. C_Payne

    C_Payne [H]Lite

    Messages:
    119
    Joined:
    Jan 6, 2017
    Well I cant be sure, but I do think so.
    The Clock-Buffer I choose is gen 4 compliant already.

    However:
    PCIe gen4 connectors are improved versions and only SMT.
    However:
    I have followed good practice with my Design. I read this paper, and seem to comply to most of these standards already. http://www.oldfriend.url.tw/article/IEEE_paper/SI/Intel_PCIe connector 16G_DesignCon2016_p89~.pdf
    However:
    I routed for an impedance Target of 100Ohms, it should be 85 now.
    However:
    In typical ITX build trace length is short.
    However:
    PCIe gen4 per spec only allows one connector between endpoints.
    However:
    PCIe gen3 allows for two connectors between endpoints yet this works without impact on bandwidth (I have tested bandwidth with an nice Nvidia Cuda Porgram)
    These are 7 Connectors between endpoints. A lot of users have actually been stacking more then one riser without issues.
    IMAG2062.jpg

    I will find out and certainly make a redesign should they not work.
    I dont think one would really need to use retimers as it would make them more expensive by an order of magnitude.

    But I do think that the common setup with flexible risers may not work anymore.
     
    Neapolitan6th likes this.
  4. chemist_slime

    chemist_slime [H]ard|Gawd

    Messages:
    1,100
    Joined:
    Jul 24, 2005
    Would love to get the x4x4x4x4 bios for the asrock x99e-itx/ac. Also, do you have an x4x4x4x4 riser? Thanks.
     
  5. C_Payne

    C_Payne [H]Lite

    Messages:
    119
    Joined:
    Jan 6, 2017
    I currently have no x4x4x4x4 PCBs on hand, but I could make one to your spec. Just drop an email risers@peine-braun.net, I will send you that bios.
     
  6. TristanDuboisOLG

    TristanDuboisOLG n00b

    Messages:
    2
    Joined:
    Nov 19, 2018
    I was wondering, I found this bifurcation chat some time ago while researching. I'm having some trouble getting my bifurcated setup working, normally I know how to troubleshoot stuff, but this is getting into the realm of "never even heard of that" for most people.

    My setup so far:
    Asrock x99e-ITX a/c (With modded bios from Asrock Eng dept)
    Asus Hyper M.2 x16
    1x Samsung 960 Pro NVME (Boot)
    4x Intel 660p 512GB

    I've got windows server loaded onto the boot drive, gone through the nightmare that is LAN drivers for this board on Windows Server (not provided and cannot install), even managed to get the wireless card working. Now I'm having issues with getting my Hyper M.2 working. Sometimes it boots and recognizes it, other times it doesn't. I've heard other people mention that when they have issues with their Hyper m.2 cards and they don't recognize drives the LED lights on the side will turn off. So if you have 1 m.2 connected, one light will be on, 2 is two lights and so on and so forth. My device (when it powers up) has all 4 lights on but only shows one 512gb drive in disk management. Is there any way to troubleshoot this?

    I have emails from ASUS where they explain to me that I was correct in some earlier inquiries and that the Hyper M.2 will work with any motherboard that has x4x4x4x4 bifurcation enabled AND I have the modded Bios from Asrock that enables x4x4x4x4 bifurcation. They mention that x8x8 works correctly, I'm going to try to change over the bios to x8x8 tonight in order to see if that enables a second drive. I'd love any other suggestions that anyone might have.


    . Asrock Support.JPG Asus Support.JPG
     
  7. C_Payne

    C_Payne [H]Lite

    Messages:
    119
    Joined:
    Jan 6, 2017
    Asrock x99e Bifurcation BIOS including x4x4x4x4:
    https://peine-braun.net/public_files/Bifurcation_BIOS/X99EITX3.81A
    I hope asrock won't mind me sharing this with you publicly.

    Also I am currently writing a summary on PCIe Bifurcation, so if you have a working Bifurcation Setup please drop an eMail to risers@peine-braun.net or PM with your mainboard, bios revision or mod-bios and tested modes (e.g x8x8, x8x4x4).
    That would be great.
     
    chemist_slime and Neapolitan6th like this.
  8. Smitty2k1

    Smitty2k1 Gawd

    Messages:
    650
    Joined:
    May 31, 2005
    This is all seriously [H] level stuff. Good job y'all.
     
    dragonstongue likes this.
  9. rekrek

    rekrek n00b

    Messages:
    1
    Joined:
    Jan 24, 2019
    Hi C_Payne, I am really interested in the card that "extend a mini ITX Board to fit into an mATX case".
    I hope you will be able to put some up for sale. I would recommend selling on ebay as it will do all international shipments.
    What price can you expect to sell them ?

    Would use it in a NAS to add additional ports (ethernet and 10gbe) to use as a cheap and energy efficient router with integrated switch (with 10gbe !). Thanks
     
  10. C_Payne

    C_Payne [H]Lite

    Messages:
    119
    Joined:
    Jan 6, 2017
    I have been working on my Online shop and the PCIe bifurcation summary I have been talking about:
    It supports Paypal Express Checkout.

    https://peine-braun.net/shop/

    There is also some information I have collected concering bifurcation there, this is however work in progress. Any additions would be greatly appreciated, please drop a pm or eMail.

    https://peine-braun.net/shop/index.php?route=information/information&information_id=7

    You can still always drop me an eMail
    risers@peine-braun.net
     
  11. Prodeous

    Prodeous n00b

    Messages:
    1
    Joined:
    Jan 25, 2019
    I have a general question on the Bifurcation. If you split a slot to 4x4x4x4, and plug in a 8x pcie device. Would the device use a single 4x slice, or would it recombine the two 4x ?

    I have Gigabyte Designare x399 board, and all it can do is split slots to multiple of 4x in BIOS. And I would prefer 8x splits.
     
  12. C_Payne

    C_Payne [H]Lite

    Messages:
    119
    Joined:
    Jan 6, 2017
    Unfortuantely that will not work, maybe it wont even POST.

    However it will probably be easy to mod that BIOS to support x8x8 as well...

    EDIT: ohh its x399... not sure about AMD BIOS modding

    EDIT2: checked the bios briefly, dont think it can be done easily with modding tools
    However Gigabyte may add it, just write a support request to them. Shouldnt be to much work for them.
    Asrock has in the past replied positively to requests like this
     
    Last edited: Jan 28, 2019
    Prodeous likes this.
  13. superray

    superray n00b

    Messages:
    16
    Joined:
    Jun 12, 2016
    Although I'm not interested SLI, it's useful to set up multiple pci-e ssds this way
     
    Neapolitan6th likes this.
  14. AmusedGoose

    AmusedGoose n00b

    Messages:
    5
    Joined:
    Feb 25, 2019
    Wouldn't an active riser with PLX chip always be better than using bifurcation? (besides higher cost that is).
    I'm interested in using for example RSC-R2UG-A2E16-A as earlier mentioned here, for compute purposes where the bandwidth between the GPU's is an important factor.

    Would you recommend using the PLX riser you tried all the way back in the beginning of this thread?
     
  15. WiKAi

    WiKAi n00b

    Messages:
    1
    Joined:
    Feb 26, 2019
    Looks like the RSC-R2UG-A2E16-A would actually fit in a HP MicroServer Gen8, with riser cables for the cards of course.
    Would be pretty awesome to cram a 10GbE NIC in it a long with an HBA card for more drives :D

    The ×16 slot on it only runs in ×8 mode at most though, does anyone know if the RSC-R2UG-A2E16-A would still work in an ×8 mode slot?
     
  16. C_Payne

    C_Payne [H]Lite

    Messages:
    119
    Joined:
    Jan 6, 2017
    Should not be an issue.
     
  17. jhsu

    jhsu n00b

    Messages:
    25
    Joined:
    Sep 30, 2018
    I have an Asrock x399 Taichi with a Threadripper 1950x and I'm interested in trying to run 2 gpu's at x8/x8 on 1 x16 pcie slot.

    Currently the bios only has an option to split the x16 slot into x4x4x4x4 mode. I'm actually trying to put in a feature request with them to support x8x8 mode on the next bios update.

    Bios features aside, if x8x8 were supported, would using one of your splitters be all I would need to run 2 gpu's on 1 slot? Or is there some other system settings/controller chip trickery that would also need to happen?
     
  18. AmusedGoose

    AmusedGoose n00b

    Messages:
    5
    Joined:
    Feb 25, 2019
    I thought x8x8 was supported since BIOS 2.00c, what bios are you on? Can you go back to 2.00c and see if it's in the list (if you are on a newer bios)?
    The splitter should be all you need.
     
  19. jhsu

    jhsu n00b

    Messages:
    25
    Joined:
    Sep 30, 2018
    I rolled back to version 2.00 from the x399 Taichi support page:
    https://www.asrock.com/MB/AMD/X399 Taichi/index.asp#BIOS

    There was only an option to switch to 4* x4 mode on the 4th slot:
    190302122145.jpg 190302122209.jpg

    I updated back to the latest version (3.50) and it only offers x4x4x4x4 mode for the x16 slots:
    190302120924.jpg 190302121104.jpg

    Where are you getting the info that x8x8 mode is available? I really wish it would work with the system I have right now. Seems like such a waste to have all those lanes coming out of the Threadripper, but not be able to split them in a practical way.
     
  20. AmusedGoose

    AmusedGoose n00b

    Messages:
    5
    Joined:
    Feb 25, 2019
    I got that info here from the first post, as I'm also about to buy that board and would need x8x8 on one slot:
    https://forum.level1techs.com/t/asrock-x399-taichi-fatal1ty-summary-thread/127541

    Could you contact ASRock about this? As far as I've seen they've been providing custom BIOS to people who requested it, it should also be possible to edit the BIOS, I might give it a go if I face the same issue.
    *Edit: I see you already did, I also contacted support, will notify here if I get any word.

    Only slot 1 & 3 would have x8x8 as 2 & 4 are getting 8 lanes.

    *Edit2: upon inspection of the BIOS with AMIBCP 5.02 I can't see any setting to add/change the bifurcation settings.

    *Edit3: Could you make sure you are on BIOS 3.5 and do a dump via AFUWINGUI? You can find it on https://ami.com/en/download-license-agreement/?DownloadFile=Aptio_V_AMI_Firmware_Update_Utility.zip (afu->afuwin->64 assuming you run Windows)

    *Edit4: Have you tried beta 3.33D?
     
    Last edited: Mar 4, 2019
  21. lygstate

    lygstate n00b

    Messages:
    3
    Joined:
    Mar 6, 2019
    Is there any place to buy
    RC1-PELY423-C5V3 or RC1-PELY423-C30V3
     
  22. AmusedGoose

    AmusedGoose n00b

    Messages:
    5
    Joined:
    Feb 25, 2019
    Yes, contact them and they'll sell it to you.
    Also consider https://peine-braun.net/shop/

    For those seeking to have x8/x8 bifurcation on the ASRock X399 Taichi, you'll need a custom BIOS L3.51.
    Contact me if you want it, not sure if I can share it publicly. Thanks ASRock!
     
  23. -Tilde

    -Tilde n00b

    Messages:
    13
    Joined:
    Feb 22, 2018
    How long does it take for ameri-rack to respond? I want a few of the flexible 2x8 risers, for some ITX servers for my homelab (I need 10Gb SFP+ on all of them, and a GPU on 2 of them, and an HBA card on one of them as well. Going to use AsRock ITX AM4 boards). I emailed them about a year ago and never got a reply, but I emailed again yesterday asking for pricing and a shipping estimate and haven't yet gotten a response.

    Awesome to see C_Payne's shop up and running! If I can't get an ameri-rack one I might have to buy from there, but I really do need the flexible one.
     
    lygstate likes this.
  24. lygstate

    lygstate n00b

    Messages:
    3
    Joined:
    Mar 6, 2019
     
  25. lygstate

    lygstate n00b

    Messages:
    3
    Joined:
    Mar 6, 2019
    Hello, Tilde, I am already give up to use ameri-rack to retrieve a flexible 2*8 risers, I am also have the requirements to getting 4*10Gb SFP+ and also 4*1Gb POE(48v) ports.
    And Now I am not using flexible riser but using a fixed 2*8 riser.
    I'd like to post a URL for you to buy, but comes from China.
    And besides, I am also ask for someone designing a custom ITX-box for two 2*8 PCI-E card(Full height).
    You may be interesting:)
    25pjsih.jpg
    2uftt7t.jpg 35ciq1s.jpg
    33c4z9g.jpg
     
  26. -Tilde

    -Tilde n00b

    Messages:
    13
    Joined:
    Feb 22, 2018
    Are you selling the Ameri-Rack ones? If you could send me a link so I can check if they're the right size that would be great!

    That's an interesting case design, but one problem I see is the GPU is going to have almost no cooling because of the PSU being in the way.

    EDIT: I got a reply from Ameri-rack, I won't need your ones.
     
    Last edited: Mar 20, 2019
  27. Fasda

    Fasda n00b

    Messages:
    11
    Joined:
    Mar 17, 2019
    I'm curious as to how exactly some of you guys got in contact with asrock support regarding bios issues.
    Was it through https://event.asrock.com/tsd.asp ??
    Reading this thread while helping a friend figuring out bifurcation issues regarding a 4slot m.2 adapter card reminded me of a issue I have with one of asrock's mITX boards regarding fan pwm controlling.
     
  28. -Tilde

    -Tilde n00b

    Messages:
    13
    Joined:
    Feb 22, 2018
    Does anybody know if the ASRock Fatal1ty B450 itx boards support x8/x4/x4? I've reached out to them but haven't received a reply yet.
     
  29. -Tilde

    -Tilde n00b

    Messages:
    13
    Joined:
    Feb 22, 2018
    Also, I reached out to AsRock rack about their upcoming matx AM4 server board, the X470D4U. The spec page lists x4/x4/x4/x4, but a rep confirmed x8/x8

    upload_2019-3-30_11-50-35.png
     
  30. AmusedGoose

    AmusedGoose n00b

    Messages:
    5
    Joined:
    Feb 25, 2019
    The epycd8-2t also does x8/x8 and x4/x4/x4/x4, giving a possible 11 Pcie x8 fully connected to CPU! Or 24 x4 if you include the 2 m.2 connectors. Combined with a 100 gbit nic it'd move data faster than throwing a drive down the hall. :woot:
     
  31. -Tilde

    -Tilde n00b

    Messages:
    13
    Joined:
    Feb 22, 2018
    Anybody have a list of boards that support x8/x4/x4? Preferably AM4.

    I contacted AsRock about the b450 fatal1ty itx and got told it only supports x16 or x8/x8.
     
  32. C_Payne

    C_Payne [H]Lite

    Messages:
    119
    Joined:
    Jan 6, 2017
    Ryzen only supports x8x8 on its lanes. Any recent Intel CPU can work in theory, depending on bios support. Asrock has been a good choice for this.
     
  33. -Tilde

    -Tilde n00b

    Messages:
    13
    Joined:
    Feb 22, 2018
    Hmm, I'd prefer AMD as a similarly priced intel system is literally half as powerful CPU wise (2200g + board vs Celeron/Pentium gold + board). The end goal for this system is not to be powerful, I want it to just be a NAS, but in the meantime it will be my main server so it can't be too weedy.

    I plan for up to 18-24 drives, so I can't go too low end (need 4 lanes for 10g card, 8 lanes for SAS card, and then it depends on which SAS expander I buy if I'll need more. Some SAS expanders need pcie power, some can run from molex), but I might be able to get away with just x8/x8. We'll see.
     
  34. C_Payne

    C_Payne [H]Lite

    Messages:
    119
    Joined:
    Jan 6, 2017
    Hi everyone. My webserver got completely wiped(actually its my fault and I really want to bang my had very hard)
    So my shop will be offline for a while, I will try to at least get the mailserver running again tonight, you will then be able to reach me via risers@peine-braun.net again.

    C_Payne
     
  35. DaHans

    DaHans n00b

    Messages:
    37
    Joined:
    May 21, 2015
    Hi there!
    I love this thread. I have a x99e ITX in a Dan A4 and want to put a PCIe splitter on the GPU side. So what I know is, I need a singe slot Riser like the Ameri-Rack ARC1-PELY423-CxV3. I also found on C_Payne Website the "x8x8 left turn -1 Slot distance" https://peine-braun.net/shop/index.php?route=product/product&path=65_59&product_id=68

    So the Ameri-Rack would fit perfect but I also wanted to check If I could just turn the the x8x8 Riser around? That would mostly depend on the length of the stock riser in the DAN A4 Case. Otherwise I would have to expand it. I still have a spare riser from ADT.

    Edit:
    breathless
    What cable ist that exactly on the mini-PCI Express Slot? mPCIe to USB?
     
    Last edited: Jun 5, 2019
  36. C_Payne

    C_Payne [H]Lite

    Messages:
    119
    Joined:
    Jan 6, 2017
    It seems the Dan A4 is a popular case. Maybe I make a custom riser for that case. Is the mounting mechanism of the cable riser the same for different revisions of the case?
    What would be the use case? Single Slot GPU+NIC/Capture Card?
     
  37. DaHans

    DaHans n00b

    Messages:
    37
    Joined:
    May 21, 2015
    It is definetly.
    Custom Riser would be great. The Cable Riser changed several times :https://www.dan-cases.com/dana4_spec.html
    If you scroll down on the spec page, there is a changelog. I have the one with v1 and the "ADT PCIe 3.0 x16 Flexible Riser × 1" one. The ADT is just 185mm long.

    Use cases are a lot different. Even mine changed several times. I tried getting a SLI with 2 KFA 1070 single slot GPUs, then I got one single slot GTX 1050 and a Sonnet Tempo SSD Mounting pcie card* and the third version would be single slot GPU and Capture Card.

    Hopefully this one will be another option:
    https://www.overclock3d.net/news/gp...s_the_world_s_first_single-slot_gtx_1660_ti/1

    By the way: Hello from Germany! :D



    *https://www.amazon.de/Sonnet-TSATA6-SSD-E2-Tempo-SATA-Drive/dp/B0096P62G6/ref=sr_1_1?__mk_de_DE=ÅMÅŽÕÑ&keywords=sonnet+tempo&qid=1559725108&s=gateway&sr=8-1
     
    Last edited: Jun 5, 2019
  38. C_Payne

    C_Payne [H]Lite

    Messages:
    119
    Joined:
    Jan 6, 2017
    OK i have an idea in my mind already. If the cable is unscrewed can it be moved aprox 10mm outwards and 15mm bottomwards?
     
  39. DaHans

    DaHans n00b

    Messages:
    37
    Joined:
    May 21, 2015
    I'm not able to check it right now. I will unscrew it later on to take a picture. Otherwise maybe dondan could say it easily.

    Regarding that picture I would say yes: Edit: I don't know.
    Edit2: I'm bad in taking pictures. Added two more. There is the power cord. In what position would you put the riser?
     

    Attached Files:

    Last edited: Jun 5, 2019
  40. C_Payne

    C_Payne [H]Lite

    Messages:
    119
    Joined:
    Jan 6, 2017
    Mhh thr power cord will be in the way... I think I will buy one and see for myself.