PCIE Bifurcation

Discussion in 'Small Form Factor Systems' started by jb1, Jul 28, 2015.

  1. Neapolitan6th

    Neapolitan6th Gawd

    Messages:
    936
    Joined:
    Nov 18, 2016
    I'd wait to see what gets announced at CES here in the next week. I've already heard asrock will be releasing a pretty killer microATX threadripper motherboard. I'm sure there will be more product releases that may entice you as well. I'm thinking if a board supports bifurcation by default, I would always reccomend that over an option that requires bios modifications (for those who haven't invested in the hardware yet).
     
    3vilgenius likes this.
  2. 3vilgenius

    3vilgenius n00bie

    Messages:
    24
    Joined:
    Jan 4, 2018
    Ooooo I can't wait!!!! Although is there any reason you say I should watch for the threadripper board besides general hype/awesomeness? I don't really have the cash for that nor would I actually take advantage of all those threads, even if I did a crossfire build, at least I don't think. I should have mentioned I want this build purely for gaming badassery.
     
  3. Neapolitan6th

    Neapolitan6th Gawd

    Messages:
    936
    Joined:
    Nov 18, 2016
    I'm meaning to say that there might be news regarding other product announcements. Not just threadripper. Say a new ITX am4 board perhaps. Or new ryzen cpus announced. (APUs for a good value.. Or maybe they will announce their refreshed ryzen line which is supposed to be available relatively soon.. Those chips may clock higher which would give you better gaming performance)
     
    3vilgenius likes this.
  4. fAiLix

    fAiLix n00bie

    Messages:
    1
    Joined:
    Jan 5, 2018
    Hey, i am interested in one x16-->2x8 Riser with the slots facing away from the CPU. Do you have any of them left?
    Is there a way to get in contact with you? i am not able to send you a pm.
     
  5. C_Payne

    C_Payne n00bie

    Messages:
    60
    Joined:
    Jan 6, 2017
    I do... I will make an email address for this later today it will be risers@peine-braun.net
     
    3vilgenius likes this.
  6. davidm71

    davidm71 [H]ard|Gawd

    Messages:
    1,411
    Joined:
    Feb 11, 2004
    Guys,

    I think I figured out how to on the fly change my link speed up to 8.0 GT/S!!! This was with issuing Setpci commands to change ‘Extended Sync’ bits on the IO controllers registers. I also noticed that the L1 exit latency speeds on the controller were much less than the downstream device of which I manually corrected in my bios so they match. This L1 exit latency I believe is the time it takes for the device to exit a low power mode. So if the controller isnt giving it enough time to recover from sleep or off powered state there could be problems with link speed but i’m no expert. It took turning on extended sync to get it up to 8 gt/s. Could be my Plextor nvme drive thinks its a Diva or something because the Samsung 950 Pro on the other end of the port was fine. In anycase if you guys could check and report your link speeds in HWInfo that would be great. Video cards might act differently than a SSD as I’ve noticed they up regulate under load. So please check after the system has been off but warm for like an hour.

    Thanks.

    Edit: Could not switch up to 8.0 GT/s on the Fly after all. Must be a bad hard drive.
     
    Last edited: Jan 9, 2018
    Neapolitan6th likes this.
  7. Neapolitan6th

    Neapolitan6th Gawd

    Messages:
    936
    Joined:
    Nov 18, 2016
    Asrock's new quad m.2 (3.0x16) card is going to require motherboard bifurcation support in order to function. Link.

    Glad to see more reasons for consumer motherboard bifurcation support to be taken more seriously.
     
  8. C_Payne

    C_Payne n00bie

    Messages:
    60
    Joined:
    Jan 6, 2017
    My MSI X299 GAMING PRO CARBON AC does actually support bifurcation, however I can only select bifurcation to be on or off.. so i cant bifurcate to x8x8 but only into x4x4x4x4.
    Also when i select bifurcation the VROC driver is loaded automatically and neither linux nor windows will boot.
     
  9. nekrosoft13

    nekrosoft13 [H]ard|Gawd

    Messages:
    1,444
    Joined:
    Jan 4, 2005
  10. TheArkratos

    TheArkratos n00bie

    Messages:
    29
    Joined:
    Nov 20, 2017
    The lower riser set up is from amerirack
     
  11. Man

    Man n00bie

    Messages:
    19
    Joined:
    Oct 8, 2017
    Has anyone got this to work with the Dancase A4-SFX?
    I am obviously not looking to run SLI GPUs or anything but more along of 2 professional soundcards. (So it doesn't need to support anything more than 1X to 4X). I'm looking for a PLX solution, with the right lenght of the ribbon cable that ends in a double pcie socke that fits within the limited heigh in the dancase. I am waiting for my V2 case to arrive, - but in my existing v1 case I actually have a pcie to double pci bridge for my older but still very good PCI soundcards. The double PCI socket is too tall to mount where I want it - instead I mounted it on the backside of the front - using the existing holes for a 2.5" drive. I'm waiting for PCI ribbon cables to try this out. (btw the reason that I have space to mount the bracket behind the front is that I use a custom PSU with a pico-psu, not a SFX PSU) Anyways - I digress.
     
    Minefoxi likes this.
  12. Neapolitan6th

    Neapolitan6th Gawd

    Messages:
    936
    Joined:
    Nov 18, 2016
    Hmm just to touch on the Dan case part: there is someone here in this thread who ran crossfire amd fury nanos. Not exactly what your looking for, but if you can find the picture somewhere in this thread, it may provide some inspiration.

    Edit: (credit: maximumburito)
    [​IMG]
     
    Jspr and Master_shake_ like this.
  13. Man

    Man n00bie

    Messages:
    19
    Joined:
    Oct 8, 2017
    Thats quite cool, but I would just use a 1080ti instead and get better performance..

    Just wanting to make a small - near silent home studio. A pair of Lynx E44 sound cards would give 8+8 in/out channels. Enough for analog 7.1 surround and 8 channel inputs of professional quality. I realize I could probably just use a newer motherboard with a m.2 socket and a m2->pcie riser, but whats the fun in making it easy? :) (besides, I would lose a m.2 slot - that's unacceptable! )

    The perfect riser would be an ultra low profile card (with PLX chip for univeral compatibility) that goes in the pcie slot and face away from the cpu, and would have 2 pci sockets closely spaced. That would just about fit without hitting the top.
    Then I could use 2 of the dancase risers , 1 for each soundcard. (I have one spare since I am using a PCIE-PCI bridge in the other A4-SFX)
     
  14. TheArkratos

    TheArkratos n00bie

    Messages:
    29
    Joined:
    Nov 20, 2017
    3vilgenius and Neapolitan6th like this.
  15. Neapolitan6th

    Neapolitan6th Gawd

    Messages:
    936
    Joined:
    Nov 18, 2016
    That's awesome! That's been basicly my dream build since I heard of bifurcation. Inspiring! Out of curiosity which software did you use to set up the VM?
     
  16. ChronoDog

    ChronoDog [H]Lite

    Messages:
    101
    Joined:
    Oct 15, 2009
    davidm71 Thank you very much for taking the time to write such a detailed guide, and thanks to everyone else in this thread sharing a whole lot of extremely useful information, it's been very helpful.

    I've tried using the above guide to enable PCIE Bifurcation in my Z87E-ITX's BIOS, alas it's APTIO v4 and AMIBCP for that version sees just a list of "undefined" categories with nothing even remotely similar to the above in any of them.

    Does anyone still have the P2.50B BIOS from Ameri-Rack for that board, and would be kind enough to share it, via PM? :)

    I plan on giving this board a sort of a second life by adding a bootable 960Pro using the Asus Hyper M.2 x4 Mini card on a bifurcated riser along with the graphics card. Already tested adding NVMe boot support to the BIOS, which seems to work surprisingly well with the parts mentioned.

    Also, speaking of Ameri-Rack, I've had no luck getting a reply from them about purchasing one of their bifurcated risers, is there a certain way I should phrase my request, or is there anyone specific there I should contact about it?

    Thanks in advance :)
     
  17. TheArkratos

    TheArkratos n00bie

    Messages:
    29
    Joined:
    Nov 20, 2017
    I feel like an idiot when I say this, but at the time Ryzen was still dealing with the NPT bug (on KVM) so I was having to try Xen for my hypervisor. However bifurcation (or something) did not get along with it. I reached out to AMD for help, they were super excited about the project to the point that instead of giving me help they sent me a threadripper 1950x and said can you make it bigger?

    Moral of the story, I didn't get the VMs working with pass through... however now the NPT bug has been fixed on KVM and you should just be able to do that.

    I plan to pick the project back up when I'm done with the threadripper version and can move stuff around. But I needed the RX480s from that because... well RX480s are impossible to get and I can't afford the only other single slot cards on the market, the 1080ti and reference Vegas. (I mean single slot watercooled)
     
    Neapolitan6th likes this.
  18. Neapolitan6th

    Neapolitan6th Gawd

    Messages:
    936
    Joined:
    Nov 18, 2016
    No worries there. I frankly wasn't sure if Ryzen VMs and gpu passthough had been ironed out yet. Most suggested to stick with Intel. I'm glad to hear that will no longer be the case.

    Wow, that's great of AMD to support your project like that! If you are still looking to go a little smaller, Asrock has announced a MicroATX Threadripper motherboard with 3 full 3.0x16 slots and 3 3.0x4 M.2 slots. Otherwise I'm sure a normal ATX Asrock Taichi board would do well.

    Good luck!
     
  19. TheArkratos

    TheArkratos n00bie

    Messages:
    29
    Joined:
    Nov 20, 2017
    I think the VMs would have worked if I wasn't using Bifurcation to split the lanes like that but I'm not sure, I'll give it a go when I'm done with the current project.

    And AMD sent me the Aorus gaming 7 x399 board with it. No fun bifurcation on that project... yet, but I'd love to one day make sure I can use one of the 4x M.2 riser cards, that is bifurcation, so one day if that motherboard doesn't get support I may have to mod the bios. This thread is turning into quite the resource. You guys are amazing.
     
    Neapolitan6th likes this.
  20. LukeD

    LukeD Limp Gawd

    Messages:
    160
    Joined:
    Jul 18, 2016
    Thought I'd share my PCIe bifurcated build in my custom case: Project Orthrus

    4k_DSC01837.jpg 4k_DSC01845.jpg 4k_DSC01847.jpg 4k_DSC01875.jpg 4k_DSC01886.jpg

    DSC01892_noside.jpg DSC01892_side.jpg DSC01897_gpu_noside.jpg DSC01897_gpu_side.jpg DSC01900.jpg DSC01901_back.jpg

    And yes, SLI does work:

    po_1.jpg
     
    Last edited: Jan 30, 2018
    aquelito, D10S, TheArkratos and 2 others like this.
  21. 3vilgenius

    3vilgenius n00bie

    Messages:
    24
    Joined:
    Jan 4, 2018
  22. TheArkratos

    TheArkratos n00bie

    Messages:
    29
    Joined:
    Nov 20, 2017
    Looks great! Do you have a link to a build log or something? I'm curious on your watercooling set up and more photos of the build with full lighting.
     
  23. LukeD

    LukeD Limp Gawd

    Messages:
    160
    Joined:
    Jul 18, 2016
    Neapolitan6th likes this.
  24. TheArkratos

    TheArkratos n00bie

    Messages:
    29
    Joined:
    Nov 20, 2017
    Now that I can see the build... I recognize that case! You sir do great work!
     
  25. ChronoDog

    ChronoDog [H]Lite

    Messages:
    101
    Joined:
    Oct 15, 2009
    Here's my own PCIe bifurcation experience so far:

    - Z87E-ITX + P2.50B BIOS + NVMe boot module
    - Ameri-Rack ARC-PERY423-C10 (dremel'd empty PCB space off the sides, removed PCIe clips)
    - JEYI MX16-1U PCIe M.2 adapter
    - Samsung SM961 + a heatsink off of some VRMs

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    I've since improved the drive's speeds a bit by tinkering with different RAID ROMs and drivers, but regardless, it seems to have been working flawlessly for about two weeks now in this configuration.

    So yay, an easily accessible and fairly well-cooled M.2 Gen3 x4 slot on a Z87 board :)
    Also, the performance difference between running a 1080Ti at x8 vs. x16 is absolutely negligible.
     
    Last edited: Feb 20, 2018
    3vilgenius and Neapolitan6th like this.
  26. 3vilgenius

    3vilgenius n00bie

    Messages:
    24
    Joined:
    Jan 4, 2018
    glorius
     
  27. -Tilde

    -Tilde n00bie

    Messages:
    7
    Joined:
    Feb 22, 2018
    Long time reader first time caller

    My current plan for my next system is an Asrock B350 itx, in a custom case (WIP). But I want an 8x slot and 2 4x slots. 8x for gpu to be passed through to windows vm, 4x for host gpu (thinking a GT 1030), and another 4x for expansion in the future (maybe a 10 gigabit NIC down the line, more NVME drives, another compute gpu etc). So is it possible to bifurcate an 8x into 2 4x? And which splitters would be appropriate?

    I recall talk of a 2 way splitter that deosn't sacrifice the motherboards PCIE slot, did anyone ever finish that?
     
  28. Awilen

    Awilen n00bie

    Messages:
    18
    Joined:
    Sep 1, 2017
    This is on hold. I designed a (non-routed) PCB layout, but I reckon it will fall flat on its face if it uses flexible printed circuit to link the slot to the additional PCB, mostly due to electromagnetic noise. PCIe 3.0 8x bandwidth just won't be possible.

    I originally considered ethernet cabling to route the signals to the daughter card, but again, to ensure proper noise protection and bandwidth, the use of shielded cables seemed mandatory, and they are just too thick.

    The plan was to be able to stack two GTX 1080 Tis in a Define Nano S, but after further "research" in SLI scaling and processor capabilities to drive two of such card at the moment, plus the various hickups to try to get SLI to work and work well, it's pretty much a huge waste of time and money. Doing it for the sake of it is still on the table, but later, way later, when I have more disposable income.

    As it stands right now, and since you are designing your own case, I suggest you go the sacrifice-the-slot route.

    (I also looked into other things like internal thunderbolt-to-PCIe daugthercard or even NVMe-to-PCIe from the backside of the motherboard, but PCIe 4x is just too bandwidth-limited for the 1080 Ti.)
     
  29. -Tilde

    -Tilde n00bie

    Messages:
    7
    Joined:
    Feb 22, 2018

    Yeah, unless I want to go custom PCB etc etc I guess I'll have to sacrifice the port. If you didn't need a PLX chip you could just split a riser and solder these pins into into here or maybe these, but I guess with a PLX chip it would make it even more difficult than soldering that many tiny wires.

    I'm just having trouble finding any dimensions for risers to use in my 3d models, which is a bit of a roadblock at the moment. I probably won't have many shots at prototyping either (the chassis is mostly made from one piece of aluminium), so getting the dimensions right first time is pretty important. So if anyone has the amerirack/any other confirmed working risers, I'd greatly appreciate some measurements.
     
    Awilen likes this.
  30. Aeneas

    Aeneas n00bie

    Messages:
    1
    Joined:
    Feb 23, 2018
    I am interested to know what effect PCIE Bifurcation has on device enumeration.
    If a PCIE x16 slot is segmented into 8 + 4 + 4 lanes,
    do each of these segments show up with their own bus/device/function ?
    Are there PCIE slot cards available right now which have 3 devices
    arrayed on the slot card in which 1 device takes the x8, 1 takes the x4 and the other x4 ?
     
    Neapolitan6th likes this.
  31. Awilen

    Awilen n00bie

    Messages:
    18
    Joined:
    Sep 1, 2017
    Not sure about 8x/4x/4x, but the Asus Hyper M.2 x16 has 4 PCIe NVMe SSDs connected through bifurcation in a 4x/4x/4x/4x configuration.
     
  32. Neapolitan6th

    Neapolitan6th Gawd

    Messages:
    936
    Joined:
    Nov 18, 2016
    I believe the Asus Hyper m.2 handles the bifurcation, instead of the motherboard so it'll work on sytems that don't support bifurcation.

    Asrock makes a cheaper version with the caveat of requiring motherboard bifurcation support
     
  33. -Tilde

    -Tilde n00bie

    Messages:
    7
    Joined:
    Feb 22, 2018
    I did look into those, and then getting m.2 to pcie 4x adapters, but it was just too expensive and large.
     
  34. Awilen

    Awilen n00bie

    Messages:
    18
    Joined:
    Sep 1, 2017
    There were quite a number of posts in this very thread clearly stating that bifurcation support must be present on both the chipset and in the BIOS to work, so I have a hard time believing ASUS made a bifurcation board that "just works" without something akin to a PLX chip, and working on boards by other manufacturers. So I did some more research on the Hyper M.2.

    It looks like the only present IC on the PCB is the clock buffer, just like on the SuperMicro and Amerirack passive risers, and like C_Payne's. Unfortunately I haven't been able to find a photo of the back of the PCB.

    ASUS only lists ASUS boards as compatible with it.
    I've looked for reviews, found some on Amazon. People have made it work with Intel x299 and AMD x399 motherboards, from MSI and Gigabyte. I've looked into their respective manuals, and I didn't find anything remotely related to bifurcation support (though I may be wrong, that already happened. -^^-) Only VROC was mentioned on Intel's side (and the Hyper M.2 is VROC-ready), and AMD Nvme-Raid driver on the other then setting the drives in RAID in BIOS, so those could be dealing with bifurcation.

    Der8auer mentions bifurcation in his video

    That's it for what I gathered.
     
  35. Neapolitan6th

    Neapolitan6th Gawd

    Messages:
    936
    Joined:
    Nov 18, 2016
    I wish I could remember where I saw a story about the Asus and Asrock m.2 cards.. They talked about how the bifurcation of each was handled.. Oh well.

    On a slightly unrelated note, I have personally been looking for a dual nvme m.2 x8 card as I will only have the bandwidth for 2 m.2 devices after I bifurcate my x16 slot.

    I happened to find a supermicro device that seems interesting.

    Supermicro NMVe AOC-SLG3-2M2
    [​IMG]
    Amazon link only $45 at that

    Amazon reviews still mention x8x4x4 is required.

    I imagine this would pose a similar issue

    Edit: happy to see my x99 itx asrock board supports x8x4x4 configurations
     
    Last edited: Feb 28, 2018
    Awilen likes this.
  36. Man

    Man n00bie

    Messages:
    19
    Joined:
    Oct 8, 2017
    ChronoDog Where did you buy the riser? I'd like one too :D
     
  37. ChronoDog

    ChronoDog [H]Lite

    Messages:
    101
    Joined:
    Oct 15, 2009
    Got it directly from Ameri-Rack, as mentioned in my post :)
     
  38. -Tilde

    -Tilde n00bie

    Messages:
    7
    Joined:
    Feb 22, 2018
    How do you order one/get in contact with them? And how much was it? Looking at getting a trifurcation riser from them.
     
  39. ChronoDog

    ChronoDog [H]Lite

    Messages:
    101
    Joined:
    Oct 15, 2009
    There's a "contact" button on the website with a form you can fill to contact them, along with an email address.
    You need to request a quote for the specific model/length you need, as they all differ in price - as well as the shipping cost - so you should just ask them directly.
    The one I got costed me $55 + spipping, if that helps.

    It may sometimes take them a while to get back to you, but once they do - they're kind and fairly quick about the rest :)

    Good luck!
     
  40. -Tilde

    -Tilde n00bie

    Messages:
    7
    Joined:
    Feb 22, 2018
    :facepalm: I can't believe I didn't see the contact form, thanks!