PCIE Bifurcation

Discussion in 'Small Form Factor Systems' started by jb1, Jul 28, 2015.

  1. jb1

    jb1 Limp Gawd

    Messages:
    494
    Joined:
    Dec 27, 2014
    Hi all,
    I'm interested in doing a build with two GPUs in SLI using one 16x PCI-E slot on a mini ITX motherboard. My understanding is that the Z97 chipset supports PCI-E bifurcation so you can just split the signal into two 8x ports. However, my motherboard has no setting for PCI-E bifurcation. Is it necessary to enable this, or will the chipset just figure it out when it sees two devices on the same PCI-E slot?
     
  2. Quix

    Quix 2[H]4U

    Messages:
    3,704
    Joined:
    Jun 12, 2011
    PCI-E Bifurcation requires bios support, I'm not actually sure the Z97 supports this feature anyway. As an example ASRock's support site says only their X99 boards support it.
     
  3. jb1

    jb1 Limp Gawd

    Messages:
    494
    Joined:
    Dec 27, 2014
  4. Kyle_Bennett

    Kyle_Bennett El Chingón Staff Member

    Messages:
    46,485
    Joined:
    May 18, 1997
  5. Quix

    Quix 2[H]4U

    Messages:
    3,704
    Joined:
    Jun 12, 2011
    Yup, contact your motherboard manufacturer to be sure.
     
  6. iFreilicht

    iFreilicht [H]ard|Gawd

    Messages:
    1,348
    Joined:
    Sep 23, 2014
  7. Phuncz

    Phuncz 2[H]4U

    Messages:
    2,634
    Joined:
    Apr 12, 2009
    Some ASRock motherboards have support but also require a specific BIOS to enable them, like the Z87E-ITX.

    Mind you that all Z87 and Z97 chipsets support PCIe Bifurcation, but only a select few BIOSes have this enabled.
     
  8. jb1

    jb1 Limp Gawd

    Messages:
    494
    Joined:
    Dec 27, 2014
    Okay great, thanks for the note–– I couldn't find bifurcation mentioned anywhere in the X99E/ITXac manual. Anyone know about gigabyte's GA-Z97N-Gaming 5?
     
  9. Phuncz

    Phuncz 2[H]4U

    Messages:
    2,634
    Joined:
    Apr 12, 2009
    Don't count on it. As far as I know, ASRock is the only brand that has consumer-oriented boards with this feature: http://www.ameri-rack.com/ARC2-PELY423-C7_m.html
    You could contact Ameri-Rack for a list of consumer boards and supported BIOS versions that do.
     
  10. jb1

    jb1 Limp Gawd

    Messages:
    494
    Joined:
    Dec 27, 2014
    Just called gigabyte... whomever I spoke with hadn't heard of PCI-E bifurcation before and said that it won't work. I believe this.

    Somehow I feel cheated of this excellent feature of the Z97 chipset–– should have anticipated this desire a year ago.
     
    Last edited: Jul 28, 2015
  11. jb1

    jb1 Limp Gawd

    Messages:
    494
    Joined:
    Dec 27, 2014
    Last edited: Jul 28, 2015
  12. chemistslime

    chemistslime n00bie

    Messages:
    3
    Joined:
    Jul 28, 2015
    dude, I'm doing the exact same thing with the asrock x99e itx board, what a small world. I have actually tried it with a few diff splitters and have had no luck so far.

    Products I have tried are, this one failed to show anything.

    http://www.amazon.com/Supermicro-RS...r=8-3-fkmr0&keywords=supermicro+pci+explitter

    I have also tried this one, also failed to show any graphics card display.

    http://www.ameri-rack.com/ARC2-PELY423-C7_m.html

    I just got into today: http://www.acmemicro.com/Product/13...E16-A-LHS-Active-PCI-E-2U-Riser-Card?c_id=356

    This product is different from others in that it has an active PCI splitter but I have yet to verify if it has a PLX chip yet because it's under the heatsink. I'll do that this weekend when I have more time with the system. Unfortunately, this splitter card is too long for the case I'm using so even if I get it working I'll have to find an alternative solution but at least I'll know it works.

    Anyone know if this is bios related or not? Do I need to turn something on/off?

    I'm really hoping to get this all working for this special project I'm working on which will be a one of a kind but getting this dual graphics card working in SLI would be amazing!
     
  13. jb1

    jb1 Limp Gawd

    Messages:
    494
    Joined:
    Dec 27, 2014
    Nice, glad to see you're trying this!

    You might want to call Asrock and see what they say–– maybe you need a special version of the BIOS to bifurcate? As posted above, the X99 ITX should work with bifurcation according to Asrock:
    http://forum.asrock.com/forum_posts.asp?TID=355&title=pcie-bifurcation-in-z97eitxac

    Also, for the active splitter that you just got (RSC-R2UG-A2E16-A), could you post the dimensions of the card? I want to see if this will fit in my case and I can't find concrete details on its length and width.

    Please let us know how it goes!
     
  14. chemist_slime

    chemist_slime [H]ard|Gawd

    Messages:
    1,095
    Joined:
    Jul 24, 2005

    Got to try out the new Supermicro PCIE Riser and unfortunately it did not work either. No boot screen at all. I sent Asrock a support ticket and we'll see what they say. Perhaps a bios update is needed to make this all work in the end. BlueFox, from another thread has it, v1.35 so we'll see. You can see progress here:

    http://hardforum.com/showthread.php?t=1855235&page=15


    As for the super micro rise, it's about 6.8 inches in length, a big longer than the actual ITX form factor of 6.7".
     
  15. chemist_slime

    chemist_slime [H]ard|Gawd

    Messages:
    1,095
    Joined:
    Jul 24, 2005
    Btw, just tried the bios 1.35 from blue fox and no luck. Looks like we'll have to wait for Asrock to officially respond.
     
  16. QinX

    QinX Limp Gawd

    Messages:
    230
    Joined:
    Jan 15, 2015
    In order to have PCIe Bifurcation to work for 2 PCIe devices you need the following:

    1) Bios Support, the Bios needs to have th code to detect en change the PCIe lane assignments from 16x to 8x/8x if the code is not there it doesn't work

    2) Each PCIe device needs a Reference Clock. Multi PCIe boards have multiple Reference clocks 1 for each PCIe slot. In order to get more then 1 PCIe reference clock into 1 PCIe slot you need either

    A) deviate from the PCIe specification and use some of the pins to bring in another reference clock. This is what SuperMicro does.

    B) Multiplex the Reference clock on the PCIe riser PCB.

    I've had a look into this as well with the same idea of running X99 ITX with 2 GPUs. I've measured all of the pins on this SuperMicro card and in my excel sheet below you can see they hacked up the PCIe slot specification to get all the signals in, that is why they only work on a select number of their own server boards.

    RSC-R2UG-2E4E
    [​IMG]

    The actual pin-out
    [​IMG]

    The other option I've looked into is the PLX chip, but those will be very expensive, this is old information but these things can cost as much as $80, plus the validation that needs to be done because of the switching involved. But with a PLX chip you can get any dual GPU setup to run on a single PCIe slot, but that riser will more then likely costs $100+
    http://www.anandtech.com/show/6170/four-multigpu-z77-boards-from-280350-plx-pex-8747-featuring-gigabyte-asrock-ecs-and-evga/31
     
  17. jb1

    jb1 Limp Gawd

    Messages:
    494
    Joined:
    Dec 27, 2014
    Unfortunate! By any chance do you have the width of the card in addition to the length?

    Maybe this isn't really a possible option given QinX's post though...

    I wonder, can consumers even purchase PLX chips and soldier the PCI-E connections to the chip?
     
  18. QinX

    QinX Limp Gawd

    Messages:
    230
    Joined:
    Jan 15, 2015
    It should be possible, the problem lies only with the motherboard manufacturer, if they have the code in the BIOS to bifurcate then you can make a PCB for it.

    PLX chips are BGA packages, yes you can solder it, but not with your regular soldering iron. Also because the PLX chip is a rather advanced switching chip, implementing it isn't easy, there is a lot of validation required to be sure it works properly.

    The best route to go is to build a PCIe riser card to duplicates the Reference clock.
    I've been wanting to design the PCB for it and I think I know what it needs, but because I couldn't find a ITX board that has a BIOS for PCIe bifurcation I haven't bothered to give it a try.
     
  19. jb1

    jb1 Limp Gawd

    Messages:
    494
    Joined:
    Dec 27, 2014
    Interesting. I'll be curious to see how the Asrock X99 ITX testing goes since they have a hardware rep confirming bifurcation support.

    As for building a PCIe riser to duplicate the reference clock, do you mind me asking what would be involved?
     
  20. QinX

    QinX Limp Gawd

    Messages:
    230
    Joined:
    Jan 15, 2015
    It seems that the only thing you really need to do is multiply the reference clock signal.
    All the other signals can be daisychained as seen in the Supermicro example. The only thing the added to the PCIe fingers was more power and 2 more RefClks.

    Something like the IDT 9DB233 should be able to double up the Reference Clock signals from the looks of it.

    You mentioned you had tried the ARC1-PELY423-C7 and the RSC-R2UT-2E8R.
    Could you identify what chip is on those 2 cards? It might be that they have the same type of chips.
     
  21. Diverge

    Diverge Gawd

    Messages:
    513
    Joined:
    Nov 22, 2005
  22. jb1

    jb1 Limp Gawd

    Messages:
    494
    Joined:
    Dec 27, 2014
    Don't really see anything insightful in that thread...
     
  23. chemist_slime

    chemist_slime [H]ard|Gawd

    Messages:
    1,095
    Joined:
    Jul 24, 2005
    Qinx, did you try to tackle this same issue a while back on another forum? I remember reading through your thread hoping that you'd come ups with a solution but I do not believe you did? Anyhow, I'd like to re-ignite this topic and get things working!!! once and for all. I can front some of the cost for the PLX chip and dev cost if that helps, I really want to get a dual 970 GTX working on my asrock x99e board and also get it working on a Hackintosh :) What can I do to help??
     
    D10S likes this.
  24. chemist_slime

    chemist_slime [H]ard|Gawd

    Messages:
    1,095
    Joined:
    Jul 24, 2005
    It's about 2.5 inches in height and 6.8 in length. It did not work for me though.
     
  25. QinX

    QinX Limp Gawd

    Messages:
    230
    Joined:
    Jan 15, 2015
    I've had a topic going over on EEVblog, but they told me it wasn't possible, but what do they know right ;) .

    I'm considering to make the PCBs and such for it to check it out, only thing is I don't have the money right now to try it out. I'll take a look into design the PCB and then seeing if there is interest from the community. Maybe I can get ASRock onboard to sponsor me a X99 ITX board.

    The cheapest way to go is going with the ASRock X99 ITX board because if it has PCIe bifurcation the components needed to get it working shouldn't be that expensive.

    You have 2 types of cases for this.
    PCIe perpendicular
    Where the cards are at 90 degrees with the motherboard, you'd need a 5 slot case for aircooled cards or 3 slots for watercooled cards that have single slot I/O minimum. The riser also needs to be flexible, so you'd need 2 PCBs

    PCIe parallel
    The cards are inline with the motherboard the PCIe slot needs to make a 90 degree turn. This could allow for 1 PCB and has a minimum requirement of 4 PCIe brackets for aircooled and 2 for watercooled cards.
     
  26. jb1

    jb1 Limp Gawd

    Messages:
    494
    Joined:
    Dec 27, 2014
    Yeah! If AMD and NVIDIA can implement this on their dual GPU cards then it has to be possible to do... doesn't mean it will be easy though.


    This sounds like a really fun project–– how much do you think it would cost to get a prototype made without the PLX chip assuming Asrock bifurcation support?

    I was actually considering having both cards parallel to the motherboard, one right behind the other, both mini ITX length. Only issue is that one card wouldn't have accessible ports, but it would allow for the thinnest design.

    Note that SLI is not supported in OS X. Both cards will register in OS X so you can use them for CUDA or OpenCL compute, but they won't operate together in a gaming situation.
     
  27. QinX

    QinX Limp Gawd

    Messages:
    230
    Joined:
    Jan 15, 2015
    AMD and Nvidia both implement the PLX chip, but this is more because their cards have to work with ALL motherboards, so even the ones that don't have PCIe bifurcation. They also have a lot more resources for it.

    Like most things this scales with quantities, making 1 is probably not worth it, making 5+ reduces price a lot. a guestimate would be that if I'd make around 10 boards you'd pay 40 bucks for each.


    Do you mean turning two 17cm cards into one 34cm card? The downside is with Nvidia, you can't SLI because the bridge won't reach.

    Of course because this has never been done before decide on the orientation isn't easy. Most ITX cases have cards perpendicular to the motherboard so that would mean a flexible riser, doable but more expensive then the parallel version, however with the parallel version you have the issue that most ITX only have 2, maybe 3 slots available where you'd need 4 to get this working with aircooled cards.

    Going for watercooled cards you only need 2 PCIe slots if you use either a reference GTX970 or a Fury X. Not the cheapest cards but if you already have the X99 ITX board you probably aren't looking at a GTX960.

    Going with aircooled cards means a minimum of 4 PCIe slots are needed. I could design a chassis for it, but then we go to a whole other level.
     
  28. jb1

    jb1 Limp Gawd

    Messages:
    494
    Joined:
    Dec 27, 2014
    Well, it sounds like there are at least three here who might want something like this!

    And yes, kindof. I was actually planning something even crazier, mounting one normally, and the other pointing vertically to reduce the length.

    This would probably be done with AMD, since CrossFireX works right over PCI-e
     
  29. Diverge

    Diverge Gawd

    Messages:
    513
    Joined:
    Nov 22, 2005
    Well I was just showing some people got it working correctly on a supermicro board w/ non supermicro PCIe extenders, and listed the part numbers they used. Whether those same part numbers work on an Asrock board is another story.
     
  30. jb1

    jb1 Limp Gawd

    Messages:
    494
    Joined:
    Dec 27, 2014
    Ah got it, must have missed that when reading through. Sorry!
     
  31. QinX

    QinX Limp Gawd

    Messages:
    230
    Joined:
    Jan 15, 2015
    Ofcourse any configuration can be made, but for just 1 piece you can count on it costing some where between $150 and $250, because making a single PCB isn't cheap, it has to be a 4 layer PCB and you'd need gold plating on it.

    I think it's best if we approach it from a somewhat "normal" use case.

    Having a perpendicular setup is a no go imho. because if you have them perpendicular why even have the ITX formfactor because you'd be right at the mATX size. Unless you go watercooled single slot cards, you'd might be able to cram 2 cards into the NCASE M1 or a Bitfenix Prodigy

    If you'd go parallel on the side of the motherboard you can have 2 cards in 4 slots. the height of the cards also allows for a ATX PSU in the front. You'd need a flexible PCIe riser.

    Watercooling the cards and CPU brings a lot of options, although you'd need space for the radiator.

    @chemist_slime

    Would it be possible for you to make some high-resolution photo's of all the different riser you have? Front and Back if possible.
    I'd like to to a look at the layout and tracing.

    You mentioned you tried the supermicro card and it didn't boot?
    You mean Supermicro RSC-R2UG-A2E16-A didn't work?
    Could you remove that heatsink and check if it's a PLX chip, because if it is it should work without a problem, unless they also f****d up the signal like the did with my card, which I bought because I want to retrace all the signals.
     
    Last edited: Jul 30, 2015
  32. jb1

    jb1 Limp Gawd

    Messages:
    494
    Joined:
    Dec 27, 2014
    Yes, definitely agreed normal use case would be best. I would implement my idea with flexible risers off of this PCB.

    Also agree parallel to the motherboard is the way to go, otherwise, yes, you have an mATX board! Maybe best to assume flexible risers will be in use so people can get the GPU's where they want them depending on the case they are using.
     
  33. jimmyb

    jimmyb 2[H]4U

    Messages:
    3,172
    Joined:
    May 24, 2006
    If you're planning on doing your own PCB you should review the PCIe card electromechanical spec. It covers things like differential skew requirements (less than 0.127mm), impedances, etc.

    You should be able to google it.
     
  34. QinX

    QinX Limp Gawd

    Messages:
    230
    Joined:
    Jan 15, 2015
    I'll look into making a draft of de PCB including a "optimal" computer case layout.

    I've made a PCIe Riser before and it worked fine. unless you are going to do some weird stuff, such as PLX chips or flexible risers you need to look out.

    I've gathered most of the important info from that last riser PCB.
     
  35. jb1

    jb1 Limp Gawd

    Messages:
    494
    Joined:
    Dec 27, 2014
    Nice, looking forward to seeing what you come up with! I guess we're assuming that Asrock will come through on proper bifurcation...
     
  36. QinX

    QinX Limp Gawd

    Messages:
    230
    Joined:
    Jan 15, 2015
    I've done some quick drafting in CAD to get a feeling at what kind of dimensions we are looking at.
    If you stick to a SFX PSU, Silverstone has a 700W version coming, I can stay just shy of 10 liters, a little smaller. Rough dimensions would be 35cm*30cm*9cm=9.54 liters

    You'd be able to run 2x GTX 980 ti without too much trouble.

    [​IMG]

    [​IMG]
     
  37. iFreilicht

    iFreilicht [H]ard|Gawd

    Messages:
    1,348
    Joined:
    Sep 23, 2014
    It's funny how a company like ASrock, that had the image of a cheapo subbrand of ASUS a few years ago is now the leading innovator in mITX boards.
    They are the ones we suspect to do things others don't have the guts for, and they deliver. Goes to show what you can make of a company if you put work and creativity into it.
     
  38. iFreilicht

    iFreilicht [H]ard|Gawd

    Messages:
    1,348
    Joined:
    Sep 23, 2014
    Why would you need the two additional risers if you're going with the Impact VII? Other than that, freaking awesome!
     
  39. QinX

    QinX Limp Gawd

    Messages:
    230
    Joined:
    Jan 15, 2015
    It's a placeholder board, I don't have a ASRock x99e-itx/ac 3D model. The 2 additional risers are there to space the PCIe brackets further away from the board, without some extra clearance the PCIe bracket might interfere with audio ports.
    The other option is to have a flexible riser like you are using in Brevis S and flip the GPUs. But then the question would be if the signal stays good, so at least for now I'm designing with this layout in mind.
     
  40. chemist_slime

    chemist_slime [H]ard|Gawd

    Messages:
    1,095
    Joined:
    Jul 24, 2005
    I've attached below some images of the 3 PCI-E risers that I've used in conjunction with the Asrock x99e-iTX that did not work. Hopefully some of the images will help. I have yet to take off the heatsink on the Supermicro one, will do that tomorrow in the evening when I get back from work.

    I have also PM'ed the tech support person who replied to the PCI-E bifurcation support on the x99e motherboard. He has read the PM but has yet to respond. Asrock support has not yet responded either. I will post results when they do. Let's do this!

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]
     
    D10S likes this.