PCIE Bifurcation

Discussion in 'Small Form Factor Systems' started by jb1, Jul 28, 2015.

  1. T_Uman

    T_Uman n00b

    Messages:
    2
    Joined:
    Oct 7, 2016
    Hello everyone

    I just wanted to say thanks to everyone who has been contributing to this thread, there is so much information in here that can't be found anywhere else. So I stumbled upon this thread earlier this week looking for this exact solution.

    First, my rig is an Ncase m1,asrock X99e-itx,1070 mini, 6800k. I'm looking to downgrade my gpu and make my machine a dedicated video editing and streaming machine. I want to install the new XFX RX 460 slim single slot and pair it with an internal single slot capture card and move the 1070 to a new rig. The problem with the RX 460 slim is that it does not have any power connection.

    My question is what powered riser card would you recommend for the job? I've looked online but couldn't find what looked like a powered riser card. If anyone has a similar setup let me know how well this works? I could easily purchase an external capture card but the less cable management the better.

    -ODB

    Edit: I see the RSC-R2UT-2E8R has a 6 pin cut out in the PCB however it's PCIE 2.0.

    Edit2: I found that there are powered riser CABLES that i could use for what i need. I found some good info on the process HERE. I'm pairing it with the ARC1-PERY423-C5 PCIE 3.0. I'll post on my progress mid Feb.

    Someone else posted looking for a single slot 6/8pin powered gpus. There is an upcoming gtx 1070 made by GALAX that is going to be single slot. There is also the MSI 1060 6gt OCV1 that can be modified to a single slot since all of its outputs are on the left side of the mounting bracket.
     
    Last edited: Jan 30, 2017
    Neapolitan6th likes this.
  2. alfaSZ

    alfaSZ n00b

    Messages:
    7
    Joined:
    Feb 18, 2016

    Nope, it's not a power delivery problem nor a signal problem - I am able to run caffe (or folding@home) under Linux for hours with both GTX1080 GPUs pegged at 100%, and the CPU too ;-). Am using the Amerirack PCie v3 version of their x8x8 riser - the only one that works as far as I can tell.
    This particular set of components (x99e-itx + 2x1080s + same riser) has never worked for me under Windows. Am pretty sure it is a DMA/IRQ issue, and in fact I am able to replicate it under Linux if I enable the hardware IOMMU (Vt-d). Am giving it another shot with the ASRock Taiwan guys and this time I am feeling quite hopeful they'll finally get to the bottom of it. Seeing this today at AnandTech (http://www.anandtech.com/show/11078...and-nightblade-pcs-with-intels-kaby-lake-cpus) only makes me more confident about the viability of the whole exercise:
    "It is interesting to note that instead of using a larger (say, microATX) motherboard with two PCIe 3.0 slots, in the case of the Aegis Ti3 MSI decided to use a Mini-ITX mainboard and then use a riser card with a PCIe switch to bifurcate the only PCIe 3.0 x16 slot between two cards."
     
  3. Hermit2001

    Hermit2001 [H]Lite

    Messages:
    114
    Joined:
    Aug 25, 2015
    Curiously, the mobo also has 4 sodium slots. Might be a custom board.
     
  4. fishell

    fishell n00b

    Messages:
    14
    Joined:
    Sep 27, 2016
    Hi MaximumBurrito, the splitter u r using is RSC-R2UT-2E8R. Ur gpus running at PCI-E gen 2.0 or 3.0? I ask this cuz I saw that card only support PCI-E Gen 2.0 on the supermicro website.
     
  5. fishell

    fishell n00b

    Messages:
    14
    Joined:
    Sep 27, 2016

    Hi Chris, I got to say I'm very interested :D.
    But I prefer a right turn, powered and dual slots for each pci-e slot riser card. (just can't find any single slot gtx1080 or 1070, and i realy don't want to mess up with those anoying DVI ports)
    maybe I asked to much. >.<
    Btw, nice work.
     
  6. T_Uman

    T_Uman n00b

    Messages:
    2
    Joined:
    Oct 7, 2016
    Interesting. I wonder if the Mobo can be purchased separately
     
  7. fishell

    fishell n00b

    Messages:
    14
    Joined:
    Sep 27, 2016
    I just contact the tech support ask the possibility of pci-e bifurcation on msi z270 itx mobo.
     
    Neapolitan6th likes this.
  8. C_Payne

    C_Payne [H]Lite

    Messages:
    78
    Joined:
    Jan 6, 2017

    Sorry i've been offline...

    I have already added Power Pads to my design.
    I consider building a right-angled one myself, however single slot... (so my dual gpus rest just above my motherboard/slim CPU waterblock)
    https://www.ekwb.com/shop/ek-annihilator
    My System will be so compact.:D:D:D:D:D:D

    I think I will start next week. I would share my design with you or even solder one for ya. PCB are pricey though... I paid around 100 dollars for 3, and if you need more surface area they will get even more expensive...

    BTW: Please do mess with those DVI Ports! It is great fun!:D
    I would just solder the dual-dvi one out instead of using a dremel.... so much cleaner. Maybe one could even put in a single slot one afterwards.
    Too bad EK doesnt make a waterblock for the Gigabyte ITX 1070...

    Cheers
     
  9. fishell

    fishell n00b

    Messages:
    14
    Joined:
    Sep 27, 2016
    Well thank u.
    About the waterblock for Ga 1070 itx, I bought one. It is not as good as EK, since it is the only one I can find made for GA1070 itx, I....can live with that.
    And Zotac just released GTX1080 itx ( PCB:177mm ), EK won't make a water block for that neither. Looks like they are not into making a waterblock for nvidia nano size graphics card. LOL
     
  10. C_Payne

    C_Payne [H]Lite

    Messages:
    78
    Joined:
    Jan 6, 2017
    Which one is it then? Couldnt find one... is it single slot?
     
  11. fishell

    fishell n00b

    Messages:
    14
    Joined:
    Sep 27, 2016
    Single slot yes, but I don't know is it available in your area.
    I'll PM you the brand and link, but I want share some pro&con with u first.
    1. The quality of this waterblock is ……, well it only cost me 73 USD, what else can i say. >.< (I don't know the price u gona get)
    2. Again, the brand is not attention to detail type, they usually make waterblock for everysingle graphics card they can get hands on. So this would be a good choice when u want a full cover waterblock for like GA1070 ITX or Zotac 1080 mini.
    3. According to the owner, "the engineering sample (GA 1070 ITX) we use has a little difference." so what happened next is i have to dremel a little bit off the copper cuz it interfere with the fan header on the 1070.
    4. U need to tell them cut the extra acrylic. U may ask "extra what?". Well, don't ask me, i don't know why a 170mm card has a 230mm(maybe?) long waterblock neither.
    extra.jpg

    5. at last, GA1070 ITX is really not a good choice if u try put two of this on your single slot bifurcation card, especially with acrylic top waterblock. Because the little backplate is really really hot.
     
  12. C_Payne

    C_Payne [H]Lite

    Messages:
    78
    Joined:
    Jan 6, 2017
    Riser.png

    Done with my right turn Design.
    I havent figured out the Edge-Connector to Slot 1 Dimension I will be going for yet.
    fishell u interested? what dimensions would you need?
    I could adapt it to your dimesions in no time and send you the .brd file.
     
    fishell likes this.
  13. fishell

    fishell n00b

    Messages:
    14
    Joined:
    Sep 27, 2016
    Riser.png

    Hi, Payne thanks, looks great! (Drooling........)
     
  14. alfaSZ

    alfaSZ n00b

    Messages:
    7
    Joined:
    Feb 18, 2016
    Update: I got a new UEFI from ASRock (3.64L) which finally allows both GTX1080's to work under Windows 10! I have been using the system for 2 days now and appears to be rock solid. Now solving the Vt-d issue under Linux is all that remains - I posted on the NVidia Linux driver forum, let's see if they are as helpful..
     
    StrawKite, fishell and Neapolitan6th like this.
  15. fishell

    fishell n00b

    Messages:
    14
    Joined:
    Sep 27, 2016
    Could you tell me who you contact? I tried to contact the tech support on their website 20days before, but no one replied.:(
     
  16. C_Payne

    C_Payne [H]Lite

    Messages:
    78
    Joined:
    Jan 6, 2017
    Would you mind sharing it? Or are you not supposed to?

    I am working on a third Design so I can put them on a big Panel and maybe safe some $$$.
    An order with your size would be around 180$ on OSH-Park. Same goes for my Dimensions.
    Be Patient! ;)

    Do you speak the language?... I am having trouble navigating that website.
    Maybe you can help me with an order there.

    I went crazy and desoldered the CPU IHS. Now with my CPU Block directly on Die I get the following Temps:
    i7 5820k @ 4.2 Ghz 1.2V
    R9 Nanos Power-Targets each +20%

    2x furmark, 2x prime95:
    water= ~70 Celsius
    GPU's= ~85 Celsius
    CPU = ~100 Celsius
    FAN at 90%

    2x Battlefield 1 Singleplayer:
    water= ~60 Celsius
    GPU=~70 Celsius
    CPU=~85 Celsius
    FAN at 75%

    20170211_150620.jpg
     
  17. fishell

    fishell n00b

    Messages:
    14
    Joined:
    Sep 27, 2016
    Did you receive the message I sent to you?
     
  18. fishell

    fishell n00b

    Messages:
    14
    Joined:
    Sep 27, 2016
    Well, I do speak the language. But I think u can try to reach the youtube channel "Declassified Systems" first, their builds use that brand waterblock a lot. I think it would be a good idea that ask that channel where to buy, cuz it will cost a lot ship from China to Europe or USA,but I'll try to help u if u need.

    wow wow w8 a minute, I'll pm u later.
     
  19. Neapolitan6th

    Neapolitan6th Gawd

    Messages:
    1,016
    Joined:
    Nov 18, 2016
    New gigabyte Z270 board announced with specific mention of PCIE Birfurcation ability on the product page.

    (Link)

    Pretty interesting... Of course you'd still have to find the right riser board for it though.
     
  20. alfaSZ

    alfaSZ n00b

    Messages:
    7
    Joined:
    Feb 18, 2016
    PM Sent
     
  21. C_Payne

    C_Payne [H]Lite

    Messages:
    78
    Joined:
    Jan 6, 2017
  22. heb1001

    heb1001 n00b

    Messages:
    9
    Joined:
    Aug 4, 2016
    Just caught up with the thread after a bit of a break.

    Seems there are a number of people with this working now and as I understand it there are two options for PCIe gen 3 support:

    1) Bifurcation with ameri-rack ARC1-PELY423-C5V3 splitter, (using 3.64L bios for asrock board). This gives 1x16 -> 2x8 so half the bandwidth but with no added latency.
    2) PLX chip with supermicro RSC-R2UG-A2E16-A splitter. This gives 1x16 ->2x16 so full bandwith for one card at a time but presumably has added latency from the PLX chip.

    Also C Payne's custom splitters which presumably perform like option 1 above and MaximumBurrito's success with RSC-R2UT-2E8R which (assuming it's PCIe gen 2) probably has the worst performance.

    Is it possible to get a fair performance comparison between the two options above to see which performs better in SLI?

    I'm thinking the GTX 1080 Ti is looking good for this BTW :) no HDMI port so it's single slot capable without modification.
     
  23. psychic_quark

    psychic_quark n00b

    Messages:
    6
    Joined:
    Aug 11, 2016
    Seeing some folks here report that they were able to purchase from ameri-rack, but... no such luck for me. Boy am I pissed at them.

    Ameri_rack_no_sale.png

    Seems like flexible bifurcated risers are out of the question for me.

    C_Payne : Do you have any of those PCB's remaining from your first run?

    Thanks!
     
  24. MaximumBurrito

    MaximumBurrito n00b

    Messages:
    7
    Joined:
    Oct 8, 2016
    heb1001, yes, it is a PCI-E 2.0 splitter, and it didn't fit in the case anyway, so I replaced it with the Ameri-rack splitter.

    psychic_quark, i was able to buy one after I talked with sales and reassured them that I wasn't going to steal their patent-pending design. They're just worried that someone else will produce their board, so you should be able to discuss that you're not going to reverse-engineer and re-sell it and you should be able to get it.
     
  25. MaximumBurrito

    MaximumBurrito n00b

    Messages:
    7
    Joined:
    Oct 8, 2016
    Ok, so it took some time (mostly waiting for the case to finally arrive), plus a hacksaw, a pair of garden shears, and two wine corks, but I was finally able to build my Dan A4 bifurcated dual Nano machine.

    IMG_4902.JPG

    IMG_4895.JPG

    IMG_4893.JPG


    This is small enough to fit behind my wife's 13" MacBook Air!

    IMG_4899.JPG


    Note that this is INCREDIBLY tight. It took multiple tries to get all the components into the case, and ended up requiring a very specific order of assembly to fit at all. Oh, and the Ameri-rack splitter had an edge that was running into the case, so it had to go.

    IMG_4878.JPG

    Aaaaand then the other side was running into the ethernet port housing on the Asrock board, so that had to go to.

    IMG_4879.JPG

    Uhhhh, and then it turned out that the other side still had too much board and was running into the RAM, so that's when the garden shears came out. The saw couldn't cut under the ends of the PCI-E slots like I wanted, but they could.

    IMG_4882.JPG

    Hey, no one has to see it! After that, it finally fit.

    IMG_4884.JPG

    So the moral of the story is that brutality can work when brains fail, I guess?

    IMG_4887.JPG

    Anyway, it works!

    Capture.JPG
    The skeptical reader may note that you could achieve a similar score with a single GTX 1080, but that's not as fun. More to the point, it doesn't quite work for my application: I'm developing some software where I want to be able to do graphics on one card and simultaneous computations on the other. Basically, this is a DIY Mac Pro.

    This particular setup, with one card facing vertically, was jb1's idea all the way back in August 2015, so credit where credit's due.
     
    Last edited: Mar 14, 2017
  26. psychic_quark

    psychic_quark n00b

    Messages:
    6
    Joined:
    Aug 11, 2016
    Thanks for the tip, let me give this a shot ONE more time since i haven't heard from C_Payne yet.
     
  27. jb1

    jb1 Limp Gawd

    Messages:
    495
    Joined:
    Dec 27, 2014
    Nice work getting this built!! I just received my A4 as well and was really tempted to try this idea, but having waited nearly 1.5 years I decided that I could wait a little longer and see what was around the corner with Vega-- I'm hoping that AMD produces an update to the R9 Nano.

    Since you actually have this up and running, I've got a question for you

    I purchased the same splitter as you did for this build, however, instead of getting one with a short length I got one that's 30cm long that would wrap around to the GPU side of the case. However, I'm concerned about crosstalk in riser since it's so long and each of the leads is not well isolated like the 3M one. Have you noticed any issues due to crosstalk in your shorter bifurcated riser? Which risers are you using to connect to the GPUs? It looks like one of them is the 3M one, but what is the black one?

    Thanks!
     
  28. Julio Espinosa

    Julio Espinosa [H]Lite

    Messages:
    71
    Joined:
    Feb 1, 2017
    You are INSANE!!!!!!!
    This world need more insane people, like Dan Hensen and you!
    Keep up the good work!
     
  29. betaflame

    betaflame n00b

    Messages:
    45
    Joined:
    Feb 1, 2017
    Noting that the ASRock Z270 Fatal1ty ITX board supports Bifurcation in the BIOS.
     
    Neapolitan6th likes this.
  30. MaximumBurrito

    MaximumBurrito n00b

    Messages:
    7
    Joined:
    Oct 8, 2016
    Thanks, couldn't have done it without the extensive development y'all put in!

    Per jb1's comments:

    Yeah I thought the same thing, but 1. there's no guarantee that AMD will issue an updated Nano (though I'd snap it up in a second if they did!), and 2. it would need to be absolutely NO bigger than the current one or there is no possible way it would fit in the case.

    The wine corks propping up the bottom of the second card may give the impression that there is room under the card, but there really isn't: the 8-pin PCI-E power plug barely has room to clear the SSD bracket at the bottom of the case. Even if the bracket were removed completely, it wouldn't free up much space. Any future cards fitting into this case would really need to be Nano-sized or smaller!

    No issues with crosstalk. It's quite stable. I've ran Firestrike, Heaven, Valley etc. and the only thing that crashes it is Furmark, but I think that's an issue with the PCI-E bifurcated Crossfire setup: it just crashes on startup, not due to any instability.

    The riser I used is a Lian Li PW-PCI-E38-1 380mm riser, which is rather expensive but seems to be quite high quality. The two things that ended up being crucial about this riser don't have to do with stability.

    The card end of the riser is at an angle, which allowed me to thread it through to the other side of the case after cutting the GPU shroud provided with the case:

    IMG_4896.JPG

    and the cables of the riser are split into 4-lead chunks instead of bundled into a single cable, which allowed me to spread them out flat in order to close the top of the case:

    IMG_4886.JPG

    You can see the riser looping through from the other side of the case right next to the main 24-pin motherboard power connector.
     
  31. sungho

    sungho n00b

    Messages:
    14
    Joined:
    Nov 4, 2016
    Is one 120mm radiator enough to cool overclocked two R9 Nano & CPU?
    If one can handle, it will be possible to squeeze your setup using ek-annihilator & 25mm thick radiator inside A4-SFX.
     
  32. C_Payne

    C_Payne [H]Lite

    Messages:
    78
    Joined:
    Jan 6, 2017
    I do... however I ran out of x16 Slots to solder...

    U interested?

    Also my right turn PCB's came in:

    Waiting for my Annihilator to come in... Will Post Pictures and test results then. 1490076002241-1922142794.jpg
     
  33. psychic_quark

    psychic_quark n00b

    Messages:
    6
    Joined:
    Aug 11, 2016
    Hi C_Payne. Thanks for replying, I did decide to take a second look at my layout and realized that inflexible risers would just not fit in, so please count me out.

    BUT your x8 + dual m.2 bifurcation riser seems great for another build I am doing. Count me in for that if you ever decide to produce it.
     
  34. russell2000

    russell2000 n00b

    Messages:
    2
    Joined:
    Apr 4, 2017
    Hey everyone, I've decided to make an account as I've also been searching for a bifurcation solution for a while now--although not for GPUs. Up until finding this thread, I kept running into forum posts of people asking about it and the post then being closed once someone told them that the BIOS must support it. My mobo definitely supports bifurcation (z9pe-d8 ws) but I haven't been able to find the physical hardware to let me plug in multiple cards.

    So I have two (2) UAD-2 PCIe 2.0 x1 DSP cards. I basically want to be able to split one of the x16 slots on the motherboard into two x1 slots. Although since the bifurcation on the mobo only goes down to x4, I suppose something that did x8x4x4 or x4x4x4x4 would be fine. Having read through the thread as well as browsed countless websites, this http://aerocooler.com/rc1-pely423-c...flexible-riser-one-pcie-x16-to-dual-pcie-x16/ seems like it's my best bet but I can't get excited about spending $70 for something that's pretty un-ideal. (I've become disabled so I'm not as moneyed as I was when I bought all this stupid stuff :/ )

    Secondly, and this is more of a shout-out to others that may come across this thread as they look for a solution to their problem, I feel like there is a large interest building in PCIe-to-m.2 solutions. Especially once Sandy Bridge-EP and Naples hit; lots of pci lanes and not enough room on the motherboard. There are options provided by HP and Dell but they both appear to be BIOS locked to the specific vendor's workstations.

    C_Payne, I don't know what sort of design/production skills you have but it looks like you're doing some crazy advanced stuff. Since it looks like you're already doing stuff with multiple m.2 slots I guess I'll just say that more people than you expect might be very interested.
     
  35. C_Payne

    C_Payne [H]Lite

    Messages:
    78
    Joined:
    Jan 6, 2017
    Would you be interested in one of my Risers?
    I have my first version laying around... I would give it away for $20 plus shipping... dont need it anymore.
    I Could also make an new one, the right angled version for you for $50 plus shipping.

    I dont know how performance dependent your dsp cards are... maybe this could be a solution:
    https://www.aliexpress.com/item/PCI...lgo_pvid=a75011fc-2519-4522-ba9b-dfbc6de37179
    And since it has a packet switch it should be compatible with all mainboards.
    Downside is that both DSP would share their gen2 x1 Bandwidth.

    Chris
     
  36. russell2000

    russell2000 n00b

    Messages:
    2
    Joined:
    Apr 4, 2017
    Hey Chris,

    Thanks for the reply. I've seen those PCIe to USB splitters around but I was never really sure that they'd work for me. Additionally, I know a bunch of this stuff is from random OEMs anyway, but AliExpress terrifies me as I have no idea what the worst possible failure scenario would be. If just the little board dies that's fine I guess, but if it fries the cards or the rest of the system I would literally cry. Additionally, I'm not sure where one would then put the PCIe cards; I can't find a chassis that isn't meant for industrial-grade PCI expansion.

    As far as your offer, it seems like the best way to get my feet wet. The left-angled one would be ideal for my case, but I don't want you to feel like you're giving something away. I'm happy to pay whatever the fair value is +shipping (I assume somewhere between $20 and $50). I'll PM you and reply with the details when you're ready :).

    Russell
     
  37. Javier Gonzalez

    Javier Gonzalez n00b

    Messages:
    2
    Joined:
    May 10, 2017
    I am in the process of setting up a small server with the Asrock x99e-itx and I wanted to ask if there is some experience with Vt-d and PCIe card passthough using ESXi using the bifurcation cables.
     
  38. Diverge

    Diverge Gawd

    Messages:
    536
    Joined:
    Nov 22, 2005
    I bought one of C_Payne's riser/splitters with the intent on using it with ESXi, passing through 2 HBA cards to 2 different VM's. I haven't gotten around to testing it yet, but will reply back when I do.
     
  39. C_Payne

    C_Payne [H]Lite

    Messages:
    78
    Joined:
    Jan 6, 2017
    I am using KVM and iommu works splendidly. Both of my GPU's end in different IOMMU groups.
    I dont see why there should be issues with ESXI. Do you have the lates 3.64 bios?

    Chris
     
  40. Javier Gonzalez

    Javier Gonzalez n00b

    Messages:
    2
    Joined:
    May 10, 2017
    I do. Still waiting for some parts. Will post once I try.