PCIE Bifurcation

Discussion in 'Small Form Factor Systems' started by jb1, Jul 28, 2015.

  1. C_Payne

    C_Payne [H]Lite

    Messages:
    120
    Joined:
    Jan 6, 2017
    Mhh that's odd, I thought I tested that with my gigabyte x570 and the cards ended up in separate iommu groups, I will look into it as well
     
    marcosscriven likes this.
  2. C_Payne

    C_Payne [H]Lite

    Messages:
    120
    Joined:
    Jan 6, 2017
    Some thoughts on how to fix IOMMU grouping:

    1. Bios Update
    2. Kernel Update
    3. UEFI only!!! Try disabling CSM in BIOE
    4. ACS override patch (security!!!)
    5. maybe there is some BIOS settings concerning IOMMU/ACS
     
    Last edited: Dec 6, 2019
    marcosscriven and TheArkratos like this.
  3. marcosscriven

    marcosscriven n00b

    Messages:
    25
    Joined:
    Oct 31, 2019
    Thanks C_Payne - I had to change ACS from 'auto' to 'enabled'. In separate groups now.

    Code:
    IOMMU Group 28:
            0a:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev e7)
            0a:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
    IOMMU Group 29:
            0b:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP108 [GeForce GT 1030] [10de:1d01] (rev a1)
            0b:00.1 Audio device [0403]: NVIDIA Corporation GP108 High Definition Audio Controller [10de:0fb8] (rev a1)
    
     
    TheArkratos likes this.
  4. C_Payne

    C_Payne [H]Lite

    Messages:
    120
    Joined:
    Jan 6, 2017
    :) Great news, hope this is it now and everything works out.
     
    marcosscriven likes this.
  5. marcosscriven

    marcosscriven n00b

    Messages:
    25
    Joined:
    Oct 31, 2019
    I just wanted to call out publicly how impressed I’ve been with your help, and excellent service with sending a replacement riser, even when it turned the problem was actually with my motherboard.
     
  6. justjosh

    justjosh n00b

    Messages:
    5
    Joined:
    Dec 8, 2019
    Hey guys,

    I know this isn't SFF related but it seems to be the most popular thread on PCIe bifurcation so I'm just going to try my luck.

    Anyone managed to get the Asus Hyper M.2 to work with X79? I currently have a C602 DP board (Dual E5-2670s on X9DRi-F) that has the 4x4x4x4 bifurcation option in BIOS but am only seeing 1 drive in the OS.
     
  7. C_Payne

    C_Payne [H]Lite

    Messages:
    120
    Joined:
    Jan 6, 2017
    I have worked with a similar board: X9DRi-LN4F
    That one needed a BIOS update for bifurcation to work properly.
    I assume you have a Linux system available. 'sudo lspci -vv' will provide more insight. Especially the amount of root ports and the link capabilities section of those ports.
     
  8. justjosh

    justjosh n00b

    Messages:
    5
    Joined:
    Dec 8, 2019
    What am I looking for in lspci -vv? I have the latest BIOS. Do you mean a custom BIOS?
     
  9. C_Payne

    C_Payne [H]Lite

    Messages:
    120
    Joined:
    Jan 6, 2017
    You are welcome to post the output here. Have it set to x16 and check the root ports. They will show a LnkCap of 8GT, x16.
    If you set it to x4x4x4x4 you should see this drop to x4 and also there should be 4 root ports instead of 1 with LnkCap of x4.
     
  10. justjosh

    justjosh n00b

    Messages:
    5
    Joined:
    Dec 8, 2019
    Toggling the bifurcation correctly adjusts it to x4 but it only shows the first drive. The rest of them go missing.
    Code:
    04:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 (rev 01) (prog-if 02 [NVM Express])
            Subsystem: Samsung Electronics Co Ltd Device a801
            Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
            Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
            Latency: 0, Cache Line Size: 64 bytes
            Interrupt: pin A routed to IRQ 32
            Region 0: Memory at dfa00000 (64-bit, non-prefetchable) [size=16K]
            Region 2: Memory at dfa04000 (32-bit, non-prefetchable) [size=256]
            Capabilities: [40] Power Management version 3
                    Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                    Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
            Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+
                    Address: 0000000000000000  Data: 0000
            Capabilities: [70] Express (v2) Endpoint, MSI 00
                    DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 25.000W
                    DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                            RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
                            MaxPayload 128 bytes, MaxReadReq 512 bytes
                    DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
                    LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM not supported, Exit Latency L0s <4us, L1 <64us
                            ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                    LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                    LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                    DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR+, OBFF Not Supported
                    DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                    LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                             Compliance De-emphasis: -6dB
                    LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
                             EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
            Capabilities: [b0] MSI-X: Enable+ Count=9 Masked-
                    Vector table: BAR=0 offset=00003000
                    PBA: BAR=0 offset=00002000
            Capabilities: [100 v2] Advanced Error Reporting
                    UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                    UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                    UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                    CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                    CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                    AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
            Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
            Capabilities: [158 v1] Power Budgeting <?>
            Capabilities: [168 v1] #19
            Capabilities: [188 v1] Latency Tolerance Reporting
                    Max snoop latency: 0ns
                    Max no snoop latency: 0ns
            Capabilities: [190 v1] L1 PM Substates
                    L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- L1_PM_Substates-
            Kernel driver in use: pciback
            Kernel modules: nvme
    
     
  11. C_Payne

    C_Payne [H]Lite

    Messages:
    120
    Joined:
    Jan 6, 2017
    The interesting part is the root ports, not the device. You can pm the complete output of lspci with x16 Mode and x4x4x4x4 Mode and I will go through. Anyway I assume the bifurcation setting is not implemented correctly.
     
  12. justjosh

    justjosh n00b

    Messages:
    5
    Joined:
    Dec 8, 2019
    I've looked around, it seems like the overwhelming consensus is that the LN4F works but the F will not. Something about a faulty bifurcation implementation. Holding out hope for someone to counteract this.
     
  13. Bucheran

    Bucheran n00b

    Messages:
    3
    Joined:
    Nov 8, 2019
    Hello, I have bifurcation riser 8x8 and asrock x399 taichi motherboard, BIOS version 3.8, in this BIOS version there is an opportunity to switch to x8x8 mode, but the problem is that one video card works on x8 and the other one falls into pcie 2.0 x1, while x4x4 mode works great, How to solve this problem does anyone know?
     
  14. luckylinux

    luckylinux Limp Gawd

    Messages:
    206
    Joined:
    Mar 19, 2012
    I have a crazy request.
    I have a X9Dri-LN4f+ Motherboard and PCIe bifurcation seems to work just fine (tested with an ASUS Hyper M.2 adapter).
    My problem is that the GPUs take 2 slots each and then I'm left without any room for other devices. So I'd like to cable using some riser extensions to another chassis and use a "splitter" there.

    @C_Payne: Would it be possible to have a PCB in the shape of an ATX motherboard (although not as deep, i.e. just the part below the IO shield) where one could just slide the cards in without hassle, i.e. without having to turn 90° (all the PCIe slots, e.g. 8 slots, and then you have 2-3 entrance points for the "uplink" extension cables) ? Otherwise I'll have to do this using the RSC-R2U-2E8 or smth similar, although I guess it's going to be limited to PCIe 2.0 and it's going to be quite a hack with drilling holes etc.
     
  15. C_Payne

    C_Payne [H]Lite

    Messages:
    120
    Joined:
    Jan 6, 2017
    This sounds a bit like @marcosscrivens issues with his B450, you could try to raise a support ticket with asrock, they are quite helpful.
    What kind of riser/cables are you using? Maybe its a signal quality issue for the second card?!?

    This sounds like a project I can help you with. There is many variables still (type of cabling, singal conditioning etc.) Please drop a pm or write an email to risers@peine-braun.net
     
    TheArkratos likes this.
  16. Bucheran

    Bucheran n00b

    Messages:
    3
    Joined:
    Nov 8, 2019
    it seems that the problem was in the additional power supply 8-pin for bifurcation riser 8x8, I just forgot to connect it) and apparently because of this there was a burnout of the first three pins on the motherboard slot PCIE x16 and bifurcation riser or it can not be? the most interesting thing is that bifurcation riser continues to work only at a lower speed
     
  17. sdfewfe

    sdfewfe [H]Lite

    Messages:
    87
    Joined:
    Dec 16, 2015
    Does anyone here know of any options for attaching two or more M.2 PCIe SSDs to a single M.2 slot? Could be either through bifurcation (if any motherboards support M.2 bifurcation), or even better would be using a PCIe switch.
     
  18. C_Payne

    C_Payne [H]Lite

    Messages:
    120
    Joined:
    Jan 6, 2017
    Not that I am aware of.

    Ryzen does support x2x2 for the CPU NVME lanes, or at least I seem to recall that I have seen a slide once. Chipset lanes on different platforms may support setups like that as well, however BIOS support is a whole other story. I doubt any Manufacturer has this implemented.

    One option I see would be to use an m.2 to x4 adapter, a short x4 to x16 cable and one of the available x16 or x8 to 4x M.2 cards with an onboard packet switch.
    Like this one:

    https://www.amazon.de/dp/B07KG253NQ/ref=cm_sw_r_cp_apa_i_q6L-Db0QFR581
     
  19. sdfewfe

    sdfewfe [H]Lite

    Messages:
    87
    Joined:
    Dec 16, 2015
    Ah, that's a bit bulky for a small form factor. :/ I was hoping for something like an M.2-size card with two riser cables. Interesting to know that an switched solution exists for full-size PCIe though - perhaps as M.2 becomes more widespread, something will be released for that form factor too. Thank you for the info!
     
  20. ARNiTECT

    ARNiTECT n00b

    Messages:
    37
    Joined:
    Aug 4, 2012
    The low profile PCIe x8 Broadcom HBA 9400-16i 'Tri-Mode' seems to support 4 no. NVMe drives at x4 (8 no. at x2, and up to 24 no. with expanders).
    This card would normally be limited to PCIe x8 (there is a x16 version), but if connected into PCIe x4 slot from M.2 then I wonder if this works, but just limited to x4 speed.
    I have just ordered one from ebay to test.
    upload_2020-1-8_10-38-3.png

    Then M.2 to U.2 adapter, U.2 cable and U.2 to PCIe x4 adapter
    upload_2020-1-8_10-37-23.png upload_2020-1-8_10-39-0.png upload_2020-1-8_10-52-50.png

    or, M.2 to PCIe x4 adapter, with a x4 flexible riser cable
    upload_2020-1-8_10-45-45.png upload_2020-1-8_10-57-49.png

    Quite pricey solutions though.
     
    Neapolitan6th likes this.
  21. TheArkratos

    TheArkratos n00b

    Messages:
    49
    Joined:
    Nov 20, 2017
    I have used several M.2 to PCIe 4x risers, they tend to work quite well in my experience. I'm curious how that whole setup will work. Please let us know.
     
  22. ARNiTECT

    ARNiTECT n00b

    Messages:
    37
    Joined:
    Aug 4, 2012
    ETA of 9400-16i end of next week (from US to UK)
    Broadcom confirmed the HBA would work with a PCIe x4 electrical connection (x8 physical slot) but with max speed halved.
    I've successfully connected a U.2 Optane 905P to an M.2 socket using that 'red pcb M.2-U.2 adapter' and the cable supplied with the drive.
    I have the 'green pcb M.2 socket to PCIe x4 adapter' and x4 cable, but not tried yet, as I used one of C_Payne's x8x8 Bifurcation Risers instead.
     
    TheArkratos likes this.
  23. TheArkratos

    TheArkratos n00b

    Messages:
    49
    Joined:
    Nov 20, 2017
    btw if you want to skip a step in M.2 -> ribboned 4x slot you could get this: https://www.aliexpress.com/item/32860198563.html?spm=a2g0s.9042311.0.0.78db4c4djZarSS
     
  24. ARNiTECT

    ARNiTECT n00b

    Messages:
    37
    Joined:
    Aug 4, 2012
    TheArkratos likes this.
  25. speedst3r

    speedst3r n00b

    Messages:
    2
    Joined:
    Jan 2, 2013
    Eagerly awaiting my x8x4x4 riser, but it looks like my Asrock board isn't showing multiple root ports, even thoughit's been set in the BIOS... :( Should they still show up even if no cards are installed?

    Code:
    $ lspci
    00:00.0 Host bridge: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
    00:02.0 Display controller: Intel Corporation Device 3e96
    00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th Gen Core Processor Gaussian Mixture Model
    00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
    00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
    00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
    00:15.0 Serial bus controller [0c80]: Intel Corporation Device a368 (rev 10)
    00:15.1 Serial bus controller [0c80]: Intel Corporation Device a369 (rev 10)
    00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
    00:16.1 Communication controller: Intel Corporation Device a361 (rev 10)
    00:16.4 Communication controller: Intel Corporation Device a364 (rev 10)
    00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
    00:1b.0 PCI bridge: Intel Corporation Device a340 (rev f0)
    00:1b.6 PCI bridge: Intel Corporation Device a32e (rev f0)
    00:1c.0 PCI bridge: Intel Corporation Device a338 (rev f0)
    00:1c.1 PCI bridge: Intel Corporation Device a339 (rev f0)
    00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port 9 (rev f0)
    00:1e.0 Communication controller: Intel Corporation Device a328 (rev 10)
    00:1f.0 ISA bridge: Intel Corporation Device a309 (rev 10)
    00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
    00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
    00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-LM (rev 10)
    02:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04)
    03:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
    05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
    06:00.0 Non-Volatile memory controller: Device 1d97:1160 (rev b0)