Ryzen and IOMMU groups. Is this ever going to get fixed?

hondaman

The OG Hondaman
Joined
Jul 31, 2004
Messages
79
Or am I just pissing in the wind?

Yes, I get separate groups for each video card, but everything else (SATA, LAN, Audio, add-in cards) is thrown together with everything else.

NOTHING EXCEPT VIDEO IS ISOLATED, which makes this nearly worthless.

Is this an AMD problem? An x370 problem? An Asrock problem? Tell me which cloud this old man should yell at.

Ryzen 1700/Taichi/Arch Linux with or without the ACS patch.
 

TType85

[H]ard|Gawd
Joined
Jul 8, 2001
Messages
1,491
This is why I went from Ryzen back to Intel. My E5-2667 V2 is nearly as fast and the IOMMU grouping even on a crappy Lenovo S30 workstation I have it in is better than the Ryzen system. My hope is that Threadripper is better.

EDIT: I had a 1700x on both a Asus B350 and Asrock Taichi X370. The X370 was the better of the two.
 

hondaman

The OG Hondaman
Joined
Jul 31, 2004
Messages
79
Ive come to the conclusion that the Taichi is falling way behind in the IOMMU scene. I bought it initially because it has the features I want and was working with ram that other boards were having trouble with. I still love this board, but after seeing other IOMMU groups from gigabyle and especially Asus, I went ahead and bought a Asus ROG Crosshair IV Hero.

Here's hoping for better luck. Lord knows I've spent WAY too much time screwing with this.
 

Sufu

[H]ard|Gawd
Joined
Sep 3, 2006
Messages
2,009
Try to set IOMMU grouping from "Auto" to "Enabled" in the BIOS. I've read Auto causes issues with the grouping.
 

gigaxtreme1

2[H]4U
Joined
Oct 1, 2002
Messages
3,577
It appear that a solution is not the responders/transponderers vested interests.
"crickets"
 

/dev/null

[H]F Junkie
Joined
Mar 31, 2001
Messages
15,190
This is why I went from Ryzen back to Intel. My E5-2667 V2 is nearly as fast and the IOMMU grouping even on a crappy Lenovo S30 workstation I have it in is better than the Ryzen system. My hope is that Threadripper is better.

EDIT: I had a 1700x on both a Asus B350 and Asrock Taichi X370. The X370 was the better of the two.
I'm on an E5-2690 and I definitely need working groupings. My S30 was $175 and my E5-2690 was about $150 some time back. vt-d works great. I have 1 VM with 2 port pci-nic, and another with a video card.
 

TType85

[H]ard|Gawd
Joined
Jul 8, 2001
Messages
1,491
I'm on an E5-2690 and I definitely need working groupings. My S30 was $175 and my E5-2690 was about $150 some time back. vt-d works great. I have 1 VM with 2 port pci-nic, and another with a video card.

My s30's only problems are it broke one of the mounts for the HSF and it doesn't like using the PCI slot with PCIe cards in other slots. I tried to use a PCI video card with 2 other graphics cards and a HBA and got PCI resource errors. The IOMMU grouping is perfect. We will see tonight how it is on a HP Z420.
 

/dev/null

[H]F Junkie
Joined
Mar 31, 2001
Messages
15,190
My s30's only problems are it broke one of the mounts for the HSF and it doesn't like using the PCI slot with PCIe cards in other slots. I tried to use a PCI video card with 2 other graphics cards and a HBA and got PCI resource errors. The IOMMU grouping is perfect. We will see tonight how it is on a HP Z420.

That would be fantastic. Please update here or PM me with how your Z420 experiment goes. I don't have any PCI cards so I haven't run into what you are seeing....

My biggest fear in jumping on consumer level stuff is mentioning issues like linux, vt-d, iommu groupings, etc to gigabyte/msi/asrock and support coming back with "Linux isn't compatible with our motherboard" or "we don't support linux" when it's a bios issue. Since I've run into that several times in the past 5 years, I now wait to read about reports in forums like this before making my purchases.
 

TType85

[H]ard|Gawd
Joined
Jul 8, 2001
Messages
1,491
That would be fantastic. Please update here or PM me with how your Z420 experiment goes. I don't have any PCI cards so I haven't run into what you are seeing....

My biggest fear in jumping on consumer level stuff is mentioning issues like linux, vt-d, iommu groupings, etc to gigabyte/msi/asrock and support coming back with "Linux isn't compatible with our motherboard" or "we don't support linux" when it's a bios issue. Since I've run into that several times in the past 5 years, I now wait to read about reports in forums like this before making my purchases.

Z420, one PCIe graphics card, one USB 3 card
Code:
IOMMU group 0
   [8086:0e00] 00:00.0 Host bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 DMI2 (rev 04)
IOMMU group 1
   [8086:0e02] 00:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04)
IOMMU group 2
   [8086:0e04] 00:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04)
IOMMU group 3
   [8086:0e08] 00:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04)
IOMMU group 4
   [8086:0e20] 00:04.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04)
   [8086:0e21] 00:04.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04)
   [8086:0e22] 00:04.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04)
   [8086:0e23] 00:04.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04)
   [8086:0e24] 00:04.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04)
   [8086:0e25] 00:04.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04)
   [8086:0e26] 00:04.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04)
   [8086:0e27] 00:04.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04)
IOMMU group 5
   [8086:0e28] 00:05.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04)
   [8086:0e2a] 00:05.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS (rev 04)
   [8086:0e2c] 00:05.4 PIC: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC (rev 04)
IOMMU group 6
   [8086:1d3e] 00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port (rev 05)
IOMMU group 7
   [8086:1d3a] 00:16.0 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #1 (rev 05)
   [8086:1d3c] 00:16.2 IDE interface: Intel Corporation C600/X79 series chipset IDE-r Controller (rev 05)
   [8086:1d3d] 00:16.3 Serial controller: Intel Corporation C600/X79 series chipset KT Controller (rev 05)
IOMMU group 8
   [8086:1502] 00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (rev 05)
IOMMU group 9
   [8086:1d2d] 00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 05)
IOMMU group 10
   [8086:1d20] 00:1b.0 Audio device: Intel Corporation C600/X79 series chipset High Definition Audio Controller (rev 05)
IOMMU group 11
   [8086:1d12] 00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 2 (rev b5)
IOMMU group 12
   [8086:1d18] 00:1c.5 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 5 (rev b5)
IOMMU group 13
   [8086:1d14] 00:1c.6 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 3 (rev b5)
IOMMU group 14
   [8086:1d16] 00:1c.7 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 4 (rev b5)
IOMMU group 15
   [8086:1d26] 00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 05)
IOMMU group 16
   [8086:244e] 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
   [11c1:5811] 09:05.0 FireWire (IEEE 1394): LSI Corporation FW322/323 [TrueFire] 1394a Controller (rev 70)
IOMMU group 17
   [8086:1d41] 00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (rev 05)
   [8086:2826] 00:1f.2 RAID bus controller: Intel Corporation C600/X79 series chipset SATA RAID Controller (rev 05)
   [8086:1d22] 00:1f.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Host Controller (rev 05)
IOMMU group 18
   [10de:1c03] 05:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] (rev a1)
   [10de:10f1] 05:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1)
IOMMU group 19
   [8086:1d6b] 02:00.0 Serial Attached SCSI controller: Intel Corporation C602 chipset 4-Port SATA Storage Control Unit (rev 05)
IOMMU group 20
   [1b73:1100] 07:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev 10)
IOMMU group 21
   [104c:8241] 08:00.0 USB controller: Texas Instruments TUSB73x0 SuperSpeed USB 3.0 xHCI Host Controller (rev 02)
IOMMU group 22
   [8086:0e80] 7f:08.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 (rev 04)
IOMMU group 23
   [8086:0e90] 7f:09.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 1 (rev 04)
IOMMU group 24
   [8086:0ec0] 7f:0a.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 0 (rev 04)
   [8086:0ec1] 7f:0a.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 1 (rev 04)
   [8086:0ec2] 7f:0a.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 2 (rev 04)
   [8086:0ec3] 7f:0a.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 3 (rev 04)
IOMMU group 25
   [8086:0e1e] 7f:0b.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04)
   [8086:0e1f] 7f:0b.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04)
IOMMU group 26
   [8086:0ee0] 7f:0c.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
   [8086:0ee2] 7f:0c.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
   [8086:0ee4] 7f:0c.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
   [8086:0ee6] 7f:0c.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
   [8086:0ee8] 7f:0c.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
IOMMU group 27
   [8086:0ee1] 7f:0d.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
   [8086:0ee3] 7f:0d.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
   [8086:0ee5] 7f:0d.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
   [8086:0ee7] 7f:0d.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
   [8086:0ee9] 7f:0d.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
IOMMU group 28
   [8086:0ea0] 7f:0e.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
   [8086:0e30] 7f:0e.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
IOMMU group 29
   [8086:0ea8] 7f:0f.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers (rev 04)
   [8086:0e71] 7f:0f.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers (rev 04)
   [8086:0eaa] 7f:0f.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
   [8086:0eab] 7f:0f.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
   [8086:0eac] 7f:0f.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
   [8086:0ead] 7f:0f.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
IOMMU group 30
   [8086:0eb0] 7f:10.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 (rev 04)
   [8086:0eb1] 7f:10.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 (rev 04)
   [8086:0eb2] 7f:10.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 (rev 04)
   [8086:0eb3] 7f:10.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 (rev 04)
   [8086:0eb4] 7f:10.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 (rev 04)
   [8086:0eb5] 7f:10.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 (rev 04)
   [8086:0eb6] 7f:10.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 (rev 04)
   [8086:0eb7] 7f:10.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 (rev 04)
IOMMU group 31
   [8086:0e1d] 7f:13.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04)
   [8086:0e34] 7f:13.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04)
   [8086:0e81] 7f:13.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Registers (rev 04)
   [8086:0e36] 7f:13.5 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring (rev 04)
IOMMU group 32
   [8086:0ec8] 7f:16.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 System Address Decoder (rev 04)
   [8086:0ec9] 7f:16.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
   [8086:0eca] 7f:16.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
 

hondaman

The OG Hondaman
Joined
Jul 31, 2004
Messages
79
I installed my ASUS Crosshair IV and checked the IOMMU groups. It was still worthless, nearly (if not completely) the same as on my Taichi. Now I saw a guy on reddit post his groups (which is why I bought this board) and it was nothing like mine. So what's the answer? You have to add "multifunction" to the acs override line so that it looks like this, "pcie_acs_override=downstream,multifunction"

I didn't see that in any documentation anywhere. I found it in 2 comment sections (reddit and level1tech). Anyway I have now passed through to win10 my 1070, a sata controller, usb controller and just finished running the unigine heaven benchmark. Some fine tuning needs to be done, but I think I'm on my way. Hopefully.
 

Sufu

[H]ard|Gawd
Joined
Sep 3, 2006
Messages
2,009
I installed my ASUS Crosshair IV and checked the IOMMU groups. It was still worthless, nearly (if not completely) the same as on my Taichi. Now I saw a guy on reddit post his groups (which is why I bought this board) and it was nothing like mine. So what's the answer? You have to add "multifunction" to the acs override line so that it looks like this, "pcie_acs_override=downstream,multifunction"

I didn't see that in any documentation anywhere. I found it in 2 comment sections (reddit and level1tech). Anyway I have now passed through to win10 my 1070, a sata controller, usb controller and just finished running the unigine heaven benchmark. Some fine tuning needs to be done, but I think I'm on my way. Hopefully.

Are you running with npt=0? Do you have any of the performance issues that the other threads have been talking about?
 

hondaman

The OG Hondaman
Joined
Jul 31, 2004
Messages
79
Is there a benchmark you'd like me to try? I haven't tried playing any steam games yet. I nearly broke out the champagne after running the heaven benchmark. As of right now I don't have NPT turned off. I'm reading into all the fine tuning stuff now. All I want in the end is a windows 10 KVM installation that will play games relatively respectably.
 

Sufu

[H]ard|Gawd
Joined
Sep 3, 2006
Messages
2,009
Is there a benchmark you'd like me to try? I haven't tried playing any steam games yet. I nearly broke out the champagne after running the heaven benchmark. As of right now I don't have NPT turned off. I'm reading into all the fine tuning stuff now. All I want in the end is a windows 10 KVM installation that will play games relatively respectably.

I think that's the issue. If you did some benchmarks, your FPS would usually be lower than an equal bare metal machine due to issues with npt and passing through GPUs. With NPT disabled, you'd gain GPU performance but then CPU performance suffers quite badly FWIR. I think it varies a lot per user and per program, but no one seems to have a 100% system compared to an Intel rig. I'd have built a nice KVM machine by now otherwise using Threadripper.
 

hondaman

The OG Hondaman
Joined
Jul 31, 2004
Messages
79
I think that's the issue. If you did some benchmarks, your FPS would usually be lower than an equal bare metal machine due to issues with npt and passing through GPUs. With NPT disabled, you'd gain GPU performance but then CPU performance suffers quite badly FWIR. I think it varies a lot per user and per program, but no one seems to have a 100% system compared to an Intel rig. I'd have built a nice KVM machine by now otherwise using Threadripper.
Yeah, I'm ok with degraded performance. As long as it's playable. And that's subjective too. Whatever it takes I suppose, I simply won't use win10 as my primary desktop whatever the cost.
 

Sufu

[H]ard|Gawd
Joined
Sep 3, 2006
Messages
2,009
Yeah, I'm ok with degraded performance. As long as it's playable. And that's subjective too. Whatever it takes I suppose, I simply won't use win10 as my primary desktop whatever the cost.

It varies per game. I would definitely do some testing to make sure it's OK for you. I'm hoping this bug gets fixed soon; I'd love to virtualize like arcade machines and my HTPC.
 

hondaman

The OG Hondaman
Joined
Jul 31, 2004
Messages
79
Well I have "native" SATA, USB, sound and video now. Briefly played a few games at 1440p and it sure seemed good enough to me initially. I ended up passing through an add-on PCI-E 1x marvell sata controller, a PCI-E sound card (giving me 5.1 sound), an onboard USB controller and of course the nv 1070. The network is using virtio for now. I'll give it a decent gaming session and come to a final conclusion.
 
Top