• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

IOMMU

R

rtangwai

[H]ard|Gawd
2FA
Joined
Jul 26, 2007
Messages
1,369
I'm getting ready to build a bare-metal hypervisor (need to learn Xen and ESXi) and I discovered that I don't own any CPU's that support IOMMU (that's what I get for sticking with Intel). I've been pricing a Core i7 3770 (non-K) and motherboard, they are a bit high for my taste as this is strictly for learning not a production machine. Anyone know what's the performance like for the various current AMD CPUs (FX4100, FX6100, FX8xxx) in VMs? Specifically what's the overhead loss due to the bare-metal hypervisor? What motherboards support IOMMU?

I will be using ESXi 5.1 so I would prefer a supported CPU/motherboard combination. I don't have any spare DDR3 RAM that I trust (they are all 1.65V) so I will be buying RAM as well, which is one less variable for me to worry about as I can buy whatever I need. I'm assuming a minimum of 16GB if I intend to make more than 2 VMs (which is actually unlikely as I will be also running Hyper-V on my Windows 2008 Server R2).
 
I have an old Athlon II X2 240 kicking around that I intend to use just to validate the platform before I go full-steam ahead on the CPU. I'm thinking about the FX-8350 - part of the learning is to build a gaming VM (I have a HD6950 waiting on the bench) so I will need a fair bit of power to keep up with that GPU. I'm quite surprised how cheap the AMD CPU's are compared to their Intel VT-d counterparts.

What's the performance hit like? Is it noticeable or something that only a benchmark would show?

Gigabyte builds good boards and the one you are using has lots of expansion slots which avoids the IOMMU problem of where to plug in all the hardware you are allocating. I only wish it had more LAN ports. Are you having any compatibility problems with the Realtek LAN or audio chipsets in ESXi 5.1? I was looking at the ASUS M5A97 R2.0 which is a bit cheaper but has nowhere near the expansion flexibility of the Gigabyte. How many VMs do you run?
 
Depends what the guests will do.

I don't think raw cpu performance matters all that much in most cases. I run proxmox on linux/kvm and it works fine with a p2-965 and an older Xen x3330.

I run mostly linux servers that have no gui however. YMMV.
 
I second what Robstar stated. More often than not, if something is slowing my performance in VM instances, it ends up being a disk access limitation and not the cpu being unable to keep up.

Intel cpus wear the overall performance crown right now and for the forseeable future, but the current AMD cpus come into their own when running multithreaded workloads. Combined with the iommu features, they work very well for running multiple vms on a single box.
 
I have a FX-8120 in that same Gigabyte GA990FXA-UD3 motherboard.

IOMMU works beautifully.
 
I'm planning on moving my Windows 2008 Server to a VM (although I'm a bit concerned about the AD controller being virtual instead of physical), set up another VM with Exchange 2010, a third one with Ubuntu Server 12.04 LTS (it hosts my website, blog, and I'm going to put in forums and a wiki), and perhaps Citrix in order to play with streaming applications in XenApps/XenDesktop. I would love to set up a SAN for the VMs but that starts getting a bit pricey. The gaming VM is strictly for learning exactly *HOW* to make IOMMU work as my personal gaming computer is way more powerful. Obviously this will swamp a dual-core so I'm debating whether to go 6 or 8 core. 16GB of RAM is probably the minimum, I should probably think about 32GB.

Can the bare-metal hypervisor share the NIC among the VMs or will I have to install more NICs? While I'm leaning towards the Gigabyte motherboard I might want to spring for something with 2 NICs. I'm also not thrilled about the Realtek chipset, I prefer Intel.
 
Last edited:
rtangwai said:
I'm planning on moving my Windows 2008 Server to a VM (although I'm a bit concerned about the AD controller being virtual instead of physical), set up another VM with Exchange 2010, a third one with Ubuntu Server 12.04 LTS (it hosts my website, blog, and I'm going to put in forums and a wiki), and perhaps Citrix in order to play with streaming applications in XenApps/XenDesktop. I would love to set up a SAN for the VMs but that starts getting a bit pricey. The gaming VM is strictly for learning exactly *HOW* to make IOMMU work as my personal gaming computer is way more powerful. Obviously this will swamp a dual-core so I'm debating whether to go 6 or 8 core. 16GB of RAM is probably the minimum, I should probably think about 32GB.

Can the bare-metal hypervisor share the NIC among the VMs or will I have to install more NICs? While I'm leaning towards the Gigabyte motherboard I might want to spring for something with 2 NICs. I'm also not thrilled about the Realtek chipset, I prefer Intel.
Click to expand...
Or a nice quad port NIC and have a 1gb port for each VM?
I would get 2 x 8 sticks, and if you need more, then get another 2.
8 core of course. :)
 
Hagrid said:
Or a nice quad port NIC and have a 1gb port for each VM?
I would get 2 x 8 sticks, and if you need more, then get another 2.
8 core of course. :)
Click to expand...

You could do that with VSwitches, but if you want to direct I/O Forward the NIC to a guest, usually everything in the same pcie slot goes to the same guest.
 
Zarathustra[H];1039818405 said:
You could do that with VSwitches, but if you want to direct I/O Forward the NIC to a guest, usually everything in the same pcie slot goes to the same guest.
Click to expand...

Hmmm, that means a dual/quad NIC wouldn't work for direct I/O. Good thing that Gigabyte board has so many PCI-E slots.

Are *ALL* the devices on the Gigabyte board (audio, NIC, USB, etc.) seen natively by ESXi 5.1 or will I have to build some sort of custom install for it? I've heard some horror stories about having to create an entire ESXi installer for specific hardware and I'd really like to avoid that on a first try - I've never built a bare-metal hypervisor before, just guest OS's using Virtualbox, VirtualPC, VMplayer, etc. I haven't tried Hyper-V either although I assume it's similar to Virtualbox and such as it does sit on a host OS and not bare-metal.
 
rtangwai said:
Hmmm, that means a dual/quad NIC wouldn't work for direct I/O. Good thing that Gigabyte board has so many PCI-E slots.

Are *ALL* the devices on the Gigabyte board (audio, NIC, USB, etc.) seen natively by ESXi 5.1 or will I have to build some sort of custom install for it? I've heard some horror stories about having to create an entire ESXi installer for specific hardware and I'd really like to avoid that on a first try - I've never built a bare-metal hypervisor before, just guest OS's using Virtualbox, VirtualPC, VMplayer, etc. I haven't tried Hyper-V either although I assume it's similar to Virtualbox and such as it does sit on a host OS and not bare-metal.
Click to expand...

The NIC definitely works, but for anything server related, I'd recommend staying away from the on-board. I only use mine for the ESXi management interface.

USB works too, though I should caution that USB forwarding can be VERY slow in ESXi, unless you forward the entire USB interface using Direct I/O.

As far as Audio goes, I have no clue at all. Mine is strictly a server box, so I have never even tried. I don't even know if ESXi has any audio compatibility at all, or if it has to be Direct I/O passed through to the guest. Never tried it.
 
The onboard NIC works fine for me.
I tested GPU passthrough and that worked, however I didn't test audio. I will test that tonight after work.
The USB 3.0 devices were also available for passthrough.
In terms of performance, the only bottleneck in my system is the single 1TB hard drive, which I am ok with as it is a budget test box. If I cared a whole lot I would throw a SSD in either for staging or cache.
 
Zarathustra[H];1039819514 said:
The NIC definitely works, but for anything server related, I'd recommend staying away from the on-board. I only use mine for the ESXi management interface.

USB works too, though I should caution that USB forwarding can be VERY slow in ESXi, unless you forward the entire USB interface using Direct I/O.

As far as Audio goes, I have no clue at all. Mine is strictly a server box, so I have never even tried. I don't even know if ESXi has any audio compatibility at all, or if it has to be Direct I/O passed through to the guest. Never tried it.
Click to expand...

I know about the slow USB which is why I'm buying a PCI-E USB 3.0 card to dedicate it to one VM (it records cable TV via a Hauppauge HD-PVR and also does my voicemail/faxing thru a USB modem on a POTS line - I know, I know, learn Asterisk).
 
benzino_86 said:
The onboard NIC works fine for me.
I tested GPU passthrough and that worked, however I didn't test audio. I will test that tonight after work.
The USB 3.0 devices were also available for passthrough.
In terms of performance, the only bottleneck in my system is the single 1TB hard drive, which I am ok with as it is a budget test box. If I cared a whole lot I would throw a SSD in either for staging or cache.
Click to expand...

Does dedicating entire hard drives to each VM help or is it a bottleneck because of the shared SATA controller? I have a number of spare 750GB hard drives (left over after updating my fileserver) which I could use for this, although I'll probably have to switch to a bigger case to hold everything (I have a Lian-Li PC2001B acting as my current fileserver and it can hold 15 hard drives).

Someone told me I could make a SAN *INSIDE* a VM so that the entire server storage system could be made available to all the VMs at once. Wouldn't I need some kind of RAID controller for that, something far better than the onboard?
 
Update:

Bought the 8350, Gigabyte 990FXA-U5, and 32GB of Corsair Vengeance RAM, plus a small case (Fractal R2 XL) and an Antec Earthwatts Green 750W PSU. Had a minor heart attack when ESXi stated that my system didn't support IOMMU - forgot to turn it on in the BIOS. The stock fan is way too loud even in the Fractal (which has sound dampening panels), ended up installing my old Hyper 212 EVO which turns out to block the first RAM slot so I'll have to fit a thinner fan.

I haven't tried AMD since my Athlon64 3000+ days. For a CPU that a lot of people trash-talk it seems to be pretty snappy to me and I use a 3570K every day as my workstation. I haven't even *THOUGHT* about overclocking yet (even though it has an unlocked multiplier and a decent HSF) as I want to keep my problems to a minimum as I try to figure out the ESXi thing.

Quick question: how do I identify which SATA controller and which USB controller is which? I'm planning on doing direct I/O to the internal Marvell SATA controller but there are 2 of them (one is for the eSATA) and I can't figure out which is which in ESXi. Same for the USB - there are 2 USB 3.0 controllers listed, can't figure out which controls what.
 
If im not mistaken the normal windows device manager should show which port is used once USB device is inserted?
 
rtangwai said:
Quick question: how do I identify which SATA controller and which USB controller is which? I'm planning on doing direct I/O to the internal Marvell SATA controller but there are 2 of them (one is for the eSATA) and I can't figure out which is which in ESXi. Same for the USB - there are 2 USB 3.0 controllers listed, can't figure out which controls what.
Click to expand...

Probably trial and error :(

I'm really curious about your experience with setting this all up. I've always wanted to do a whitebox ESXi server at home, but getting compatible parts was always such a hassle. AMD seems to have it right with this IOMMU support. I would like to migrate my bare metal unraid server into a VM and separate out all the services (sickbead/sab/couch/sql) into ubuntu server VMs. Picking up a 990FX mobo, 8350 and RAM is a fairly cheap investment to get onto VMware.
 
I've gotten the ESXi host to do mostly what I want. It can't run OS X VMs, can't handle my USB modem for TAM, and the USB passthough performance sucks. On the other hand the datastores turn out to be fast enough to keep up with my HD-PVR for recording HD cable and strangely enough the shared USB is fast enough to keep up with the HD-PVR (whereas the passthrough could not).

This particular hardware combination installed very easily in ESXi 5.1 as it recognized everything but the onboard sound (ESXi never recognizes sound). I didn't have to install drivers for the NIC, the extra SATA controller, etc. It was in fact *TOO* easy - hard to practice making custom installers when the default works pefectly.
 
That's good news. I'll have to do some research on the Asrock or MSI 990FX boards, I'd prefer to stay away from Gigabyte (personal preference).
 
Dephcon said:
That's good news. I'll have to do some research on the Asrock or MSI 990FX boards, I'd prefer to stay away from Gigabyte (personal preference).
Click to expand...

Yeah, I wouldn't have chosen Gigabyte either, but the fact that I could get both my FX-8120 AND the GA-990FXA-UD3 at Microcenter for $200 was a little difficult to pass on.
 
The UD5 actually looks pretty good on paper, it has up to (3) PCI-E x16(x8) and (2) x16(x4). Would be ideal for me since I have two LSI controllers and will probably need at least 1 quad Broadcom NIC to get started.

The Asrock Extreme 4 only has two x16 and one x4 for about the same price.
 
That's why I got it instead of the UD3. Be careful adding PCI-E NICs - the PCI-E x1 slot closest to the CPU is blocked by the chipset heatsink unless you use a *REALLY* short card.

I might take another crack at quad-fire - unlike my ASUS P8Z77-V LK board it has PCI-E lanes to burn and it was the x4 slot that bottlenecked me the last time I tried 4 GPUs. Of course I can't do that under ESXi so I'll probably either multiboot or install another hard drive w/Windows 8.
 
Dephcon said:
The UD5 actually looks pretty good on paper, it has up to (3) PCI-E x16(x8) and (2) x16(x4). Would be ideal for me since I have two LSI controllers and will probably need at least 1 quad Broadcom NIC to get started.

The Asrock Extreme 4 only has two x16 and one x4 for about the same price.
Click to expand...

You could also look at the Asus 990FX Pro 2 and see if it will fit your needs. It has official ecc support as well!

Link

Typically it's about $130-$140
 
theinquirer is reporting that IOMMU v2.5 is essential for HSA support, and logically this will be included in Kaveri (or its chipset?), but where does this leave what is technically AMD's 'premium' platform, AM3+?

My 890FX is IOMMU compliant, and the 990FX series carry this feature too, but will that extend to v2.5 and thus enable HSA support with some future HSA enabled AMD GPU?
 
Any one have this board and use ECC?

Looking at the memory QVL for Asus 990FX Pro 2 I see no ECC memory. I wounder if that ECC is a typo.

If any on has used ECC on this board, please list model used.

Thanks!
 
Last edited:
flexcore said:
Any one have this board and use ECC?

Looking at the memory QVL for Asus 990FX Pro 2 I see no ECC memory. I wounder if that ECC is a typo.

If any on has used ECC on this board, please list model used.

Thanks!
Click to expand...

They don't always publish ECC support in the QVL.

My Gigabyte GA-990FXA-UD3 has no documented ECC support. I have never tried it, but others in this forums have successfully used ECC, and had it properly recognized.

(I really should switch over to ECC though)

So, unfortunately it's a lot like IOMMU, it's a feature that you kind of get with trial and error in consumer hardware.
 
R3MF said:
theinquirer is reporting that IOMMU v2.5 is essential for HSA support, and logically this will be included in Kaveri (or its chipset?), but where does this leave what is technically AMD's 'premium' platform, AM3+?

My 890FX is IOMMU compliant, and the 990FX series carry this feature too, but will that extend to v2.5 and thus enable HSA support with some future HSA enabled AMD GPU?
Click to expand...

Well, firstly, HSA is really only for integrated graphics though, right? I don't think it was ever intended to work with addon GPU's, as then data still has to travel over a relatively slow PCIe bus, negating the benefits of HSA.

Secondly, we all know AM3+ is not long for this world. I'm not sure what is going to replace it for non-APU systems, or if it even will be replaced, but of the current sockets, the only one with a future beyond the next generation is FM2, as I understand it.
 
I wasn't too keen on ASUS for this build because of their complete lack of support for VT-d on their Intel Z77 boards - I didn't want to take any chances that it was an ASUS trend on all their motherboards. My whitebox is for ESXi lab purposes and Gigabyte was highly recommended due to its known compatibility with various ESXi versions and relatively low cost. For a production host I wouldn't go whitebox at all, for a Windows or Linux workstation I probably would have gone ASUS or MSI.
 
flexcore said:
Any one have this board and use ECC?

Looking at the memory QVL for Asus 990FX Pro 2 I see no ECC memory. I wounder if that ECC is a typo.

If any on has used ECC on this board, please list model used.

Thanks!
Click to expand...
I'm sure it's supported since I can confirm that it works perfectly fine with a crosshair IV (890FX) and V (990FX), see the kernel logs with some crucial ecc unbuffered modules :
Code: 
EDAC MC: Ver: 2.1.0
 [    5.165740] AMD64 EDAC driver v3.4.0
 [    5.165771] EDAC amd64: DRAM ECC enabled.
 [    5.165781] EDAC amd64: F15h detected (node 0).
 [    5.165819] EDAC MC: DCT0 chip selects:
 [    5.165821] EDAC amd64: MC: 0:  2048MB 1:  2048MB
 [    5.165822] EDAC amd64: MC: 2:  2048MB 3:  2048MB
 [    5.165823] EDAC amd64: MC: 4:     0MB 5:     0MB
 [    5.165824] EDAC amd64: MC: 6:     0MB 7:     0MB
 [    5.165825] EDAC MC: DCT1 chip selects:
 [    5.165826] EDAC amd64: MC: 0:  2048MB 1:  2048MB
 [    5.165828] EDAC amd64: MC: 2:  2048MB 3:  2048MB
 [    5.165829] EDAC amd64: MC: 4:     0MB 5:     0MB
 [    5.165831] EDAC amd64: MC: 6:     0MB 7:     0MB
 [    5.165832] EDAC amd64: using x4 syndromes.
 [    5.165833] EDAC amd64: MCT channel count: 2
 [    5.165896] EDAC amd64: CS0: Unbuffered DDR3 RAM
 [    5.165898] EDAC amd64: CS1: Unbuffered DDR3 RAM
 [    5.165900] EDAC amd64: CS2: Unbuffered DDR3 RAM
 [    5.165901] EDAC amd64: CS3: Unbuffered DDR3 RAM
 [    5.165973] EDAC MC0: Giving out device to 'amd64_edac' 'F15h': DEV 0000:00:18.2
 [    5.166051] EDAC PCI0: Giving out device to module 'amd64_edac' controller 'EDAC PCI controller': DEV '0000:00:18.2' (POLLED)
 
Zarathustra[H];1039852464 said:
Well, firstly, HSA is really only for integrated graphics though, right? I don't think it was ever intended to work with addon GPU's, as then data still has to travel over a relatively slow PCIe bus, negating the benefits of HSA.

Secondly, we all know AM3+ is not long for this world. I'm not sure what is going to replace it for non-APU systems, or if it even will be replaced, but of the current sockets, the only one with a future beyond the next generation is FM2, as I understand it.
Click to expand...

seems strange for a business that creates high end platforms and high-end add-in gpu's to bet the farm on a technology that is only relevant to their low-margin mass market products......?

i would hope so, i'd like to see steamroller on FM2+ with PCIe 3.0 and excavator on FM4 with DDR4 and SATA4, but, we still haven't heard anything other than there being yet one more AM3+ CPU after piledriver.
 
If it was ECC RDIMM then I would be super interested as i have a ton of 8/16GB RDIMMS kicking around. ECC UDIMM is meh.
 
flexcore said:
Any one have this board and use ECC?

Looking at the memory QVL for Asus 990FX Pro 2 I see no ECC memory. I wounder if that ECC is a typo.

If any on has used ECC on this board, please list model used.

Thanks!
Click to expand...

I have ECC on it. Works fine. I use FreeBSD:

Code: 
[root@zfs ~]# dmidecode -t memory
# dmidecode 2.11
SMBIOS 2.7 present.

Handle 0x0028, DMI type 16, 23 bytes
Physical Memory Array
	Location: System Board Or Motherboard
	Use: System Memory
	Error Correction Type: Multi-bit ECC
	Maximum Capacity: 32 GB
	Error Information Handle: Not Provided
	Number Of Devices: 4

Handle 0x002A, DMI type 17, 34 bytes
Memory Device
	Array Handle: 0x0028
	Error Information Handle: Not Provided
	Total Width: 72 bits
	Data Width: 64 bits
	Size: 8192 MB
	Form Factor: DIMM
	Set: None
	Locator: DIMM0
	Bank Locator: BANK0
	Type: DDR3
	Type Detail: Synchronous Unbuffered (Unregistered)
	Speed: 1333 MHz
	Manufacturer: Kingston     
	Serial Number: CE03706
	Asset Tag: AssetTagNum0
	Part Number: 9965525-055.A00LF 
	Rank: 2
	Configured Clock Speed: 667 MHz
 
You must log in or register to reply here.
Back
Top