VT-D/VGA Passthrough?

L

LstBrunnenG

Supreme [H]ardness
Joined
Jun 3, 2003
Messages
6,676
So apparently I fell asleep for a few years. In that time, hardware IOMMUs became relatively commonplace.

http://www.youtube.com/watch?v=Gtmwnx-k2qg

Sadly, I also understand my 980X does *not* support it :(

Has anyone else played around with this tech? How much of a performance hit is running an actual game on the guest vs bare metal? Will LGA 2011 require extra motherboard support to run a guest with VGA passthrough? I can't find any LGA 2011 motherboards that mention VT-D support.
 
That may be because none of the first batch of LGA2011 CPUs supported Vt-d (which, quite honestly, I think is a joke). That is coming in the next stepping/revision.
 
DuronClocker said:
That may be because none of the first batch of LGA2011 CPUs supported Vt-d (which, quite honestly, I think is a joke). That is coming in the next stepping/revision.
Click to expand...
It was a last minute thing though. If they hadn't found that bug, it would have been in C1. I'd be surprised if the motherboard makers would have had time to react.
 
I do some passthrough on Xen, haven't tried VGA. Just want to say, don't assume things are just going to work. Assume they won't and be pleasantly surprised when they do.
 
Passthrough video is still wonky on Xen and ESXi. The tech is progressing, but it's not really ready for primetime yet. For the systems and cards that it works on, performance is near-native.
 
Hi !

Here is VGA passtrough setup working :p



Uploaded with ImageShack.us

I run Debian (linux 3.1.0, xen 4.1.2) as host OS, FreeBSD and Windows are virtualized

With Vt-d i am able to passtrough the SB800 sata controller to freeBSD for 1Go/s 5*1To RaidZ

For windows i passtrough this :
- USB 3 controler
- HD 5850
- Asus Xonar D2

I am able to play lot of games,but not BF3, i think it is server 2008 R2 related

Note that UVD is not working on Vt-d with HD5000 but with HD6000
 
Last edited:
Asus crosshair 4 formula
Phenom x4 @ 6core @ 4Ghz
4*4 GB Gskill 1600 CAS 9

you need an ATI card, hd6000 is better if you need UVD, a Vt-d or AMD iommu enabled chipset AND BIOS, like 890FX or all 900 chipset from ASUS or GIGABYTE

on intel it is harder since the chipset (x58, x79) has Vt-d but not the bios on a lot of mobo
 
Good to know HD6000 series works on Xen. It still has issues on ESXi.
How is gaming performance on virtualized Windows, Avlin?
 
Very good, like native

No time for benchmark now, will post more later ;)

It brings a lot of time, but once you have understand it is easy.
 
@Avlin: I'm very curious about your setup, as I've been thinking about merging my home server and HTPC in a single setup. Have you encountered any major issues (random BSODs, etc.) with the HD5850 VGA passthrough?
 
@CEpeep

like native ! now disk and network can suffer if not optimised. As you can see I use PV driver, my average ping on BF2 is 20 ms

@LstOfTheBrunnenG

On x58, it is working on gigabyte and Intel Board, Asus has a bios that enable Vt-d but it is buggy

On x79 it should be OK, but i think you should use an Intel board to be sure. New asus board should be good, you have to buy&try :p

@CKTurbo128

- If your virtual machine crashes your hypervisor is not affected
- If you use an HD5000, UVD is buggy.
- If your GPU is crashing, the main OS (xen) is not affected.
- I have only 1 GPU, so once the GPU is passed trough, the only way to manage my hypervisor is distant ssh. If Windows crash, i have to bring another PC and make some ssh comand to reboot the VM
- perfs are native, Windows see the true card, ther is no emulation, I doesnt encoutered any Vt-d relative bug
- BF3 is not launching, but it need a crapy dll fixe, no time for that
- you can run 2 windows and 2 card, 1 windows = 1 card
- crossfire may not working, since the VM dont see the chipset
- but bi-gpu ? I think yes, need money and time :eek:

Please understand that my english is buggy like a windows millenium , dont know if you undertsand what i say :D
 
Avlin:

Could you paste your Windows cfg file? Just the meat of it - grep -v "^#" | grep -v "^$" please.
Also, could you paste what you did in /etc/modprobe.d/*, and, if you had to, anything you did in /etc/initramfs-tools/modules (just paste it).
Also, how are you dealing with keyboard/mouse? Are you passing through a USB controller?

I'd ask in a PM, but I think the answers would be contributive to the thread. Thanks in advance.

EDIT:

And I assume you cannot use the Xen VNC port on the host to connect to the display.

EDIT2:

Erm... forgive my stupid questions, I can see your script in the upper left, you do it differently from what I had in mind.
 
Last edited:
Ok, first it is great that you are interested, i'll post below what is needed.I plan to make a blog,but lack of time.

First you need to have some basic linux experiment.

1. Check out Vt-d or IOMMU option in your bios
2. Install Debian 6.0.4 stable and no other OS :eek: (i have 6.0.3 testing, so it is the same)
edit : use testing package, to get xen 4.1.2 and linux 3.1.0 or newer

3. install xen package and only that, no virtmanager, python bla blah blah
4. list what u want to passtrough with lspci and lspci -n
5. hide your pci card in /etc/default/grub :

GRUB_CMDLINE_LINUX="xen-pciback.hide(00:11.0)(07:00.0) etc..."

6. update-grub and reboot
7. put config files in /etc/xen/MyVM.cfg like on screen below

8. make an ethernet bridge caled br0
9. see screen below : my windows get an HD5850, a xonar 2 and nec USB 3, my nas on freebsd get the whole SATA controller in raidz :cool:

10.a hide all pci used from dom0 OS (see first screen)
10b. launch your VM with xm create MyVM.cf and monitor it with xm top

Nothing else is changed, no modprobe blablah

My nas is pure rockstable SSH, no vnc ! It uses 3 Mo of ram and deserve my Windows by NFS.I give to my nas 6 GB of ram for insane ram cahing. got 5 * 1 To caviar green

the only bad thing is 30MB/s between vm; even pv driver

The only way for usb or mouse is whole USB3 controller passtrough, no time for dealing with USB over network or USB 1.0 emulation by qemu



Uploaded with ImageShack.us

;)
 
Last edited:
Whoa, you've even got RAID-Z going on? Awesome build, Avlin. Thanks again for posting it here and sharing the details.
 
yes , pv driver on FreeBSD and Windows

i can have 300MB/s between dom0 and FREEBSD, but not from windows to freebsd

maybe the mtu ? i have 1500 in windows
 
I ran into the 30MB speed limit myself. I got 3 NICs on my dom0 - 2 onboard and 1 extra pci card and I just passthrough both my onboard NICs to two VMs where I care for speed.

What controller are you passing through to FreeBSD? I gave up trying to passthrough my Marvell 88SX7042, maybe it works natively but not passed through. Works fine in a PV OpenMediaVault (basically Debian based FreeNAS) though.
 
i get 300MB/s from domU Freebsd to dom0

but 1 domU to another is 30MB/s

but i have tried now and i get 100MB/s :confused:

it must be random with CPU core affinity in XEN, since the wiki says you should reserve 1 core for network.

but as you can see in my lspci i have an Intel T350 dualport :D but no time to manage it now. It will permit traffic between VM with 0% CPU overhead.
 
The stuff you added to grub kernel command line - that's xen-pciback driver and you're using pci-stub. Typically in 2.6.x specific xen kernels xen-pciback is compiled in so grub command line is how you pass options to it, but I see you're using 3.1 upstream where it's a module. Check your dmesg it could be complaining that it doesn't know what you're asking it to do with that, but, regardless, that's not the driver you're using for passthrough. You're using pci-stub which is different. My guess is if you remove that stuff from grub it'll still work, I don't think it does anything.

You can use xen-pciback to passthrough stuff to PV domUs, you'll have to put that stuff in /etc/modprobe.d/ though. I had to blacklist the original module responsible for the device on dom0, or I guess you could probably manually unbind like you're doing it. If you do it right xen-pciback is what shows up as the driver when you list lspci -vvv.

I think there're just general problems with Intel VT-d I see people passing through all sorts of stuff on AMD no problem, for me hardware passthrough with an i5-2400 on a Q67 chipset has been hit and miss. Thankfully I have PV passthrough to fallback on, but that's not an option for VMWare people.
 
Last edited:
you are right, no need to pass grub command line

but also no need of modprobe hack

On Intel you need intel board and x58 or newer lga2011 steping on amd you can do iommu with 140$ budget (sempron 130 + cheap mobo)

The controller that i pass i the SB800 sata controller, works fine

EDIT : running correctly Folding at home with FAHCore 16 in virtualized Windows, but GPUz says it is pcie 16x 2.0 @ 16x 1.1

is iommu breaking pcie speed ? :mad:
 
Last edited:
Is VT-d needed, or can you use VT-x for this? I'm running Hyper-V right now, and RemoteFX performance is only about 25% of native. I'd love for it to be quicker.
 
TeeJayHoward said:
Is VT-d needed, or can you use VT-x for this? I'm running Hyper-V right now, and RemoteFX performance is only about 25% of native. I'd love for it to be quicker.
Click to expand...

You need VT-d or IOMMU for PCI-E passthrough. On Xen you can also do PV (paravirtualized) passthrough for certain devices without having VT-d, but I don't believe it works in Windows with video cards.
 
I have HyperV for my company's main's server and my workstation , because we are 100% windows, but it is a pure s..., using it every day is so annoying ! :(

HyperV has poor disk IO perfs, the difference between Xen is oustanding (no benchmark but real life of developper day to day work )

I don't want to troll. but working on a HyperV virtual machine every day since 1 year is hard. HyperV is not so bad, but i will not include in a new setup, as well as @home pr0n seedbox or @work exchange server

I think that for a new project you should go with ESXI or xen or KVM. I lost 30 minutes to my boss today because HyperV's remote console was crashing while my xen at home can do FAH on hd5850 + freebsd zfs + another debian vm with audio card passtrough on same computer :p

From what i know, disable remote FX, it is a no go, and use remote desktop on W7 client, it is the main advantage of virtualized windows desktop.
 
Avlin said:
I have HyperV for my company's main's server and my workstation , because we are 100% windows, but it is a pure s..., using it every day is so annoying ! :(
Click to expand...
Did you ever get a chance to run a benchmark comparing native to virtualized? I'm debating swapping out my 2500K for a 2500 and switching over to Xen, but I'd like to see something showing just how close to native 3D performance is before I drop $200 on a plaything.
 
no time for benchmark now, but you asked, so i'll try :)

on passed-trough HD5850 i get 3 minutes per frames on folding at home, sames as native
 
Anyone know if the AMD FX6100 has support for this?

Toying with the idea of running my media PCs in one box instead of mutiple...
 
nry said:
Anyone know if the AMD FX6100 has support for this?

Toying with the idea of running my media PCs in one box instead of mutiple...
Click to expand...

It's all in the motherboard with AMD equipment. If your motherboard has a 970 or 990 chipset, it will work. If your motherboard has a 890FX chipset, there may be a BIOS update to support it.
 
Iv got a AMD 760G, guess it may be time to switch boards!

Is it a stupid idea to have a system like this used for all the media PCs

Would be a FX6100
16GB DDR3
Dual gigabit lan to file server

Just be 1080p playback so would need DXVA hardware acceleration!
 
Working VGA passthrough is my holy grail for my home virtual server setup.

Right now I have my VM host as ESX 5 with 16GB ram
My current VM's are:
OpenSolaris/Napp-it VM with 8 port LSI SATA card passed through
Windows XP for some older apps
Windows 7 Pro (.NET development machine)
Server 2k8 R2 (SQL/Web dev machine)

I would love to roll my Windows 7 HTPC into this box as it is currently about 4 feet from the TV.

So far I have the ATI 5450 video card passed through a windows 7 vm (both HDMI & Video passed, 2GB ram assigned to the VM) and last night any time I tried to enable the card it would BSOD. I was playing with it a bit here from work and for giggles I enabled the ATI card and it didn't BSOD. No monitor is plugged in to it at the moment so we will see when I get home if it works when I plug the monitor in.

I started playing with Xen last night and holy@#%# I am over my head in Linux, I have a lot to learn. Is there any graphical front end that pulls all the configuration in one place?

Enough rambling.
 
What free VM software should I be looking at for passthrough?
Is there such a thing lol
 
nry said:
What free VM software should I be looking at for passthrough?
Is there such a thing lol
Click to expand...

Xen is what others in this thread are using. It is open source and completely free.

TType85 said:
I started playing with Xen last night and holy@#%# I am over my head in Linux, I have a lot to learn. Is there any graphical front end that pulls all the configuration in one place?
Click to expand...

XenServer comes with its own web-based GUI, but it doesn't have a GUI option for all of Xen's features. The best thing to do is just get comfortable with the Linux command line. It won't take you that long if you have solid sys admin experience with another OS. The core Xen stack evolves quickly, and the GUIs are developed by different teams who can't always keep up with the pace of Xen's development.
 
CEpeep said:
Xen is what others in this thread are using. It is open source and completely free.
Click to expand...

This. I installed it on Fedora 16, just watch out how you set your drive up or you won't have space for your VM's :p
 
CEpeep said:
XenServer comes with its own web-based GUI, but it doesn't have a GUI option for all of Xen's features. The best thing to do is just get comfortable with the Linux command line. It won't take you that long if you have solid sys admin experience with another OS.
Click to expand...

I'll have to look closer at it. I think my main thing is figuring out the passthrough. I need to pass the OI VM the sata card and my HTPC the video card.

It would be nice if I can have Xen on the OI (zfs) install, but I don't know if thats possible.
 
You must log in or register to reply here.
Back
Top