Haswell virtualization with discrete graphics

[crABtoad]

Hey all, sorry if this come across like I'm a neophyte. I run linux as my base OS these days, made the switch after the NSA thing (I have used unix systems for some time now, and I'm an accomplished noob with them). I kinda miss playing games though, and I'd like to sort out VGA passthrough in regard to DirectX.

So I found this guide for running a Xen-based gaming rig a couple years ago, bought a compatible motherboard (ASRock Z77E-ITX), CPU (3770 non-K), and then promptly forgot about it until recently. The other route seems to lie with KVM; I saw a youtube video where someone was able to get a Heaven benchmark running fullscreen with KVM. Direct3D always crashed when I tried virtualizing that in VBox, so they made it at least one step further. As far as I'm aware, launching Heaven is a good proof-of-concept for launching a game, and if one works the other should.

It seems like it works, I just wanted to know if anyone around here had any testimonials, recommendation on one way vs the other, or any particular pitfalls to watch out for.

My other 2 questions would be:

a) In terms of software security (which I'm not totally well-versed in), how isolated would a Win7 virtual machine be? I ideally want to keep it locked in a box with no information about myself or the host system. If I go the Xen route, will I have to run Windows as Dom0? Does this have any further ramifications regarding security? This is actually my biggest concern with having Windows installed anywhere near my hardware these days. A few months ago I thought it was silly to be overly concerned with security, but times' changed.

b) My next rig is going to be a Haswell gaming ultrabook, and all the Haswell CPUs I've looked at have VT-d enabled at the chip, and they all seem to use HM87 (which has VT-d enabled). So it should just work, right? In hindsight this seems like a silly question, because it should work if it works on a desktop.

Yeah, sorry again if this come across as being somewhat simple to you all...just trying to find some info before I scrap my current OS.

 

[GaryS]

Not sure if this answers your question since I'm using VMWorkstation.
But, I've had reasonable success in recently setting up a Win7 guest within a CentOS 6.4 host.

My biggest concern was getting my 3 screen display working (I do a lot of writing/design work and the 3+ monitors are absolutely the way-to-go).

I went with SuperMicro and a FirePro GPU.
And, I'm not using CAD or CGI right now within Win7 currently. Mostly design tools such as Rational, Word, and various open-source 'static' design tools.

I have not tested 'pass-through' with ESXi on my FirePro graphics card...

VM Workstation does provide password protection for the guest image. So, you can add a second level of security. You could also try using additional security layers via secure file-system for the VM image itself.

 

[crABtoad]

I was planning on having the VM files installed to a mounted, hidden truecrypt volume (volume inside a volume). My concern would be if I had Windows and Debian running side-by-side in Xen, or nested as a KVM...how much information does Windows have about Debian, or what's in Debian? I honestly don't know, and this is why I was asking.

I think I was mostly looking for success stories on VGA passthrough for mainstream discrete GPUs. I know that pro cards have a lot more success for passthrough, but that's a luxury I can't afford. I saw a thread around here that linked to modding GTX 6xx cards into their GK104 pro counterparts, but I won't be able to do that on a laptop.

VM Workstation seems like it needs SoftXpand for VGA passthrough? Provided I could run Linux as my "primary" OS, and just play games in Windows while keeping it in the dark about Linux I'd be happy to pay for the software.

I guess my questions are a bit muddled maybe, but I'm interested in running at least 3 virtual machines:

Debian/Arch (primary),
Debian/Qubes/TAILS (TOR),
Windows 7/8 (completely untrusted, just plays games, has passthrough for discrete graphics).

I'm really not terribly picky on which way to do it provided it's reliable, effective, and keeps Windows in the dark.

 

[GaryS]

Please follow-up if you get anywhere with a good passthrough config on the graphics.

I've given up on high frame-rate games (FPS, etc.). I'll re-visit though with the fall release schedule.

Last time I built a gaming machine it was quickly 'provision' out to needy relatives... (just like the toy factory - never build a new computer prior to first day-of-school!).

 

[crABtoad]

Seems like people have been having some good results with Xen server, I'm going to look into XCP as an open-source alternative to Xen.

Good thread here, if anyone's curious. I'm going to do some research for a couple weeks, get a new graphics card (gave my last card away), and see what I can get running.

 

[GaryS]

Thanks for sharing the link.

The physical workstation has an advantage with this configuration in that it's not out-of-the-question to put an extra graphics card in the machine (or use hypervisor console) for "desktop" access into the the other guests not benefiting from direct GPU. And, most monitors have at least 2 input ports - thereby mitigating the need for a physical KVM switch.

The laptop is at a loss though. Once that "primary" guest starts up with direct-access to the ONLY graphics card... (assuming) - that's all it gets. If that control is somehow lost/corrupted then it sounds like a complete re-do for the platform.

Balancing the risk of failure - I was considering this sort of portable setup but use a cube-like (light-weight) portable without a keyboard nor folding (clam shell) screen. Power-on would boot-up the primary guest image which would then grab control of screen, keyboard, mouse. The other guests would then follow but be controlled by the primary (envisioning some sort of hyper-visor console). Ideally this would all go into just the one phat labtop... but, not sure if this is feasible.

 

[unholythree]

Xenserver passthrough has been working pretty well for me with a few caveats:

GPU pass-though in my case was setup through XenCenter (easy, just a drop down menu) with a single 6950 videocard in the machine.

When the machine first boots it goes to the XenServer shell, after I start the Windows 7 guest that has pass-through assigned, the video card restarts, this is where IOMMU comes in. When the video card restarts it's hidden to the Xenserver Dom0 and now under the control of the guest. With this single GPU setup once you start the guest with pass-though there no way to hand the GPU back to the Dom0, but you can SSH into the Dom0 shell if you want.

The part that seems to be most difficult to people is passing-though a USB controller: as far as I can tell the best method is to use is PCI pass-though through the command line. Any PCIe device can be passed though individually but legacy PCI can only be passed through as the entire PCI bus. I give my Windows 7 guest the onboard USB3 controller a PCIe device; unfortunately this hides the device at boot so it only ever usable to the guest.

I get occasional video artifacting in my guest, usually right after the guest boots. I also get some popping with the sound from the video card, I haven't tried passing though the motherboard's onboard sound since I plan to get a DAC.

It isn't too hard with the right hardware but your Windows guest is going to loose some hardware level control you might be use to. With this setup there isn't TRIM if you have a SSD, there is no way to monitor motherboard sensors (temperature, or fan control), anything not passed through simply doesn't exist as far as the guest knows.

 

[crABtoad]

Thanks for sharing unholy, I've heard some complaints that Xen heavily favours AMD graphics, and that's definitely something I'm going to have to work around. I prefer Nvidia for a couple reasons (namely better drivers, more efficient cards with Kepler, and virtually all the high-end Haswell portables run Nvidia cards. You lost me a bit with SSH...Do you mean that you can shell into the Dom0 from the windows guest, and then use the graphics card as a resource there?

I would be okay with losing some hardware and putting up with a couple nuisance-level bugs if it were to work otherwise. I'm actually really curious how the USB passthrough would work on a laptop, that's got me a little nervous. I would be looking at getting a USB Amp/DAC as well. Honestly, I'm more concerned about the USB controller/passthrough than I am about the graphics card after doing some quick initial research here and there.

Provided I can use the Intel HD series IGP for Linux and run the discrete card in Windows, I'll be happy. I'm running Mint and a Debian VM on an Intel HD4000 right now, and it's completely fine for everything. Really anything I do in Linux is either processor-bound or it's just web-browsing.

You've both been really helpful so far, thanks a bunch!

 

[unholythree]

Do you mean that you can shell into the Dom0 from the windows guest, and then use the graphics card as a resource there?!

Once the GPU has been given to the guest afaik there is no way to give it back to Dom0 until you restart the entire host, but you can always just use Putty or whatever to access it though SSH.

The only difference is the console is red instead of blue :

I generally manage the host through a headless XP guest with XenCenter that I RDP into or if I totally bork something I have XenCenter on my HTPC just in case.