ESXi 5 + VT-d + GT520 HDMI issue

D

dga

n00b
Joined
Jan 2, 2012
Messages
14
This seems the most appropriate forum to ask this question, hopefully someone will have been here too.

I've got a Shuttle SH67H3 + i7 2600 running ESXi 5 and a Zotac GT 520. Everything seems to be working fine, the Passthrough is showing up and recognised in the Gentoo VMs with lsmod. Issue I have is there's no video output no matter what I do.

When I switch over to the Nvidia-drivers package I lose the console in vSphere and there is no output still. I've ensured that the BIOS boots with just the onboard Intel 2000 and I've verified that there's a working output on boot from the Nvidia card.

Having never done this it feels like I'm missing something basic. The idea is I wan't to run XBMC as a dedicated VM, currently its used as a Plex Media Server for 5 iOS devices, PMS for two PS3's and a sandbox for tinkering.

So had anyone done XBMC via VT-d with a GT520?
 
As far as I know, I haven't heard of anyone getting vga passthrough under esxi working with nvidia. (I'd love for someone to correct me on this). VGA passthrough is really pretty new as is (and completely unsupported of course), but the limited success stories I've seen, namely on the vmware boards, focus on amd passthrough. Don't ask me why that's the case, because I honestly don't know. At the end of the day though, I have tested various 5, 6, and 7 series amd cards and gotten them working in esxi 5, so I can attest to at least that side of things.
 
Hmm, maybe I could swap the Nvidia for an AMD, it will fit a HP Microserver which is a bonus I suppose.

So AMD = ATi of old nowadays? My other option is to just buy an Apple TV 2 as I have Plex running just fine on all the other clients.
 
Do you have both the VMWare video card and the Nvidia one detected in the VM?

Or just 1 of the 2?
 
Just the Nvidia and the HDMI audio stub.

I'll fire that VM back up in a sec.

5anH7.png


xVUJO.png
 
Last edited:
Since the card is an addon card and you have the Intel 2000, you should treat it that way. Use the onboard vga for esxi console and vm consoles (aka the card that is used as the vmware virtual device on the machines). Most mainboards let you configure to use onboard video as the main display adapter or an addon card.

Then on the vm in question, leave the vmware virtual video card and pass the nvidia card to it with VT-d, and configure xbmc in linux to use the nvidia card for its output and not the main display, aka the vmware one.

EDIT: you say that the nvidia card gives output at boot. I think this shouldn't be so. The Intel card should be the preferred display.
Otherwise you'll be passing trough a device esxi actually uses for itself, and that never works right.

The same can happen with raid controllers, VT-ding a raid controller that holds the disks on which ESXi is installed doesn't work all to great.
 
Last edited:
When the card was first installed it was tested but since then the GFX init sequence was changed before VT-d was enabled. So the ESXi yellow management screen is available on the Intel HDMI and the Nvidia HDMI has no output (as wished for)

dmesg | pastebin
http://pastebin.com/46TFfcXj Ignore the last segfault

lsmod | pastebin
http://pastebin.com/H1CCCRQ9

I'll attempt with the XBMC Live ISO again now.
 
Even if the vga gives output at boot, the resource isn't actually passed through until esxi loads up. Granted, you're dealing with experimental things here, but from what I've seen in a couple cases has been the vga to be passed through just stops outputting in the middle of the esxi load up, and then correctly switches over when the vm loads that uses it. Is this "normal" operation? I'm not sure, and I don't think there's a "normal" defined yet for this type of thing - but it is how I've seen it function so far.
 
One other thought, have you tried the passthrough with windows? The majority of responses for vga passthrough seem to be under windows, and it would be nice for you to be able to rule out an os-related conflict.

edit: http://communities.vmware.com/thread/297072?start=0&tstart=0 if you haven't seen this thread already, there's some interesting reading on the topic. The earlier pages are pre-esxi 5, where this seemed to be much less likely to run.

second edit: glancing back at that link, I remembered that there was a whole issue with having to set pci.hole start and end, as well as initial memory settings for the vm. I don't know if you've considered that at all, but you can find some good suggestions in that thread.
 
In my current config there is no output from the card regardless of the state of the machine even during boot AFAIK. Maybe I will have to switch to PCIe + Intel in the BIOS.

Should I expect to see the Nvidia card mirror the internal card under normal circumstances during the boot sequence?

I will check a Win7 VM tomorrow, it's worth a go. Non of the linux installs seem to be happy regardless.
 
On the boards I have that do Onboard + Addon, the bios is only shown on the card I select as primary.

But correct me if I'm wrong, but is your problem with those live CD's? Or did you do an actual install and configure it yourself too?

Because I could imagine that those live CD's might try to do some default settings that are unaware of your VM + addoncard situation. Switching to main display to the Nvidia card, screwing up your console etc, etc.


(EDIT: yes in my previous reply, I didn't see the screenshots of your setup, which seems correct. The SS's didn't load for me :p)
 
My VMs are not running live isos and I've installed XBMCbuntu which still doesn't show anything.

Although this is the first time I've attempted to install XBMCbuntu, I've been using nothing but Gentoo Linux and Nvidia cards since 2006 and I'm used to dealing with its setup via package-managers, kernel modules and eselect. I've never used Passthrough before as I've never had the hardware to up until late.
 
Does the nvidia driver load at boot or do you load it scripted when the entire boot process is done?
 
Its a kernel module + binaries IIRC (290.10) so yes it's called at boot.

The dmesg output shows it being called, modprobe -r nvidia && modprobe nvidia gives:

[ 5151.421934] vgaarb: device changed decodes: PCI:0000:13:00.0,olddecodes=none,decodes=none:eek:wns=none
[ 5151.422145] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 290.10 Wed Nov 16 17:39:29 PST 2011

then startx:

X.Org X Server 1.10.4
Release Date: 2011-08-19
X Protocol Version 11, Revision 0
Build Operating System: Linux 3.0.0-sabayon x86_64 Gentoo
Current Operating System: Linux sabayon 3.3.0-sabayon #1 SMP Wed Mar 21 12:19:47 UTC 2012 x86_64
Kernel command line: BOOT_IMAGE=/kernel-genkernel-x86_64-3.3.0-sabayon ro init=/linuxrc splash=verbose,theme:sabayon vga=332 console=tty1 quiet dokeymap keymap=uk domdadm resume=swap:/dev/mapper/vg_sabayon-lv_swap real_resume=/dev/mapper/vg_sabayon-lv_swap dolvm root=/dev/mapper/vg_sabayon-lv_root docrypt
Build Date: 31 January 2012 02:36:48PM

Current version of pixman: 0.24.4
Before reporting problems, check http://wiki.x.org
to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
(++) from command line, (!!) notice, (II) informational,
(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.0.log", Time: Tue Apr 3 01:18:42 2012
(==) Using config directory: "/etc/X11/xorg.conf.d"
(==) Using system config directory "/usr/share/X11/xorg.conf.d"
(EE) Failed to load module "dri" (module does not exist, 0)
(EE) Failed to load module "dri2" (module does not exist, 0)
(EE) Failed to load module "vmwgfx" (module does not exist, 0)
(EE) vmware: Please ignore the above warnings about not being able to to load module/driver vmwgfx
(EE) open /dev/fb0: No such file or directory
(EE) Failed to initialize GLX extension (Compatible NVIDIA X driver not found)

So it's either a driver issue or config, I'll have a look on the Nvidia forums for VT-d and drivers.

I will try Win7 soon.
 
Last edited:
Nvidia Kernel Modules and Nvidia X driver are 2 different things.

In your case, it seems the kernel module is loading fine, but the X driver isn't (or is missing).

I would change your system so that it can load the vmware graphics driver for X and then go ahead and try to get the nvidia driver working too.

If I'm not mistaken, XBMC lets you choose which display you wish to output it to, so once you verify you can actually load X and XBMC, you can work from there getting it to use the nvidia card as an output.

Sabayon, isn't that Gentoo derived?
 
Sounds sensible, I'll revive a fresh VM with vm-tools and start there.
 
Win7 VM tested. Nvidia web driver wizard recognised enough to recommend v290 although once downloaded it didn't install saying there's no Nvidia card available. Manually installing doesn't offer anything either.

m4YCE.png



JLfW1.png
 
Last edited:
ah bummer. from what i've read people have had success with ATI gpus.
 
Some GPUs work and some don't. FYI, unless VMWare's GPU passthru support has been much improved in a 5.0 update or in 5.1 (I doubt it), Xen is typically better for GPU passthru, but I find Xen Cloud Platform a lot more confusing than VMWare ESXi.
 
Alrighty o-o here goes my answer.. though I dunno if this will fix it on nvidia.. Originally when I did mine I had issues with mine actually displaying and such.. What I had to do was use it as the boot device.. So when the VMware console booted it showed through the video card.. then when windows started it would freeze for about 30 seconds then show the windows logo.. To get it to mount in windows though I had to do some driver manipulation because it never showed properly until i mounted the wddn driver.. I dunno about linux builds but I can see about getting an nvidia card working in my setup.
 
My post from another thread..

I'm currently using Windows 7.. I set the video to the GPU instead of onboard so you lose kvm. After that I set it up on pass through along with the USB. I'm running ESXI stock 5.0. After the restart I built a Win 7 Machine with the devices already passed to it and

One thing I did was turned on 3d on the Virtual machine settings.

To get more than 2 gigs of ram add this to the end of the VMX file

pciHole.start = "1200"
pciHole.end = "2200"

You MUST set the reserve as high as the allocated memory..

When it came up it already saw the USB controller and was using it properly. The GPU wasn't working yet. I installed just the driver and it saw it.


Another thing I did was go to "C:\Program Files\Common Files\VMware\Drivers\wddm_video" and updated the vmware svga driver to this one.. It changes the description in device manager of the VMWare video driver to VMware SVGA 3D (Microsoft Corporation - WDDM). After this I restarted. I then went into the display panel and said "Detect Displays" and it showed my 6870. After that I set the 6870 to the primary and such(you lose vmware console doing this). After this GPU worked perfect inside the VM.. Played Black Mesa source actually on it. I will say there was 1 or 2 "hickup" moments where the fps dropped to about 45 FPS but usually runs 300..
 
jerky,

that's awesome, i might pick up a 6870 if i can find a quiet one

dragonrage,

i'd run xen from source on a linux distro but it involves recompiling a kernel, writing your own conf files, etc, it's just a bit overkill for the amount of time I have hence why ESX is nice but in the end it might be xen that proves to be the most successful of the two

http://wiki.xen.org/wiki/Xen_VGA_Pa...ted_graphics_adapters_for_Xen_VGA_Passthrough - this bodes well

think about a win 8 install plus gpu passthrough and remotefx...drool
 
Last edited by a moderator:
gigatexal said:
dragonrage,

i'd run xen from source on a linux distro but it involves recompiling a kernel, writing your own conf files, etc, it's just a bit overkill for the amount of time I have hence why ESX is nice but in the end it might be xen that proves to be the most successful of the two
Click to expand...

Xen Cloud Platform should make things easier than that. Just not as easy as VMWare VSphere.
 
i'm looking into it. thanks for the tip.

i'm trying to get gpu passthrough and ibm m1015s passed through to a fbsd host
 
You must log in or register to reply here.
Back
Top