The "Use my ESXi host for multiple HTPC's using passtrough" project

[Zarathustra[H]]

Hey all,

So here is my scenario: I have a few cheap hacked Chromeboxes as HTPC's in the house. They are great. Unfortunately Intel's Linux EGL + VAAPI video decoode drivers are causing me all kinds of headaches, and I really wan to move away from integrated Intel graphics, but I can't with the Chromeboxes, which won't accept GPU:s (no expansion slots in that tiny form factor)

Since I already have a ESXi server running way under capacity (CPU load average: 0.15, 0.13, 0.12) the somewhat complicated solution is as follows:

Replace all my HTPC's with virtual machine guests, each with a passed through GPU, an HDMI over Ethernet extender and a USB over Ethernet extender.

I figure this will both save power (the host is on all the time anyway, so I will just be using the extra power when active) and eliminate having to deal with multiple physical boxes around the house, and manage everything from one central location.

This could be a fun project, and should be cheaper per HTPC, than buying new cases, power supplies, motherboards, etc. etc, even the low end types needed for a HTPC.

At first I wanted to use Geforce GT 720's as they sip power, are single slot width, and are PCIe 8x, and fit nicely into my Supermicro X8DTE server boards 8x slots without modification.

Unfortunately, Nvidia - in their wisdom - blocks GPU passthrough on consumer boards. While AMD passthrough generally works, the VAAPI support in Linux has been flaky, just like with Linux, The whole point of this project is to get a set of nice Nvidia VDPAU supporting GPU's

After some research, the Quadro/Tesla/GRID driver release notes have a list of compatible GPU's for pass through. The cheapest ones on eBay right now are the Quadro 2000's, essentially a downclocked GTS 450. I lost a few auctions for boards that sold used for $20-$30 (I suck at that last minute bid thing everyone does) but eventually wound up with one for just under $40. I figure I'll start with one as an experiment and add one or two more later as I bid on decent ones on eBay and win them cheaply.

They are 16x PCIe though, so I'll need to take a dremel to them to make them 8x compatible. Also they are only VDPAU feature set C, so no 4k or HEVC support, but that is OK for now. By the time I go 4k or HEVC in a big way, there might be cheap GPU's used that can do that, and be passed through, but those definitely don't exist now at a reasonable price.

My planned parts list for each HTPC guest is thus as follows:

• Nvidia Quadro 2000 GPU
• Monoprice HDBaseT Extender Kit
• Monoprice USB over Ethernet Extender
• Basic 2 port USB hub for the TV side so I can connect both my Logitech K400 Keyboard/Touchpad and my Windows Media Center Remote.
• Displayport to HDMI with Audio adapter
• Lots of cables

Would appreciate your comments and this project with any thoughts.

I understand USB passthrough of Keybaords andMice can be tricky in ESXi, but according to this Vmware KB I should be able to do so after manually editing the .vmx to include the following lines:

usb.generic.allowHID = "TRUE"
usb.generic.allowLastHID = "TRUE"

Hopefully this will do the trick.

What would be even cooler would be if I could pass through the USB hub. That way I can treat the USB hub as if it were a local machine, unplugging and plugging back in the devices of my choice, without repeatedly going back in and forwarding the devices.

By default HID devices and hubs are filtered out, but since there is a config line (like the above) to unfilter HID devices, is there also a config line to unfilter hubs?

I know, I could just get a USB controller and pass that through, but with all these GPU's going in, PCIe slots are not plentiful, so that isn't an option.

Another potential concern (Which I have discussed in this thread) is that x8 slots are generally 25W, (16x slots are 75W) and the Quadro's can draw up to 62.5W at full load. My server board has two 8 pin EPS connectors, and is using low power CPU's though, so I am thinking it will probably work without putting too much of a load on the 24pin connector. Besides, the video cards will only ever see idle loads or video decoee loads, never full 3d rendering or compute loads, so they might just stay in that 25W window on their own...

Hoping for some good feedback on this one

 

[ToddW2]

If I recall you won't be able to pass through individual USB Ports but only the 'group' of them on the controller. I purchased PCIE USB Controller for this purpose, but I also only plan to run 1 or 2 video cards (AMD). I have a handful of parts to do the same project, and really think there's a lot of use cases for this kind of project. WHen doing my research I also found HDMI over Ethernet can run $100 to $400+ depending if you want 4K and/or how 'quality' you want your feeds to be... so keep that in mind, I got some cheap ones and some under $200, didn't see sprining for the 300+$ doing my testing, and no 4K TV anyway.

The other thing I haven't looked into much is the features you need in ESXI... can you do it all with the free version, VMUG Advantage or do you need a licensed version?

Following along to see how this works

 

[Zarathustra[H]]

If I recall you won't be able to pass through individual USB Ports but only the 'group' of them on the controller. I purchased PCIE USB Controller for this purpose,

Yeah, this is my understanding.

You can pass through individual USB devices, or entire USB controllers, but nothing else.

It should be possible to forward a hub though, and have anything you plug into that hub go to the guest, but I can't say for sure. Looks like VMWare in their wisdom blocks this...

 

[Zarathustra[H]]

So I've had a momentary setback.

Got my Quadro. Tested it right out of the box. Worked great in my test system (an old AMD mobo). Broke out the dremel, and cut it down to size using a diamond wheel, and ground the cuts nice and smooth. Cleaned it off and popped it back in the test box, where it runs perfectly, now at PCIe 8x.

SUCCESS! (Or so I thought)

With that test completed, I shut down the ESXi server, and popped it in there, and nothing. In ESXi it doesn't show up in the list of devices available for forwarding. Using SSH togo to the ESX console, lspci shows no sign of the Quadro. Thinking I might be dealing with a ESXi issue, I booted the server from a Ubuntu USB stick. lspci shows nothing. Like it isn't even there.

Maybe I damaged it while taking it out of my test rig and put it in my server? To check I put it back in the test rig, and it still works perfectly...

Now I'm confused. Spent some time in the BIOS but didn't find any setting that could obviously be preventing this from working properly. Also tried plugging in a monitor to it, incase it needs a monitor attached during POST in order to work, but nothing.

Maybe there is a fused power supply to the PCIe slot and it is drawing too much power? Nope. Looking at it, the fan is spinning when in the server, and the heatsink is getting slightly warm to the touch...

It just does not want to work in my Supermicro X8DTE server motherboard...

I have four theories.

• I didn't shave enough material off the back of the card, due to being afraid of damaging the last 8x pin. This works fine in a 16x slot as there is nothing there, but being put in a 8x slot, it might be forced too far forward.
  
  Potential solution: (Carefully) sand it closer to the last lead.
  
• I need to short the PRSNT sense pins (pin 48 & pin 1) for it to work properly. (but why then would it work in the test system?)
  
  Potential solution: Somehow short the required pins. (god I hate soldering tiny things)
  
• Something non-obvious in the bios I need to change?
  
• Physical incompatibility with the X8DTE? (hope not).

I'd appreciate any thoughts regarding this issue!

 

[HammerSandwich]

How about an alternative approach?

Forget the expensive HDMI over Ethernet kits. You're already extending USB, so use USB display adapters. Just pass USB controller cards to the VMs, and you should be in business. Less work & cheaper!

This method would suck for gaming, but it seems viable for HTPCs.

 

[Zarathustra[H]]

How about an alternative approach?

Forget the expensive HDMI over Ethernet kits. You're already extending USB, so use USB display adapters. Just pass USB controller cards to the VMs, and you should be in business. Less work & cheaper!

This method would suck for gaming, but it seems viable for HTPCs.

That would be a cool solution, but there are a few problems with it:

1.) USB extender kits that support more than USB 1.1 are few and far between, and very expensive.

2.) Virtualization (or at least ESXi) USB pass through performance has never been particularly good. Enough for keyboards, mice, MCE remotes, etc, certainly, but nothing requiring real performance.

3.) The goal of this project is to be able to take advantage of Nvidia's excellent video decode engine, which has good Linux compatibility. Are there even any USB based 400 series or newer GPU's?

 

[Zarathustra[H]]

So I've had a momentary setback.

Got my Quadro. Tested it right out of the box. Worked great in my test system (an old AMD mobo). Broke out the dremel, and cut it down to size using a diamond wheel, and ground the cuts nice and smooth. Cleaned it off and popped it back in the test box, where it runs perfectly, now at PCIe 8x.

SUCCESS! (Or so I thought)

With that test completed, I shut down the ESXi server, and popped it in there, and nothing. In ESXi it doesn't show up in the list of devices available for forwarding. Using SSH togo to the ESX console, lspci shows no sign of the Quadro. Thinking I might be dealing with a ESXi issue, I booted the server from a Ubuntu USB stick. lspci shows nothing. Like it isn't even there.

Maybe I damaged it while taking it out of my test rig and put it in my server? To check I put it back in the test rig, and it still works perfectly...

Now I'm confused. Spent some time in the BIOS but didn't find any setting that could obviously be preventing this from working properly. Also tried plugging in a monitor to it, incase it needs a monitor attached during POST in order to work, but nothing.

Maybe there is a fused power supply to the PCIe slot and it is drawing too much power? Nope. Looking at it, the fan is spinning when in the server, and the heatsink is getting slightly warm to the touch...

It just does not want to work in my Supermicro X8DTE server motherboard...

I have four theories.

• I didn't shave enough material off the back of the card, due to being afraid of damaging the last 8x pin. This works fine in a 16x slot as there is nothing there, but being put in a 8x slot, it might be forced too far forward.
  
  Potential solution: (Carefully) sand it closer to the last lead.
  
• I need to short the PRSNT sense pins (pin 48 & pin 1) for it to work properly. (but why then would it work in the test system?)
  
  Potential solution: Somehow short the required pins. (god I hate soldering tiny things)
  
• Something non-obvious in the bios I need to change?
  
• Physical incompatibility with the X8DTE? (hope not).

I'd appreciate any thoughts regarding this issue!

Now the card doesn't work at all anymore in either my test rig or my server.

I must have wrecked it somehow.

Don't understand why it worked perfectly in the test rig after cutting it, then put it in the server, nothing, and now it doesn't work in the test rig anymore.

Maybe I killed my first computer component with ESD in moving it between the two.

Amusing that the difficult part (cutting the board) worked fine.

The simple part (moving it from machine to machine) failed.

Going to have to hit eBay and try again...

 

[FnordMan]

Now the card doesn't work at all anymore in either my test rig or my server.

I must have wrecked it somehow.

Don't understand why it worked perfectly in the test rig after cutting it, then put it in the server, nothing, and now it doesn't work in the test rig anymore.

Maybe I killed my first computer component with ESD in moving it between the two.

Amusing that the difficult part (cutting the board) worked fine.

The simple part (moving it from machine to machine) failed.

Going to have to hit eBay and try again...

Personally i'd suggest modifying the port on the server. Very carefully cut the a little bit of the back of the slot off, that way you won't have to modify the card at all.

 

[HammerSandwich]

That would be a cool solution, but there are a few problems with it:

1.) USB extender kits that support more than USB 1.1 are few and far between, and very expensive.

2.) Virtualization (or at least ESXi) USB pass through performance has never been particularly good. Enough for keyboards, mice, MCE remotes, etc, certainly, but nothing requiring real performance.

3.) The goal of this project is to be able to take advantage of Nvidia's excellent video decode engine, which has good Linux compatibility. Are there even any USB based 400 series or newer GPU's?

1) I hadn't realized the Monoprice extender was just 1.1. They do have long, active USB cables, but I get why you'd rather avoid those.

2) Is this an issue if you VT-d an entire controller?

3) Gotcha. And I don't know any specifics about the GPUs in USB adapters.

Anyhow, interesting project. I'll be watching to see where you end up. Good luck!

 

[Zarathustra[H]]

Personally i'd suggest modifying the port on the server. Very carefully cut the a little bit of the back of the slot off, that way you won't have to modify the card at all.

Yeah,

I've done this before successfully. Last time.e I did it I carefully melted out the back of the slot with a soldering iron, and it worked perfectly. The circumstances were different this time though.

In this case the server is a dual CPU LGA1366 Xeon board which runs like a home production server. It is my router, my firewall, my file server, my media server, my TV system, etc. etc. etc.

If I wreck a cheap sub $50 GPU modifying it, its not a big deal and I have no server downtime.

If I wreck the motherboard, I have no internet, no TV no media, nothing until I can replace it, and it will likely cost a lot more than a cheap GPU, if I can even find it.

I determined modifying the GPU - while higher risk of destroying something - was lower risk overall.

The best solution would have been a powered riser. Little risk there at all. The problem here though is that I'd probably have to have it shipped from china, whould take forever, and if I use one, now I have to figure out how to custom ghetto-mount the GPU's since they wouldn't be in standard slots...

Having destroyed one GPU, I'm not sure what I am going to try next. Definitely going to be bidding on another, and when I win one I will see what approach I will take next...

 

[Zarathustra[H]]

1) I hadn't realized the Monoprice extender was just 1.1. They do have long, active USB cables, but I get why you'd rather avoid those.

Those would be cool, but in my case the Ethernet is already wired in the walls, so I was going for that. If I needed higher speed USB I'd try something else, but for my implementation I don't. 1.1 will be fine

2) Is this an issue if you VT-d an entire controller?

No, I don't think so. If you forward a controller (and it works properly after forwarding) it should be just as if it were native USB in the guest.

My problem is that even with my 6 PCIe slots, I don't have any spare slots for USB controllers.

Expansion cards in the server are as follows:
1 - LSI 9211-8i SAS HBA (IT Mode)
2 - LSI 9211-8i SAS HBA (IT Mode)
3 - Nvidia Quadro 2000 (planned)
4 - Nvidia Quadro 2000 (planned)
5 - Brocade BR-1020 (10gig fiber Ethernet)
6 - Intel PRO/1000 PT Quad Port Gigabit Ethernet

Now, the motherboard es appear to have a metric ton of on board USB controllers I could forward. I COULD get a USB header PCI bracket, connect two of them, and forward those. Might be a bit of a trial an error task to figure out which USB device in the list of devices to forward corresponds to which USB header on the motherboard though...

And while experimenting, there would be downtime...

3) Gotcha. And I don't know any specifics about the GPUs in USB adapters.

Yeah, would be cool if you could get a really lightweight Nvidia GPU (like the GT 720) in USB form. Despite its low 22W TDP it would likely need supplemental external power though, as that is more power than USB typically provides.

Anyhow, interesting project. I'll be watching to see where you end up. Good luck!

Thanks! There will likely be a delay now though, as I have to hunt down my next Quadro 2000

 

[HammerSandwich]

My problem is that even with my 6 PCIe slots, I don't have any spare slots for USB controllers.

You have the same number of slots for USB cards as for Quadros, so that detail doesn't convince me. The other details make sense, though.

 

[Zarathustra[H]]

You have the same number of slots for USB cards as for Quadros, so that detail doesn't convince me. The other details make sense, though.

Ahh. Right. I forgot you were suggesting USB GPU's.

I was thinking in terms of using direct io forwarding to pass then through along with the quadros

 

[FnordMan]

If I wreck a cheap sub $50 GPU modifying it, its not a big deal and I have no server downtime.

If I wreck the motherboard, I have no internet, no TV no media, nothing until I can replace it, and it will likely cost a lot more than a cheap GPU, if I can even find it.

I determined modifying the GPU - while higher risk of destroying something - was lower risk overall.

The best solution would have been a powered riser. Little risk there at all. The problem here though is that I'd probably have to have it shipped from china, whould take forever, and if I use one, now I have to figure out how to custom ghetto-mount the GPU's since they wouldn't be in standard slots...

Having destroyed one GPU, I'm not sure what I am going to try next. Definitely going to be bidding on another, and when I win one I will see what approach I will take next...

Gotcha.. well, depends on how ghetto you want to get. There's a few things I see on amazon that looks like they'll work that'll get here quick. x8 to x16 riser or a x16 extension cable. I'm sure there's more if you dig around amazon some. Anything marked prime should be here in the states and ship relatively quick.

 

[Zarathustra[H]]

Gotcha.. well, depends on how ghetto you want to get. There's a few things I see on amazon that looks like they'll work that'll get here quick. x8 to x16 riser or a x16 extension cable. I'm sure there's more if you dig around amazon some. Anything marked prime should be here in the states and ship relatively quick.

The first one you link, is a 90 degree left model, which would be really difficult to fit in. The ribbon model might work, but it would have to be squeezed in somehow to make it fit...

I actually did some poking around and found an x8 to x16 adapter on eBay that extends it straight up.

It does add about 8mm of height though. I'm hoping I might be able to mount the quadro in it by simply using a motherboard standoff (if I can find one with the right threads) with a washer on top of the PCIe slot.

I'll post back with how it goes! Should have a new Quadro and the adapters to play with late next week.

 

[Zarathustra[H]]

Expansion cards in the server are as follows:
1 - LSI 9211-8i SAS HBA (IT Mode)
2 - LSI 9211-8i SAS HBA (IT Mode)
3 - Nvidia Quadro 2000 (planned)
4 - Nvidia Quadro 2000 (planned)
5 - Brocade BR-1020 (10gig fiber Ethernet)
6 - Intel PRO/1000 PT Quad Port Gigabit Ethernet

Come to think of it, long term I probably could free up one slot, by getting rid of one of the LSI SAS HBA's and replacing it with a good SAS expander. My total throughput to my 16 drives never exceeds the 4GB/s of a single PCIe 2.0 x8 slot anyway.

Only reason I went with the two controllers to begin with was because I didn't need the extra slots, and another 9211 used on eBay was cheaper than a good SAS expander.

Will have to think about this.

 

[RyC]

I tried to virtualize my HTPC/workstation like this back in 2013 with little success. Ironically, I got it to work more or less with a Mac OS X VM, but there were major stability issues and I gave up. I went down the USB video card route as well and the DisplayLink drivers for some reason still required a "real" GPU to work, but maybe that's changed now.

I hope you get it working, I want to try this kind of project again if works out!

 

[olskool]

I have virtualize 2 HTPC's using 2 SAPPHIRE Radeon HD 6450 1GB. I think the max power consumption on these is 14w

I've used them in ESXi severs with TYAN S5512WGM2NR and SUPERMICRO MBD-X8SIL motherboards.

 

[MrGuvernment]

FYI for last minute bidding use a free service instead.

 

[Angry]

Following!

 

[0ptional]

I ended up using inexpensive AMD cards for my HTPC ESXi passthrough project, and bought a cheap usb 2.0 pci card and passed that and the AMD card to my HTPC VM, all worked like a charm once the right combo of hardware was found.

I had to roll my own ESXi 6 installation media because I have a non-supported SATA/AHCI card and wanted the fully updated web interface included.

http://www.v-front.de/p/esxi-customizer-ps.html

 

[Zarathustra[H]]

I ended up using inexpensive AMD cards for my HTPC ESXi passthrough project, and bought a cheap usb 2.0 pci card and passed that and the AMD card to my HTPC VM, all worked like a charm once the right combo of hardware was found.

I had to roll my own ESXi 6 installation media because I have a non-supported SATA/AHCI card and wanted the fully updated web interface included.

http://www.v-front.de/p/esxi-customizer-ps.html

What guest OS did you use for your HTPC(s)?

I would have loved to be able to use AMD's easily forwardable consumer GPU's, but I felt forced to go Nvidia, which meant I couldn't use consumer cards, and had to get Quadro 2000's.

I found AMD's video decode acceleration to be disappointing under Linux. Better than Intel's, but still not ideal.

One of my big motivators for this project is that I can get Nvidia VDPAU support, which is what they call the PureVideo equivalent in Linux, since my platform is Kodi/XBMC streaming media from my file server, and running the MythTV plugin for TV on top of Linux

 

[0ptional]

Windows 10 with plex, I got off easy with hardware by choosing that as my front end.

 

[Zarathustra[H]]

The first one you link, is a 90 degree left model, which would be really difficult to fit in. The ribbon model might work, but it would have to be squeezed in somehow to make it fit...

I actually did some poking around and found an x8 to x16 adapter on eBay that extends it straight up.

It does add about 8mm of height though. I'm hoping I might be able to mount the quadro in it by simply using a motherboard standoff (if I can find one with the right threads) with a washer on top of the PCIe slot.

I'll post back with how it goes! Should have a new Quadro and the adapters to play with late next week.

Doh,

I read those specs too fast.

It adds 0.8"... A little taller than I was expecting. (I know I wrote 0.8mm above, but I was thinking in my head ~8mm.

Turns out this isn't a problem, just add more motherboard standoffs

So, three motherboard standoffs, and then a washer below the PICe card, a washer above th ePCIe card and the screw.

It's not as tight as it would have been if installed directly in the slot, but it's a hell of a lot better than being ghetto-mounted with a flexible riser!

I do wind up losing the top Displayport though, but it isn't a problem as I only need one.

So, it's up and running well, with a quick furmark test in the test rig just to give it a little stress and make sure the used quadro is good.. Time to move it over to the server

 

[HammerSandwich]

That looks MUCH easier than hacking the card apart.

 

[Zarathustra[H]]

That looks MUCH easier than hacking the card apart.

It was (except for when I dropped the non-magnetic washers down inside the case and had to go fishing for them )

The issue is, it isn't particularly easy to find these adapters.


For the longest time I couldn't find any at all, until I came across the eBayer linked above. All the ones on Amazon, eBay and anywhere else I found, where angled risers, which wouldn't have worked, or flexible ones which would have been awful to custom mount.

 

[Zarathustra[H]]

Alright, it took some work, but I now have it working (well, at least have kodi displaying out via the Quadro 2000 in a guest in ESXi.

After passing through the GPU to the guest Nvidias Quadro driver release notes state that there are two manual tweaks to the .vmx file you need to make.

1.) Disabling msi by adding the following line:

pciPassthru0.msiEnabled = "FALSE"

(if you have more than one passed through device it may be pciPassthru1 or 2, you'll have to find the number in the guest configuration screen and match it)

2.)changing the start of the PCI hole by adding:

pciHole.start = “2048”

After these two tweaks it was SUPPOSED to work, but it still didn't. I've read of people being successful with only these two tweaks, but this was in Windows. In Linux, whenever I'd start x, x would immediately crash and nothing would happen.

The X logs didn't tell me anything useful. I suspected there was some sort of conflict going on between the ESXi consoles emulated SVGA adapter, and the Quadro, but the guest configuration screen does not allow you to remove the SVGA adapter for some reason...

Glancing at the vmx file I found this line:

svga.present = "TRUE"

On a whim, I changed the TRUE to FALSE, and it worked! X started, and displayed its output over the Quadro.

I haven't been able to run any tests yet, as I don't have USB inputs working yet. In their wisdom, to protect us from ourselves they filter out all HID devices from USB passthrough, so you don't lock yourself out of your box, by losing your keyboard . This VMWare KB suggests that if you want to pass through a keyboard and mouse all you have to do is add the following lines to your .vmx file:

usb.generic.allowHID = "TRUE"
usb.generic.allowLastHID = "TRUE"

For me this still doesn't work. I add these lines, and HID devices are still not visible in the USB passthrough screen.

In order to get around this, I am going to do what many others have done, and just pass through a USB controller to the guest.

Problem is, I don't have any free PCIe slots for a USB controller. I DO however, have many USB controllers on board, and I am going to try to pass one of them through. The problem is, I'm going to have to figure out which header on the motherboard corresponds to which USB controller in the passthrough screen. This might take some trial and error, unless anyone has any ideas?

The USB controller count (8) matches what the manual says I have. What doesn't make any sense is that only two of them are listed as EHCI USB2, and the others are listed as UHCI... According to my motherboard manual, they are all supposed to be USB2. I checked the bios, and HiSpeed USB2 mode is enabled for all of them...

Oh well, I'm starting to see double from looking at this for too long. Going to get some rest and continue this weekend...

Any thoughts appreciated.

 

[extide]

I know this is going against the grain a bit -- but you could easily run XBMC/Kodi on one of these: https://www.kickstarter.com/projects/pine64/pine-a64-first-15-64-bit-single-board-super-comput or these: http://up-shop.org/4-up-boards

 

[Zarathustra[H]]

I know this is going against the grain a bit -- but you could easily run XBMC/Kodi on one of these: https://www.kickstarter.com/projects/pine64/pine-a64-first-15-64-bit-single-board-super-comput or these: http://up-shop.org/4-up-boards

I know this is possible, but the goal here was to be able to use an nvidia GPU with the standard linux binary blob driver, supporting Nvidias excellent VDPAU implementation.

That, and the Kodi plugins I want to run don't exist in ARM form.

 

[Zarathustra[H]]

Problem is, I don't have any free PCIe slots for a USB controller. I DO however, have many USB controllers on board, and I am going to try to pass one of them through. The problem is, I'm going to have to figure out which header on the motherboard corresponds to which USB controller in the passthrough screen. This might take some trial and error, unless anyone has any ideas?

The USB controller count (8) matches what the manual says I have. What doesn't make any sense is that only two of them are listed as EHCI USB2, and the others are listed as UHCI... According to my motherboard manual, they are all supposed to be USB2. I checked the bios, and HiSpeed USB2 mode is enabled for all of them...

Oh well, I'm starting to see double from looking at this for too long. Going to get some rest and continue this weekend...

Any thoughts appreciated.

Hmm. So I took another look at the motherboard manual, and it states that there are 8 USB2 ports, not controllers.

[root@localhost:~] lspci |grep -i usb
0000:00:1a.0 Serial bus controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
0000:00:1a.1 Serial bus controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
0000:00:1a.2 Serial bus controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
0000:00:1a.7 Serial bus controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
0000:00:1d.0 Serial bus controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
0000:00:1d.1 Serial bus controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
0000:00:1d.2 Serial bus controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
0000:00:1d.7 Serial bus controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
[root@localhost:~] lsusb
Bus 008 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Looking at the above, I'm going to assume that there are two USB2 controllers on board, and that integrated hubs break them out to the 8 ports in total. My best guess is that one of the controllers is used for the rear panel, and one for the internal headers, but I could be wrong here. Trial and error will have to resolve this.

Problem with this is, that if I forward one each of the USB2 controllers to my two planned HTPC's, I wont have any USB controllers left for the host. This isn't a HUGE issue for me, as I don't boot off of USB as many do with their hosts, and if I need a console keyboard, I can always connect it via PS2, but I still would ahve preferred to retain some host USB connectivity...

I have absolutely NO IDEA what the 6 USB 1.1 (UHCI) controllers are there for. At first I thought they were internal on board used to connect such things as the serial port and PS2 mouse/keyboard connectors, but if that were the case, these USB devices would ahve shown up in lsusb. Maybe they are just included with the chipset, and not connected to anything?

Or is it more complicated than that? The Wikipedia USB page suggests that EHCI does some sort of intelligent handoff of connectivity to UHCI for 1.1 devices, but it isnt exactly clear how this happens. I wonder if I would have to forward a group of EHCI and UHCI at the same time to the guest. Hopefully if this is the case, ESXi will figure it out for me when I check the box, as it does with other things when it automatically detects if a PCI device is dependent...

 

[Zarathustra[H]]

Another note, unrelated to USB...

When I first connected the Quadro to the server, the on board GPU (Matrox MGA G200eW) still initialized first and provided host console output (that yellow and black screen).

At some point - however - it switched to the Quadro, without me doing anything. I have looked in the BIOS, and unlike with desktop boards, there does not appear to be a BIOS option to tell it which GPU (on board or PCIe) to initialize first (probably because they didn't consider that people might stick a GPU in an x8 slot on a server motherboard)...

The downside of this is that during boot of the server, I can see POST and the beginning of boot output on the Quadro, up until the point when ESXi loads the IOMMU module, then the screen freezes, until X starts up on my HTPC guest and displays the Kodi screen on it.

The host continues its normal boot, and works properly when remotely connected to, this just isn't displayed on screen.

The downside to this is, that if I ever either accidentally mess up network settings, or for some other reason need to access the server locally in order to shut it down or reboot it, I can't.

I wish there were some way to force ESXi to use the internal GPU for its output...

 

[Zarathustra[H]]

Another note, unrelated to USB...

When I first connected the Quadro to the server, the on board GPU (Matrox MGA G200eW) still initialized first and provided host console output (that yellow and black screen).

At some point - however - it switched to the Quadro, without me doing anything. I have looked in the BIOS, and unlike with desktop boards, there does not appear to be a BIOS option to tell it which GPU (on board or PCIe) to initialize first (probably because they didn't consider that people might stick a GPU in an x8 slot on a server motherboard)...

The downside of this is that during boot of the server, I can see POST and the beginning of boot output on the Quadro, up until the point when ESXi loads the IOMMU module, then the screen freezes, until X starts up on my HTPC guest and displays the Kodi screen on it.

The host continues its normal boot, and works properly when remotely connected to, this just isn't displayed on screen.

The downside to this is, that if I ever either accidentally mess up network settings, or for some other reason need to access the server locally in order to shut it down or reboot it, I can't.

I wish there were some way to force ESXi to use the internal GPU for its output...

I may have thought of a way to solve this.

While the BIOS doesn't have the standard (for desktops/workstations) option of which GPU (on board or PCIe) to initialize first, it DOES have a server feature (that I wish my desktops had) that might work.

The BIOS allows me to disable the loading of Option ROM's on a per PCIe slot basis. If I disable the reading of the Option ROM at boot time for the slot the Quadro is in, it SHOULD prevent it from initializing until the guest it is forwarded to loads drivers for it.

This might just allow the on board GPU to initialize first, and maintain my console output...

I will have to test this theory when I get home!

 

[HammerSandwich]

<USB>

I'd boot the server from the UBCD to try the SYSTEM tools, such as HWiNFO.

Also, I have a UPS on USB. (This is a simple device connection, not controller passthrough.) In the VM settings, I see "host: localhost path:1/0/3 version:2". Pretty safe bet that 1/0/3 includes a controller ID. Like 1?

Anyhow, just stray thoughts. YMMV.

 

[Zarathustra[H]]

I'd boot the server from the UBCD to try the SYSTEM tools, such as HWiNFO.

Also, I have a UPS on USB. (This is a simple device connection, not controller passthrough.) In the VM settings, I see "host: localhost path:1/0/3 version:2". Pretty safe bet that 1/0/3 includes a controller ID. Like 1?

Anyhow, just stray thoughts. YMMV.

Good thoughts.

If I had to guess, those numbers probably correlate to the BUS and Device numbers when you do an lsusb from the ESXi console (or any Linux box)

I presume the BUS id is unique to a USB controller, but I am not certain. I have - however - not found a way to correlate the BUS number to the PCIe ID of the USB controller.


Actually, I take that back. Looking at the lsusb output and the BUS numbers, the two USB2 hubs are on BUS 1 and 2. These are also numbered 1 and 2 in the "Mark Devices for Passthrough" screenshot.

To your point, I should be able to plug a device in to a USB port, do an lsusb, check which BUS number it is connected to, and then pass through the corresponding controller in the list!

With a little luck, this will be how it works!


If you don't mind me asking, how to do control your UPS? I was hoping for a local solution, but in the end, the only way I got it to work, was to forward my UPS to a guest, and have that guest automatically ssh in to the ESXi console and issue a shutdown command when battery runs low.

I started with USB, but found that this wasn't as reliable as I had hoped, so in the end I did UPS control from my APC SmartUPS 1500 serially.

 

[HammerSandwich]

To your point, I should be able to plug a device in to a USB port, do an lsusb, check which BUS number it is connected to, and then pass through the corresponding controller in the list!

You motivated me to SSH & lsusb.

For both buses, device 1 is the root hub, while dev 2 is an "Integrated Rate Matching Hub". I assume this is all integral to the chipset USB.

Moving a flash drive to different ports, it looks like bus 1 is the mobo rear, & 2 goes to front of case. Sure enough, the UPS is device 3 on bus 1.

This all makes sense, so your approach sounds right.

If you don't mind me asking, how to do control your UPS? I was hoping for a local solution, but in the end, the only way I got it to work, was to forward my UPS to a guest, and have that guest automatically ssh in to the ESXi console and issue a shutdown command when battery runs low.

That's about it. Mine's a Cyberpower CP1350PFCLCD, so I use CP's prebuilt agent VM. The default VM settings are silly big, but that's an easy fix. I believe it runs fine with 256MB & a 300MHz limit.

 

[Zarathustra[H]]

Yeah, so the USB thing didn't work out wuite as planned at all. Turns out there is no corellation between the BUS number and the number in the PCI passthrough screen.

Furthermore, as I read on Wikipedia, even when plugged in to the same port, it will use a different controller for EHCI (USB2) and UHCI (USB1.1).

So I was using an internal motherboard header hooked up to PCI slot USB ports. For USB 2 devices those two ports wound up being EHCI controller #1. For USB 1.1 devices (like my keyboard and WMC remote) they wound up being UHCI controller #4.

So, I forwarded them both to the guest, and it appears to be working. Took a lot of trial and error to get there though. (put host in maintenance mode, enable port for passthrough, reboot, ssh in, lsusb to check if the device is still listed. If it was, change passthrough to a different controller, reboot again, etc. etc.

Took forever too, as my server doesn't exactly get through POST quickly

 

[HammerSandwich]

What a PITA.

If I ever need to do this, I'll try creating as many testing VMs as I have USB controllers. By passing 1x USB to each VM, I should be able to map all of the ports with 2 reboots. I think.

 

[Zarathustra[H]]

So, I stayed up way too late last night (I have to do these things when the TV is not in use, which is surprisingly challenging sometimes) and finally got it up and running.

Picture quality over the Monoprice HDBaseT adapter is great. Video acceleration on the Wuadro works perfectly. I have seen no studdering or dropped frames, and CPU utilization in the guest is very low. Deinterlace and scaling quality seem much better than the Intel IGP on the Haswell Celeron 2955u I was using previously.

Thus far it is a success.

I've run into a few issues though, as follows:

• The Monoprice HDBaseT adapter is supposed to also be able to carry a 100Mbit ethernet signal between the two sides. Each side has two 100mbit ethernet ports. I ahve NOT been able to get this to work. I plug in cables on each side, and I get link lights, but I never get an actual signal between the sides. This is not a huge deal for me for this screen, but when I install the second GPU I was planning on using it. I will troubleshoot and make sure that it's not my switch or ethernet cables that are the problem.
  
  
• I tried using the infrared adapter in the Monoprice HDBaseT unit. It would not work at all with my Windows MCE remote. This is also not a huge problem, as I can just run a powered USB hub on the other side of the Monoprice USB over ethernet adapter. Would have been nice to save some power and not need to though.
  
  
• Everything works, but the MCE remote disconnected for some reason overnight, requiring a reboot of the guest to become usable again. I'm not 100% sure why, or if it was just a fluke. I am continuing to monitor to see if I need to do anything else to address this. Maybe I need to forward more than just one of the UHCI controllers for USB 1.1
  
  
• While the forwarded Quadro displays perfectly when the graphics driver is loaded up, and you are in the GUI, text consoles (like the BIOS screen or a non-graphical Linux or DOS command prompt) do not work at all. So it is not exactly like having a local machine with a video card installed, but for my purposes it is sufficient.
  
  
• I was never able to get the on board GPU to initialize before the Quadro, which means I still have no local access to ESXi to configure network or shut down/reboot the server. I do all of this via the network anyway, but I sincerely hope that I never wind up locked out of the box and unable to regain access due to not being able to reset network settings. I feel this is a shortcoming of my server motherboard more than anything else. There is no "GPU to initialize first, IGP or PCIe" option, and even with "Option ROM" disabled on all the PCIe slots, the BIOS is picking up that slot 5 contains a GPU, and giving it video output, before initializing the on board graphics. It's a bit of a bummer, but I can live with it (at least until something goes wrong, and I get locked out )

So, I am cautiously declaring success for right now. Hopefully I will still consider it so a week from now. If no major problems surface, I'll be picking up a second Quadro for forwarding to another guest for Virtual HTPC #2 soon. Second time around should be much easier I hope

 

[Zarathustra[H]]

Well,

This was a fun little project, but I am going to have to abandon it, unfortunately.

The virtual HTCP was up and running for almost a week, and then all of a sudden it started having freezing and crashing problems.

It was really odd to me, as problems like these are usually brought on by making changes, but I didn't change anything.

Went through a ton of trouble shooting, and I just cant make it work again.

It's a bummer. The Quadro 2000's officially support passthrough on ESXi, but I guess even with officially supported hardware, passthrough can be a bit of a crap shoot.

I suspect that my issues might be related to the interrupt hand-off crashing issue in ESXi, which is officially solved in 6.0 Update 1A, but Update 1A and Update 1B are both only available to licensed Vsphere customers, not by us mere mortal hypervisor users. The highest we can get is Update 1, which is a major bummer.

Anyway, I'm going back to the drawing board. Might build a couple of low end HTPC's for traditional HTPC duty instead.

 

[HammerSandwich]

Well,

This was a fun little project, but I am going to have to abandon it, unfortunately.

That stinks. OTOH, I enjoyed watching your progress & learned a bit from it...