• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

ESXi passthrough with incompatible device?

M

__McB__

n00b
Joined
Apr 1, 2011
Messages
10
I want to add another sata hba card to my all-in-one ESXi box. The card I have is not on the HCL, nor have I found whitebox drivers, but I don't know if this is a problem as I plan on passing through the pci card to my SE11 VM. The device is on the solaris hcl fwiw.

Can you passthrough an ESXi incompatible device to a VM?

Thanks!

Mick
 
Yeah, I found that out yesterday after a lot of testing.

I was able to get ESXi to see the cards and connected drives after modifying the oem.tgz. Unfortunatelty, no matter the OS, the guest was not able to recognize the controller card once passed through. Didn't work even if the card was on the guest OS HCL.

I ended up configuring RDM to pass the raw drive to the guest. Before I give up and go with RDM I do have one more question:

Does Passthrough of a sata card (with a compatible card) perform better than RDM? The drives will be used to create a zfs array passed back to ESXi as a datastore to hold other VMs, so performance is critical.

Thanks!

Mick
 
I don't know why you're trying to use an RDM or use non-virtualized storage, either way, but RDMs do not offer a performance advantage (or if anything is neglible). If you're doing something like MSCS, I could see setting up an RDM as its required, but I'm going to guess being supported isn't a major driver here. Why not just use vmfs now that ESXi can see the card/storage now?
 
I want the datastore to be controlled by my Solaris guest machine instead of ESXi controlling it. Giving Solaris physical versus virtual drives gives me better performance. The question is which physical access (RDM or passthrough) will give me better performance.

My goal is to set up a ZFS mirror then share it back to ESXi via NFS. I prefer ZFS over RAID. So my options are to get a supported card and passthrough the 2 drives to my solaris guest, or keep the unsupported card and set up RDM to give my solaris guest direct access to the raw drive. I will do which every gives better performance.
 
Should be about the same (RDM vs VMFS), passthrough of storage devices is experimental only at this point. That being said, do you have proof of ZFS performance differences on VMFS VMDKs vs RDMs? I haven't heard of any.
 
__McB__ said:
I want the datastore to be controlled by my Solaris guest machine instead of ESXi controlling it. Giving Solaris physical versus virtual drives gives me better performance. The question is which physical access (RDM or passthrough) will give me better performance.

My goal is to set up a ZFS mirror then share it back to ESXi via NFS. I prefer ZFS over RAID. So my options are to get a supported card and passthrough the 2 drives to my solaris guest, or keep the unsupported card and set up RDM to give my solaris guest direct access to the raw drive. I will do which every gives better performance.
Click to expand...

pci-pass-through is the only ESXi-supported and always the fastest option.
You do not need to look at the ESXi HCL. If you activate pass-through, ESXi do not
touch the controller at all. Only the guest needs to support it at driver level.
But sometimes, a controller is not trouble-free with pass-through-
independent from the driver question- best to use are the usual LSI 1068 or 2008 based ones.

-> Solaris and ESXi mostly likes the same server class controller.


Gea
 
_Gea said:
You do not need to look at the ESXi HCL. If you activate pass-through, ESXi do not
touch the controller at all. Only the guest needs to support it at driver level.
But sometimes, a controller is not trouble-free with pass-through-
independent from the driver question- best to use are the usual LSI 1068 or 2008 based ones.
Click to expand...

Yeah, that was my understanding and my hope. Unfortunately, the Sil3132 controller, which is on the Solaris HCL, did not like the pass-through at all. I also tried Windows OSs and got the same result.

My on-board LSI1068 works perfect so I'll just get another one and pass-through it.

Thanks!
 
_Gea said:
pci-pass-through is the only ESXi-supported and always the fastest option.
You do not need to look at the ESXi HCL. If you activate pass-through, ESXi do not
touch the controller at all. Only the guest needs to support it at driver level.
But sometimes, a controller is not trouble-free with pass-through-
independent from the driver question- best to use are the usual LSI 1068 or 2008 based ones.

-> Solaris and ESXi mostly likes the same server class controller.


Gea
Click to expand...

Not correct. Device must be compatible with ESX for VM DirectPath. At least, if you want it to be even remotely supported.
 
You can pass-through any pci-device.
But not all of them are working without problem.

From driver level, you do not need ESXi driver.
Pass-through means, that a controller is passed to a
guest.The guest needs to have the driver.

Gea
 
lopoetve said:
Like I said before, no, it's not supported, and almost certainly won't work (as most people are finding out). :)
Click to expand...

Curious. Cuz I've got a machine that until last week was running just fine with ESXi 4.1. Had a SuperMicro AOC-SASLP-MV8 PCIe SAS HBA - definitely and absolutely not on VMWare's supported list. There aren't even third-party drivers to support it on VMware.

Yet it passed it through to a Server2008 VM and it worked perfectly.

That is, in fact, the whole idea of PCI passthrough is that the PCI/PCIe device is passed through by the CPU/Chipset at the low layers and presented to the virtual machine as if it were a bare-metal host with the device plugged it. ESX/ESXi stay completely out of the way. There is no need for the hypervisor to "support" the card at all.
 
I'm hearing two conflicting messages here. Lopoetve keeps saying it it won't work and others that it should (and will.) I've seen a couple of people who had problems with interrupts storms, but that's most likely an issue with chips, bios or whatever...
 
danswartz said:
I'm hearing two conflicting messages here. Lopoetve keeps saying it it won't work and others that it should (and will.) I've seen a couple of people who had problems with interrupts storms, but that's most likely an issue with chips, bios or whatever...
Click to expand...

As Gea pointed out, you do not need ESXi support to do passthrough, but there is no guarantee that any particular device will work. The device and Mobo must both adhere perfectly to the PCI/PCIe specs - and the sad truth is that there is lots of hardware out there that plays a little loose with parts of it.
 
PigLover said:
Curious. Cuz I've got a machine that until last week was running just fine with ESXi 4.1. Had a SuperMicro AOC-SASLP-MV8 PCIe SAS HBA - definitely and absolutely not on VMWare's supported list. There aren't even third-party drivers to support it on VMware.

Yet it passed it through to a Server2008 VM and it worked perfectly.

That is, in fact, the whole idea of PCI passthrough is that the PCI/PCIe device is passed through by the CPU/Chipset at the low layers and presented to the virtual machine as if it were a bare-metal host with the device plugged it. ESX/ESXi stay completely out of the way. There is no need for the hypervisor to "support" the card at all.
Click to expand...

Experimental support only - not supported. It might work, it might not, it's still a phase one release. Technically, the tested cards are a set of network cards. Anything outside of that is "good luck!" only, and may have some very... "interesting"... results after being used for a bit. ;) I've seen some very, very funny happenings.
 
danswartz said:
I'm hearing two conflicting messages here. Lopoetve keeps saying it it won't work and others that it should (and will.) I've seen a couple of people who had problems with interrupts storms, but that's most likely an issue with chips, bios or whatever...
Click to expand...

I've got infinitely better information than _Gea.

Some things work, some things don't. The only "supported" cards for passthrough at this point are some network cards. :)
 
Not to be snarky, but the internet is full of crap and people making baseless assertions. Where does your infinitely better info come from? Are you a vmware employee? Sorry if that is a well-known fact - I am fairly new here...
 
@lopoetve: backing off, eh? Just a few posts ago you claimed "almost certainly won't work". Now its "some things work, some don't" (which is pretty much what the people you are arguing against have been stating).

The truth is even better than you claim. The FACT is that MOST things work and a very, very few don't.

Basically, its "not supported", which means if you call up VMWare for support they will tell you do go away. But that is a long, long way from "almost certainly won't work", isn't it? But then again, nobody expects them to "support" the hundreds of devices that are out there, do they?

Unless the OP is looking for a commercial production system, "support" from VMWare probably isn't very interesting to him anyway. And from the tone of the post that is not what he is looking for.

Regardless of its source (including the possibility that you are a VMWare employee), your "infinitely better" information appears to be FUD.
 
PigLover said:
@lopoetve: backing off, eh? Just a few posts ago you claimed "almost certainly won't work". Now its "some things work, some don't" (which is pretty much what the people you are arguing against have been stating).

The truth is even better than you claim. The FACT is that MOST things work and a very, very few don't.

Basically, its "not supported", which means if you call up VMWare for support they will tell you do go away. But that is a long, long way from "almost certainly won't work", isn't it? But then again, nobody expects them to "support" the hundreds of devices that are out there, do they?

Unless the OP is looking for a commercial production system, "support" from VMWare probably isn't very interesting to him anyway. And from the tone of the post that is not what he is looking for.

Regardless of its source (including the possibility that you are a VMWare employee), your "infinitely better" information appears to be FUD.
Click to expand...

He is a VMware employee, and it is not really FUD tbh. If you're doing this on ESXi standalone servers, I don't see the huge deal, especially for learning. Using this for keeping important data (seeing as this is a Storage Controller) seems like a very bad idea. That isn't FUD, it is common sense unless you're cavalier enough to absorb the risk. I've used pass through for a lot of things testing, and I've had very mixed results. A lot of stuff works, but I don't think I could say almost everything does, it isn't good enough for that yet. Most likely, that is a result of VMware's implementation thus far, I think that is fair to say. In the enterprise, the thought of doing anything that pidgeon holes a VM to a particular host is mind-numbingly stupid anyway supported or not.

I imagine lopoetve is used to supporting production customers all day, and I doubt he can easily say do this when he's seen people try it with important data. From his point of view, customers don't call with success stories and say hey look what I made work, its usually a sob story trying to save an impending disaster, so I'd cut him a small break on his hesitation to give any green lights.

There are other issues with using VMDirectPath I/O, like the required reservation equal to the VMs configured memory. Snapshots no longer work, etc. Reservations in a prod environment can have an effect on HA slot sizing if you're not careful in your design, and this of course would completely break DRS/vMotion for the VM. I can understand the use case for pass through, for instance I used it regularly (pre-vSphere 4.1), to pass through the USB controllers from ESXi on the standalone branch servers we had at the time. It was just for the APC UPS software to work, so no data was at risk, but the above stipulations really put a kink the overall benefit.
 
thanks for the info. can you clarify the impression i was getting from him that disk I/O with emulated environment is not supposed to be slow? e.g. passing 5 disks into a guest that having it do I/O to the RAID5 array as opposed to the same code on the raw metal (e.g. i create a single large file on vmfs datastore on each disk and pass that vhd into the guest and have it build a raid5 array on those 'disks'). i typically see about 50% performance hit when doing that, but he tells me that should not happen...
 
danswartz said:
thanks for the info. can you clarify the impression i was getting from him that disk I/O with emulated environment is not supposed to be slow? e.g. passing 5 disks into a guest that having it do I/O to the RAID5 array as opposed to the same code on the raw metal (e.g. i create a single large file on vmfs datastore on each disk and pass that vhd into the guest and have it build a raid5 array on those 'disks'). i typically see about 50% performance hit when doing that, but he tells me that should not happen...
Click to expand...

Of course directpath-IO to the controller is faster than using an vmfs emulated disk on the same controller. That point is provable.

The question is whether or not the difference is 50% or something more tolerable (like less than 5% degradation - still slower, but close enough that the hassles associated with directpath are not worth the trouble).

I've seen many people's benchmarks on this question and done my own recently too. It is clear that the degradation is not 50%, but it is far more than 5%. I won't bother to debate what the exact degradation is, but it is definitely enough that an application that cares more about disk IO performance than features like VMotion or snaps would be concerned.

And it is disingenuous for VMware to claim that vmfs emulation is just as fast and that there is no degradation. It would seem more appropriate for VMware to be transparent about the difference, characterize it (benchmark it), and discuss the trade-off against losing access to their advanced features. Makes a lot more sense than trying to claim there is no performance difference when anybody with a brain knows that there must be (but it might not matter).
 
defuseme2k said:
He is a VMware employee, and it is not really FUD tbh. If you're doing this on ESXi standalone servers, I don't see the huge deal, especially for learning. Using this for keeping important data (seeing as this is a Storage Controller) seems like a very bad idea. That isn't FUD, it is common sense unless you're cavalier enough to absorb the risk. I've used pass through for a lot of things testing, and I've had very mixed results. A lot of stuff works, but I don't think I could say almost everything does, it isn't good enough for that yet. Most likely, that is a result of VMware's implementation thus far, I think that is fair to say. In the enterprise, the thought of doing anything that pidgeon holes a VM to a particular host is mind-numbingly stupid anyway supported or not.

I imagine lopoetve is used to supporting production customers all day, and I doubt he can easily say do this when he's seen people try it with important data. From his point of view, customers don't call with success stories and say hey look what I made work, its usually a sob story trying to save an impending disaster, so I'd cut him a small break on his hesitation to give any green lights.

There are other issues with using VMDirectPath I/O, like the required reservation equal to the VMs configured memory. Snapshots no longer work, etc. Reservations in a prod environment can have an effect on HA slot sizing if you're not careful in your design, and this of course would completely break DRS/vMotion for the VM. I can understand the use case for pass through, for instance I used it regularly (pre-vSphere 4.1), to pass through the USB controllers from ESXi on the standalone branch servers we had at the time. It was just for the APC UPS software to work, so no data was at risk, but the above stipulations really put a kink the overall benefit.
Click to expand...

This is a much more well reasoned response than "it doesn't work". Thank you.
 
okay, piglover, thanks. i have a production VM running on the iscsi datastore. later tonight, i will migrate it to the sata datastore, and repeat the test i was referring to.
 
PigLover said:
@lopoetve: backing off, eh? Just a few posts ago you claimed "almost certainly won't work". Now its "some things work, some don't" (which is pretty much what the people you are arguing against have been stating).

The truth is even better than you claim. The FACT is that MOST things work and a very, very few don't.

Basically, its "not supported", which means if you call up VMWare for support they will tell you do go away. But that is a long, long way from "almost certainly won't work", isn't it? But then again, nobody expects them to "support" the hundreds of devices that are out there, do they?

Unless the OP is looking for a commercial production system, "support" from VMWare probably isn't very interesting to him anyway. And from the tone of the post that is not what he is looking for.

Regardless of its source (including the possibility that you are a VMWare employee), your "infinitely better" information appears to be FUD.
Click to expand...

Heh. Most definitely doesn't work -> It may appear to work, but in my experience, most things with directpath that "appear to work" end up being "well, that was a pretty fireball" later on. ;) Except for the tested and supported stuff, which had its fair share of issues early on too.

I come from a purely production level experience - I recommend things that I know will work, stably, reliably, and without fault (data is important!) when it comes to storage. That's the one place you can't afford to have something go wrong, because when you do, there may be no way to get it back.

danswartz said:
thanks for the info. can you clarify the impression i was getting from him that disk I/O with emulated environment is not supposed to be slow? e.g. passing 5 disks into a guest that having it do I/O to the RAID5 array as opposed to the same code on the raw metal (e.g. i create a single large file on vmfs datastore on each disk and pass that vhd into the guest and have it build a raid5 array on those 'disks'). i typically see about 50% performance hit when doing that, but he tells me that should not happen...
Click to expand...

There should not be a significant overhead on VMFS/VMDK at this point. If there is, that's something that should be investigated and fixed. I saw your reply in the other thread and I'm gonna dig into that and see what I can find. Can you do some iometer tests for IOPS levels (can't remember what OS you listed)? That's what I'm normally most concerned with. Also, is this only with solaris, or also with other POSIX-esque operating systems?
 
PigLover said:
Of course directpath-IO to the controller is faster than using an vmfs emulated disk on the same controller. That point is provable.

The question is whether or not the difference is 50% or something more tolerable (like less than 5% degradation - still slower, but close enough that the hassles associated with directpath are not worth the trouble).

I've seen many people's benchmarks on this question and done my own recently too. It is clear that the degradation is not 50%, but it is far more than 5%. I won't bother to debate what the exact degradation is, but it is definitely enough that an application that cares more about disk IO performance than features like VMotion or snaps would be concerned.

And it is disingenuous for VMware to claim that vmfs emulation is just as fast and that there is no degradation. It would seem more appropriate for VMware to be transparent about the difference, characterize it (benchmark it), and discuss the trade-off against losing access to their advanced features. Makes a lot more sense than trying to claim there is no performance difference when anybody with a brain knows that there must be (but it might not matter).
Click to expand...

That's just it - it's supposed to be pretty negligible at this point, or at the very least very minimal. If you all are seeing things that are far more than just negligible, I'll try to get them fixed - that's not something we should be seeing.
 
okay, cool. my most recent test was with freebsd 8.2 (zfsguru), but i can also try this with openindiana, which i have available. stay tuned... :)
 
I'm not clear on somethings discussed, is RDM not advisable on any whitebox hardware? I read in another thread about RDM and thought it could be a solution to my problems, (able to move drives to/from real servers and most importantly, not having to datashuffle my 8tb raid array) So as a test I made a 2tb drive a RDM, and it working for now. But after reading lopoetve's "well, that was a pretty fireball" comment, I'm a bit puckered. :eek:

This is an Gigabyte GA-MA785GM-US2H, FYI.
 
newdamage1 said:
I'm not clear on somethings discussed, is RDM not advisable on any whitebox hardware? I read in another thread about RDM and thought it could be a solution to my problems, (able to move drives to/from real servers and most importantly, not having to datashuffle my 8tb raid array) So as a test I made a 2tb drive a RDM, and it working for now. But after reading lopoetve's "well, that was a pretty fireball" comment, I'm a bit puckered. :eek:

This is an Gigabyte GA-MA785GM-US2H, FYI.
Click to expand...

So, that's a bit of a different story. RDM's are just passing through the underlying volume (not the hardware) to the VM. Local RDM's are filtered due to various things that can happen when doing some clustering bits, pass through the boot volume, etc, but there is nothing inherently "broken" about them that causes fireballs, or even significant sparks, as long as you're not trying to do some kind of goofy clustering in a box on them :) The whitebox portion isn't the issue, it's the local storage-as-an-rdm bit.

In fact, they were supported-ish in 3.0. I'd say you're probably fine, but then again, I wouldn't use them in a production environment where you're wanting prod level support - get a cheap array instead :)

That being said, there ARE whitebox solutions that are on the HCL (or have been, I haven't looked in some time).
 
danswartz said:
okay, cool. my most recent test was with freebsd 8.2 (zfsguru), but i can also try this with openindiana, which i have available. stay tuned... :)
Click to expand...

heh, the zpool is running rings around the linux vm (software RAID5) currently.

/me goes back to testing and fiddling.
 
Okay, here you are. Note: I had to step down from 5-disk raidz to 4-disk raidz due to being short on disks. The numbers are worse than I remembered, writes more so than reads. OpenIndiana was somewhat faster than zfsguru (freebsd 8.2), but not that much. Screenshot of the 4 results here:

http://www.flickr.com/photos/61659595@N02/5611537457/

Note also: I did not do any esxi tuning (if there is something I should have done, the defaults are not very good, IMO). My esxi setup was: create a datastore on each of the 4 data disks, then create a single huge virtual disk file on each of them, and add that disk file to the VM. I keep being told the different in performance should be minimal. 50% or so is not minimal. What am I missing?
 
That's what I'm trying to figure out.

Do me a favor... I'm gonna be stuck with limited hardware (just given what I have to work with) - test on 2 disks (i know, not much of a zpool :(), as I think that's what I'm giong to be able to get. I'll be back in the office and testing more soon.
 
Okay, I will try with a 2 disk mirror. I can do the virtualized one now, the physical one will have to wait until I get home.
 
Just so I know I'm not a dummy, there is no tuning or tweaking I should need to be doing?
 
Another variable I can check: like I said earlier, the data drives are on the mobo sata controller, which is nvidia chipset (mpc555 or somesuch). I have a relatively low-end LSI logic 4-port sata card, which is running the data drive. I can try swapping the drives so that the 4 drives are on the LSI card and the data drive/optical drive on the mobo ports. It's possible maybe that the vmware driver for the nvidia chipset has a bug/issue - this would explain why running zfsguru or OI natively had good performance. That also needs to wait until I am home.
 
No major tuning that I'm doing at this point. Performance should be very similar to physical (especially on the storage side) with local disks. Sorry, been a bit tied up at work.
 
Righty-o. I got tied up writing code today and tomorrow, so I hope to get back to this soon.
 
Boy this gets weirder and weirder. After experimenting and googling I discovered that since my HBA makes the sata drives look remote, I can use RDM mapping in the GUI to pass them thru to the guest. I can only do 4 drives (only 4 ports on the HBA), but I gave it a try using zfsguru.

VM/write 123MB/sec VM/read 169MB/sec
phys/write 299MB/sec phys/read 317MB/sec

so, this *seems* to have nothing to do with vmfs3 then, no? Totally stumped here...
 
OpenIndiana virtualized (same 4-disk RDM setup as above). write = 145MB/sec read = 262MB/sec

Kinda surprised the writes are that much slower. I wonder if there is some esxi tuning for host caching or whatever? Anyway, the write speed might be acceptable, keeping in mind this is primarily a backup server, and the gating speed will be the 80MB/sec or so of windows clients writing to it over (at best) gigabit enet. I'll have to set up a iscsi target and see...
 
You must log in or register to reply here.
Back
Top