ESXi 5 - local datastore gone after PCIe passthrough

S

Spike42

n00b
Joined
May 15, 2012
Messages
7
Hi,
on my ESXi 5 I'm trying to passthrough the onboard SATA controller to use the connected HDDs in a fileserver VM.

Therefor I install ESXi on SSD connected to an ASRock SATA3 PCIe controller and checking the onboard cougar point SATA AHCI controller for passthrough (see screenshot). But after rebooting ESXi the local datastore on the SSD is gone with every VM I created before. When I try to create a new datastore the SSD is not listed as storage device.

Moreover after unchecking passthrough on the controller and rebooting ESXi the controller is still checked for passthrough. The only way to undo this is reinstall ESXi.

Any idea what I'm doing wrong? Any help will be appreciated!



Used hardware:

Core i5-2500 (without 'k')
MSI H67MA-E45
16GB RAM
SSD Crucial m4 128GB
ASRock SATA3 Controller (http://www.asrock.com/mb/spec/Card.us.asp?Model=SATA3 Card)


 
As far as the local datastore no longer being visible, this is expected behavior.

When you pass a device through to a VM, the device is no longer usable directly by ESXi.

Also, you have to pass through the entire device, so if you pass through the SATA controller, ALL sata ports are passed through to the VM.

The content of the datastore is likely still on the drive, but since you passed it through to the VM the VM would see the drive instead of ESXi, but it never does because it can't boot from the drive because ESXi doesn't see it...

If you want to have a local data store AND pass through your SATA controller you need to have two separate controllers. Some motherboards have this, or you can just add another one via PCIe or PCI.

As far as how to undo the change, I am not entirely certain. ESXi has some odd quirks regarding stuff like this sometimes. I'm sure someone will pop in and tell you how to get the unchecked box to stay so you get your datastore back and visible again.
 
Zarathustra[H];1038730949 said:
If you want to have a local data store AND pass through your SATA controller you need to have two separate controllers. Some motherboards have this, or you can just add another one via PCIe or PCI.
Click to expand...

That is exactly what I'm doing! I'm using the ASRock SATA3 PCIe controller to connect the SSD. So the onboard SATA controller is completely unused at the moment.

But as soon as I check passthrough for the onboard SATA controller, ESXi still boots but the local datastore is gone.
 
Spike42 said:
That is exactly what I'm doing! I'm using the ASRock SATA3 PCIe controller to connect the SSD. So the onboard SATA controller is completely unused at the moment.

But as soon as I check passthrough for the onboard SATA controller, ESXi still boots but the local datastore is gone.
Click to expand...

That is a little odd.

Do you know if your motherboard uses any kind of PCIe bridging chip? It could be that the slot in which you have the SATA controller installed is somehow identified by ESXi as being on the same PCIe channel as the onboard SATA. If possible, try moving it to a different slot, and try again?
 
I didn't found anything about that in the manual but I found the attached image.

I'll try the other PCIe slots especially the PEG slot and post the results.

Thanks for your replies so far.


 
Hm, no success on the other slots.....
Perhaps the board is not able to passthrough? But it's selectable in the BIOS.
 
Spike42 said:
Hm, no success on the other slots.....
Perhaps the board is not able to passthrough? But it's selectable in the BIOS.
Click to expand...

If the option to pass through shows up in ESXi, the it typically works, as long as the device you are trying to forward is compatible. Not all are.

If your system were not capable of VT-D, you'd get something similar to what I get on mine:

6888848773_85d17bbf25_b.jpg


As you can see towards the bottom on the left it says "DirectPath I/O: Not supported"

Apparently the non-K sandy bridge models support VT-D, but the tricky part is finding motherboards that do. Yours appears to, so I don't know what the issue is.
 
If you can figure out how to reverse what you did, maybe you'd have more luck if you install the local data store on the on board SATA and forward the addon board to the VM instead?

Some devices just don't like being forwarded (though I feel like I have read about people forwarding Intel's on board SATA before...

I fear I have reached the end of my knowledge on the subject. Hopefully someone else can help you out.
 
Direct Path I/O is shown supported.

I can't reverse passthrough of the onboard SATA controller but simply reinstall ESXi ;-)

My first try was to use the onboard SATA for the SSD and pass through the ASRock PCIe SATA controller. It didn't worked. I thought the ASRock controller doesn't support passthrough so I tried vice versa.

See the screenshot in my first post: 00:1f.2 Cougar Point SATA Controller is the onboard controller I try to pass through. The entry below is the ASRock PCIe controller in PCIe slot. It's greyed out so I thought it does not support passthrough...

As next step I try Proxmox (I used before) instead of ESXi .... or I should buy a NAS for iSCSI storage ;-)

Thank you very much for your help!
 
I am very interested in your results here.

I have an asRock x58 supercomputer mobo, i7930 and they included the exact same 2 port 6Gb/s PCIe sata card.

Running ESXI 5.0 I successfully passed through the onboard 6 port sata controller but never got the two port card working. I ended up putting the datastore on a SIL3114 PCI sata card I had. It works like a charm. I could not get the 2 port card working in ESXI or passedthrough to a VM. I only tried it in my motherboards bottom 2 PCIe slots. Good Luck!
 
Spike, did you ever find a solution for the two port card. I'm in need of an additional port now and the 2 port card would be perfect
 
No, I didn't. I tried Proxmox but had no success. Perhaps I will try to passthrough other cards but at the moment I have no time to do that.
 
I now tried ASRock B75 Pro3-M with Windows 7 64Bit VM:

- Passthrough of onboard Panther Point ports doesn't work. Same problem as above (datastore gone)
- Passthrough of ASRock SATA3 Card doesn't work.
- Passthrough of "onboard ASM1061" works!
- Passthrough of simple "Silicon Image 3114" 4 port PCI card (like this one) works - even with two cards in both PCI slots. Both PCI slots are using the same pci-bridge so you can only passthrough both slots to the same VM, I guess. I know that PCI is a little bit slow for multiple SATA drives.

Hope this helps!
 
Last edited:
I have an Asrock Z77 Extreme4-M motherboard and couldn't get passthrough working. As per post above, I installed on drive attached to Panther Point Controller & passed through the ASMedia controller.

However, this drive was not visible. I ssh'ed into the box and a ls -l /dev/disks did not show the physical disk.

But, I solved my problem by using RDM, Raw Device Mapping, as described here:

http://blog.davidwarburton.net/2010/10/25/rdm-mapping-of-local-sata-storage-for-esxi/

I'm very pleased with this solution and it seems far more flexible than pass through. If you read the comments, it seems there is an easier way to do RDM, which I haven't tried yet.

In esxi 5, goto:

Configuration > Software > Advanced Settings > RdmFilter and there you can disable the filter so LUN’s can be RDM’d directly from the “add disk” dialogue.
(not supported on all hardware, but it does work on all LSI IT flashable cards (SAS2008 and the likes)
 
I've heard of people getting corruption with RDM (particularly I have seen some ZFS users mention it, and I use Illumian with ZFS so...), so I personally don't trust it.
 
Well, I got corruption (and complained), but to be fair, I was doing something unsupported. e.g. some hack I read about on a blog where you RDM local physical disks. Technically, disks on a supported HBA are treated as remote, and that IS supported. That said, I still prefer pass through so you can hotplug, see smart info, etc...
 
Well, I was not aware of the corruption potential when I posted. I will continue to run my disks as RDM and see what, if any corruption, I get. What OS did you get the corruption with and what version of ESXi?
 
ESXi 4.1 don't remember the OS. Keep in mind this was the hack where you patch up rdm files by hand -possible I messed up, but I've seen mention by others.
 
Spike42 said:
Hi,
on my ESXi 5 I'm trying to passthrough the onboard SATA controller to use the connected HDDs in a fileserver VM.

Therefor I install ESXi on SSD connected to an ASRock SATA3 PCIe controller and checking the onboard cougar point SATA AHCI controller for passthrough (see screenshot). But after rebooting ESXi the local datastore on the SSD is gone with every VM I created before. When I try to create a new datastore the SSD is not listed as storage device.

Moreover after unchecking passthrough on the controller and rebooting ESXi the controller is still checked for passthrough. The only way to undo this is reinstall ESXi.

Any idea what I'm doing wrong? Any help will be appreciated!


Used hardware:

Core i5-2500 (without 'k')
MSI H67MA-E45
16GB RAM
SSD Crucial m4 128GB
ASRock SATA3 Controller (http://www.asrock.com/mb/spec/Card.us.asp?Model=SATA3 Card)


Click to expand...

i had the same problem with my P8B Workstation... when i gave a virtual machine access to the controller my datastores where gone (selected only the sata 600 ports)... the solution is a dedicated controller (i bought a IBM M1015 and all the problems are solved)
 
yeah, there are things that can go wrong with local disks and RDMs as we assume they're a real remote device. Passthrough works better but the motherboard design depends on "how" it works as it's all in the trace design and PCIE layout.
 
Lopoetve, can you give me the cliff notes explanation as to why a disk on an HBA is considered 'remote'? Not complaining, just wondering what 'local' vs 'remote' means in this context :)
 
danswartz said:
Lopoetve, can you give me the cliff notes explanation as to why a disk on an HBA is considered 'remote'? Not complaining, just wondering what 'local' vs 'remote' means in this context :)
Click to expand...

So, HBA is a misnomer - in effect, all "local" controllers are HBAs the same as any other - they connect the PCI bus to a SCSI bus.

Except local controllers aren't SCSI for what most people are doing. So, we're expecting something that fully supports all SCSI commands, and doesn't really support them. In addition, RDMs have a pair of commands that are modified to them at all times: REPORT_LUNS, which doesn't matter, and BUS_RESET, which we convert to a LUN_RESET (because resetting a bus is good for local devices, which the VM thinks it has, but a really bad idea for SAN devices). Thus, we're trying to change commands to something that a remote device would normally be perfectly happy with, but a local controller may go "WTF" to. On top of that, we're expecting a certain set of responses to INQUIRY that a local device won't return (Because it doesn't support them), which means we're making assumptions about device capabilities that may not exist :) End result - we send commands, local controller goes "... ?!? ... purple? ... YAHTZEE!" and your VM gets unhappy.

In effect, we're expecting to be talking to a certain device (a remote RAID controller on an array), and instead talking to something totally different (a single, local SATA disk most of the time), and the commands aren't 100% compatible.

It works "better" on certain local true SCSI controllers, as they can interpret the commands, but they're still not 100% identical to what a SAN or remote device would present, so there are still potential issues there. :)

That help?
 
Wow, yes, thanks :) I was curious because building an all in one, before setting up passthrough, esxi reported the drives on the m1015 as 'remote'.
 
And, does this mean that the disk I/O performance will be less than optimal? I've just started playing with Esxi and trying to understand the environment and performance of various Java applications.

And, does that mean for disk intensive applications, it's best if the datastore resides on a SAN?
 
danswartz said:
Wow, yes, thanks :) I was curious because building an all in one, before setting up passthrough, esxi reported the drives on the m1015 as 'remote'.
Click to expand...

That means that your controller returned a full VPD Page 0x83 to our INQUIRY, and reported remote capability - or something that sure as hell smacked of all of that, so we enabled it because we thought they might actually be on a DAS / SAS array.
 
Chiphead said:
And, does this mean that the disk I/O performance will be less than optimal? I've just started playing with Esxi and trying to understand the environment and performance of various Java applications.

And, does that mean for disk intensive applications, it's best if the datastore resides on a SAN?
Click to expand...

VMFS and RDM are within a percentage point of each other for most things.
 
lopoetve said:
That means that your controller returned a full VPD Page 0x83 to our INQUIRY, and reported remote capability - or something that sure as hell smacked of all of that, so we enabled it because we thought they might actually be on a DAS / SAS array.
Click to expand...

the m1015 is a rebadged LSI SAS HBA, so it seems not unreasonable it would respond to the above. again, thanks!
 
You must log in or register to reply here.
Back
Top