• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

ESXi 5 second AHCI adapter gone when passthrough enabled

C

CrEOF

n00b
Joined
Apr 29, 2011
Messages
5
I am having difficulties passing through the onboard AHCI adapter and continuing to use a PCIe AHCI adapter for my datastore. The hardware setup is:

Apple Mac Pro (MacPro5,1 with onboard ICH10 AHCI SATA adapter)
4x WD Red SATA disks

PCIe Apricorn Velocity Solo x1 (Asmedia ASM1062 AHCI SATA adapter)
Samsung 830 SSD

CPU, memory, NICs, etc.

The goal is to passthrough the 4 WD Red disks containing a ZFS pool on the ICH10 controller to a Solaris 11.1 VM. In addition to being unsupported, using manually created (hacked) RDMs with Solaris 11.1 PSODs the ESXi server.

Most of my hardware is handed down, and I can't afford to purchase a SAS HBA and external enclosure at the moment. I finally came upon the PCIe controller above. The only other inexpensive PCIe SATA adapters I could find were based on the Marvell 9182 controller. In addition to being unsupported (though the AHCI driver can be hacked) it also has a DMA/IOMMU bug which prevents the use of DirectPath I/O.

The ASM1062 is on the VMware HCL. I am able to install ESXi 5.1u1 to the SSD attached to the card, and the system and VMs contained in the datastore on it run without issue.The problem arises when I enable passthrough for the onboard ICH10 controller.

After rebooting the server stops loading the ahci module and complains there is no datastore. I can however ssh into the server and manually vmkload_mod ahci, do a rescan, and then access the datastore on the SSD without issue. After doing this though the vSphere console complains about crash dump and logging locations, and the later requires a reboot to take effect - at which point the ahci module again hasn't loaded.

Once the ahci module is loaded the Solaris 11.1 VM using the passthrough ICH10 operates as expected without any issues (import ZFS pool, etc.)

All this leads me to believe this a strictly software issue, and some configuration tweak must exist to overcome this problem. I've tried creating a number of different storage claim rules but have thus far been unsuccessful. I've come across a number of reports of the same/similar issue (http://hardforum.com/showthread.php?t=1693500 for example), but without any solutions short of purchasing hardware.

Some current thoughts are changing the module order in the boot.cfg file, or creating a custom VIB containing a renamed ahci module and only the PCI device/vendor id's for the ASM1062. Both of these seem a little hackish and I would much rather stay off that path if possible.

Any help or suggestions would be greatly appreciated!
 
What about creating a physical RDM to the VM using the 4 WD red disks?
(i've never personally tried this with local storage but sure there's a way to make it work)
 
When using physical RDMs a few seconds after the Solaris 11.1 VM comes up all the way it crashes/PSODs the ESXi server, so bad the server doesn't even create the crash dump file or auto restart (it is configured to do this after 30 seconds). I spent quite a while trying to resolve this issue myself, but eventually came across these two posts:

http://effnorwood.com/unix/vmware-vsphere-esxi-5-1-and-solaris-11-1-with-rdm-do-not-do-this/15/
http://timkieschnick.blogspot.com/2013/01/storage-server-gone-green.html

Every time it crashed at the same instruction mentioned in note #2 of the second URL. It appears some change in Solaris 11.1 causes this. Virtual RDMs aren't an option as the drives are 3TB each.
 
I must have gotten something mixed up somewhere along the line. I took another look at the device ID's for the ahci module and the ASMedia device (1b21:0612) is not listed.

Once I realized this I was able to get it working. Using the ESXi5 Community Packaging Tools (http://esxi5-cpt.v-front.de) I created a new VIB containing a single line file in /etc/vmware/driver.map.d with the correct PCI id and specifying the ahci driver. For completeness I also included a file with the device name in /usr/share/hwdata/driver.pciids.d. I packaged up the VIB and installed it, enabled passthrough on the ICH10 device, and rebooted. Server came back up and the datastore was detected.
 
Good that you got it working though it confused me what you were saying about SAS HBAs. You do not need an external enclosure to use one (why would you think that?) and the common IBM M1015 is only about $60-100 on Ebay. If you only need 1-2 drives then yes the crappy 2-4 port consumer cards can be cheaper, but those cards are pretty much junk, and not THAT much cheaper.
 
dandragonrage said:
Good that you got it working though it confused me what you were saying about SAS HBAs. You do not need an external enclosure to use one (why would you think that?) and the common IBM M1015 is only about $60-100 on Ebay. If you only need 1-2 drives then yes the crappy 2-4 port consumer cards can be cheaper, but those cards are pretty much junk, and not THAT much cheaper.
Click to expand...

You are absolutely correct. I hastily skipped over explaining my requirements and design goals.

My current server is a 2010 Apple Mac Pro. It has 4 internal SATA/SAS bays which connect to the the onboard ICH10 controller through a backplane built into the motherboard. The only way to access these ports (other than with the ICH10) is using an Apple Mac Pro RAID card, as the SATA/SAS signals are routed through unused (I can only guess) lines in PCIe slot 4. I'm usually a Apple fan, but this design is absolutely f*****g stupid! The pre-2009 Mac Pro's where not designed this way, and had a Mini-SAS connector on the motherboard.

There is an aftermarket product which allows one to run cables to the 4 bays, but it's nothing more than an SATA power passthrough PCB with a right angle SATA connector. The cost for this setup is almost $150, which is a good bit towards an enclosure. On top of that upgrading anything other than memory on this server would be ridiculously expensive, so if and when I decide to move onto a new server the whole contraption would be useless.

Unfortunately the silly Apple RAID adapter isn't supported in anything other than OS X. I went as far as buying one to try, but the heat sinks were glued onto the controller chipset leaving no easy way to determine what drivers would need hacking. Even if I had, looking through the driver source of existing SAS controllers in the Linux/VMware kernel showed I'd most likely have to compile my own to modify the PCI device id's. That would have pretty much lowered my confidence in the resiliency of the entire ZFS setup close to nil.

So... external enclosure ;)

For the controller... I want to have the ability to boot from it with the EFI Mac while I'm using it as the server. This will also leave options open for the future, don't want to disregard the potential of getting a hand-me-down (as this one was) new Mac Pro with Thunderbolt PCIe cage :rolleyes: That eliminates a lot of the bargin buys.

So the cheap SATA card for my SSD/datastore is a short term solution while I continue to accumulate hardware.

After all this I purchased an ATTO SAS HBA yesterday since the price was right, so this whole adventure may have been an incredibly time consuming detour. Solaris 11.1 is working like a champ though so at least it's a scenic one :cool:
 
You must log in or register to reply here.
Back
Top