Problems with IBM M1015's in passthrough mode in ESXi

I am trying to setup an unRAID server running on ESXi. I have 3 IBM Serveraid M1015 8-port HBA's that have been flashed to IT modeand I'm running them on a SuperMicro X9SCM-F mobo. ESXi sees all three cards and they are all set to passthrough and assigned to the unRAID server.

The problem is that unRAID is not seeing any drives connected to one of the cards. The unRAID log sees all three cards shown as mpt2sas0, mpt2sas1 and mpt2sas2. The problem is that when it gets to mpt2sas2 it throws the following error:


As far as I can tell, it's erroring on the card that is in the 4x PCI-E slot. If I move them around so that I have one card in the 8x slot and two in the 4x slots, unRAID will show that same error on mpt2sas1 and then never even see mpt2sas2.

The strange thing is that if I run unRAID in the same system outside of ESXi, all three controllers show without errors and all my drives are accessible. So it would seem to be something in ESXi causing the problem. This is my firat go around with ESXi so I may very well have something set up wrong, but I can;t seem to figure it out and the unRAID forums haven't been able to help either. Any help would be greatly appreciated

I seem to recall esxi only supports two pass through cards per guest. Why it even let you add the third is a good question.

Well that would be a bummer if true. The other issue then would be why it errors on the second card and doesn't even see the third if two are in the 4x slots. It's like as soon as it encounters a card a 4x slot it errors.

Actually, I misread. The limit of two was in ESXI 4.0, in 4.1 and later it supports 4. Again, though, unsupported, so who knows why it fails. You'd be better off getting a jbod chassis/expander and going that route maybe?

Can you show a screenshot of ESXi/VCenter's display of your devices? Your 4x slots are not behind additional components such as switches, are they? What ESXi version? Latest firmware and drivers for the cards? Latest motherboard EFI version? (I like to make sure everything is the latest version before doing other troubleshooting, personally. As a programmer, I cannot stand it when people report bugs using old versions that were already fixed in a newer version, and they didn't even think to try that.)

Keep in mind that this is my first time with ESXi and my first time with true server hardware.

ESXi ver: 5.0.0, 623860
M1015 cards: all P14 firmware (latest as far as I can tell). Their drivers are built in to unRAID and i can't change them.
Not really familiar with EFI, but it reports EFI revision 4.653

I'm not sure what you mean by "Your 4x slots are not behind additional components such as switches, are they?" Here is the screenshot:


Untitled by rockdawg2232, on Flickr

I should mention that I did try removing two of the cards and leaving the lone card in one of the 4x slots and I get the same error and no drives show in unRAID. Again, if I boot straight into unRAID (bypassing ESXi) unRAID boots without the error and all drives show.

Sounds like it's something with that slot then

It happens with either 4x slot, but only through ESXi. Without using ESXi everything works fine.

Sorry, what I meant was: something about the wiring/whatever for that slot that causes grief for passthru.

I have noticed one difference in the unRAID syslogs between straight unRAID and unRAID through ESXi:

Straight unRAID

unRAID through ESXi

Why wouldn't they both be 64 BIT?

One would think they would, but again, remember this is unsupported, so who knows what is being presented by unraid?

I realize that a lot of stuff is unsupported in ESXi. I'm talking with another unRAID user who is using the same setup as I am trying here and it works fine for him though his cards show as 64 bit when run through unRAID.

Exact same setup? e.g. same mobo+3 cards in exact same slots? In your case, is the hba that fails showing as 32-bit and the others as 64-bit?

The guest is 64-bit and ESXi is set to 64-bit, right?

And the screenshot I was talking about is in Configuration tab -> Hardware box -> Advanced Settings. The screen's titled "DirectPath I/O Configuration"... Hit the edit button (that part's important) and show us a screenshot of that if possible please.

EFI version is like BIOS version. EFI replaced BIOS... Sometimes updating BIOS/EFI can fix issues with VT-d (IOMMU).

unRAID is 32-bit Slackware. I never manually set ESXi to 64-bit.


Untitled by rockdawg2232, on Flickr

I'm wondering about those "Unknown PCI/PCI bridge" entries and if they could cause problems. Didn't realize UnRAID only came in 32-bit... Not really a fan of it, personally.

Any suggestions how to deal with that?

Well, personally I always recommend the latest BIOS/EFI especially when using VT-d because so many motherboards have had problems with it and some have fixed the problems with later releases. Latest release is from June of this year. The version of the download is probably not the same version number that you are talking about. http://www.supermicro.com/support/resources/results.aspx

Well if the BIOS and EFI are tied together, then I have the latest because my BIOS version is 2.0a. I don't see a separate download for EFI

I tried installing a spare PCI-E NIC I had laying around and if I install it any of the top three slots, ESXi shows Generic PCI/PCI Bridge. If I put it in the last slot, it will show Cougar Bridge...

Since all three of the M1015's work outside of ESXi, and I've seen reports that other people's Supermicro X9SCM-F-O's work just fine with this very same configuration, I'm assuming something may be amiss with the mobo so I'm going to replace it.

I have similar problem, same Supermicro X9SCM-F, same firmware version 2.0a, same three M1015 cross flashed to IT mode, same ESXi 5u1, same "mpt2sas2: _base_wait_for_doorbell_int: failed due to timeout count(10000)" message, same working fine out of ESXi, but I'm using Debian 7 instead of unraid.

Anytime I assign the third M1015 to any guest, the guest will took forever/never to boot and the card won't work, assign only the first or the second or first + second is fine.

By "any guest" I mean Debian 6/7 and/or Windows 7, Debian 7 prints the exact same messages during boot like yours which leads me here, Debian 6 prints similar messages but not the same, I guess that's because its relatively old kernel, they took too long to boot that I lost my patience and shut them down. Windows 7 boots, but there is a yellow exclamation mark on the card in device manager, which I didn't bother to test if it's working.

I don't think the "64/32 BIT DMA" message has anything to do with this problem, you might have x86 version of unraid instead of x86-64 or you just didn't set enough RAM(total mem (2074648 kB) vs total mem (16608336 kB)) for 64 bit DMA to be useful.

And I would like to point out that is "Unknown PCI/PCI Bridge" rather than "Generic PCI/PCI Bridge", but I don't think this is a problem, you must be using an Ivy Bridge CPU like me(Xeon E3-1230V2), which is - I guess - too new for it's PCI bridge to be recognized by ESXi.

Someone on another forum reported good result on the same mobo and cards, I noticed that he said firmware version 1.0, that indicates he is using a Sandy Bridge CPU(this mobo needs 2.0 firmware to support Ivy Bridge), I think that might be the key difference, ESXi not compatible with IB or firmware 2.0, or firmware 2.0 has buggy VT-d/PCI passthrough support.

I bailed to plan B: use Debian or FreeBSD as my primary OS and use Xen or VirtualBox to host other guests, I don't have time to wait for this bug to be fixed. so I'm out of this situation. but I have some suggestions you might need:

1, By finding me - another instance of the very same problem, I think you can rule out any faulty hardware and/or installation mistake.

2, Does M$ Hyper-V supports PCI passthrough? if positive I suggest you take a test on Hyper-V, if the symptom disappear then we can confidently narrow it down to ESXi otherwise Supermicro.

3, I guess supermicro is the one to blame, but that's only a guess.

Thanks a ton for that feedback. You're right, it should be "Unknown PCI/PCI Bridge" and it makes sense that the Ivy Bridge processor might be the problem. The couple people I've talked to who have everything working are using the E3-1230 like me, but they are running V1 while mine is V2. Maybe I'll sent the processor back and get the V1.

We had a problem like this with ESXi also (passing through 4 RAID cards to the same guest), moved to Xen on that system instead and it's been fine ever since.

Yeah, Xen is better at IOMMU than ESXi. In fact it has the best hypervisor support for IOMMU that I know of.

I assume you guys are talking about Xen.org and not Citrix's XenServer, correct? If so, I don't think it would be good for me as it looks like it's CLI oriented and I'm not that savvy. Especially in Linux.

bozrdang said:

I assume you guys are talking about Xen.org and not Citrix's XenServer, correct? If so, I don't think it would be good for me as it looks like it's CLI oriented and I'm not that savvy. Especially in Linux.

Click to expand...

I've never used Citrix's version, but from my understanding, they lag on features but offer support whereas the OS version tends to get features first but has little/no support.

Xen Cloud Platform is what you'd want to check out, but it's definitely not as easy to use as VMWare vSphere.

bozrdang said:

I assume you guys are talking about Xen.org and not Citrix's XenServer, correct? If so, I don't think it would be good for me as it looks like it's CLI oriented and I'm not that savvy. Especially in Linux.

Click to expand...

There's Xen Cloud Platform which has a web-based GUI application. CLI use is relatively minor on an established system, but I wouldn't dive in head-first without at least some basic *NIX skills. Might be worth evaluating on a test system or in a VM and seeing if it suits your needs. As Dan said, Xen's IOMMU implementation is top-notch and a full part of the "supported" software package. With ESXi it sometimes feels like you are flying blind and in unknown territory when working with complicated pass-through setups, but you never get that sense with Xen, since there's users who are doing all sorts of crazy stuff like passing through 8 RAID cards or a quad-SLI setup who are happy to talk about their configs and help you out.

ESXi does not support PCI pass through on slots that are behind any kind of PCI switch or bridge. Your motherboard (X9SCM-F) uses a PCIe switch and you won't be able to use pass through for any board on those slots. Break out the motherboard manual and find the block diagram to see exactly which ports are 'native' and which ones are doubled up with the switch. As I recall, there are only two native x8 slots on that board and everything else is switched (well, except the x1 slot, which doesn't really matter for you).

This means, of course, that you can only use two m1015s in pass through mode,

VMWare released ESXi 5.1 yesterday. I installed it and the PCI-E ports are now recognized as Ivy Bridge. Unfortunately, that didn't solve my problem. I still get the handshake/doorbell fault!

PigLover - I haven't checked the manual yet, but there is no 1x slot and I;ve talked to people who have a working system in ESXi with the same hardware so it must be possible.

anyone check this thread out?
http://hardforum.com/showthread.php?p=1038273582

PigLover said:

ESXi does not support PCI pass through on slots that are behind any kind of PCI switch or bridge. Your motherboard (X9SCM-F) uses a PCIe switch and you won't be able to use pass through for any board on those slots. Break out the motherboard manual and find the block diagram to see exactly which ports are 'native' and which ones are doubled up with the switch. As I recall, there are only two native x8 slots on that board and everything else is switched (well, except the x1 slot, which doesn't really matter for you).

This means, of course, that you can only use two m1015s in pass through mode,

Click to expand...

I guess you checked the wrong doc

http://www.supermicro.com/manuals/motherboard/C202_C204/MNL-1270.pdf
page 1-10, the 1st/2nd/3rd slot comes out of CPU, the 4th slot comes out of PCH, no switch involved. and BTW the board has no x1 slot.

I finally found the problem! It turns out it was the Xeon E3-1230 V2 Ivy Bridge processor I was running. There is no sign it is defective as I had zero problems outside ESXi, but swapping it out for a Xeon E3-1230 V1 Sandy Bridge processor solved my problem. It seems like Ivy Bridge may currently be a problem in ESXi. Over at the unRAID forums a poster who was helping me was mentioning that he was having problems with a board that had a newer Ivy Bridge compatible BIOS even though he was running a Sandy Bridge CPU. He had the same board with an earlier BIOS that was before Ivy Bridge support and it worked fine in ESXi.

Food for thought!