IBM M5015 - poor performance with ESXi 6

B

BSDMan

Weaksauce
Joined
May 23, 2012
Messages
105
Hi All

Banging my head against the wall with this one! About 2yrs ago I bought an IBM M5015 RAID card with a BBU for my (at the time) ESXi 5.5 server. This was to improve the (write) disk performance. I went from +-20/MB/s without the RAID card to +-400MB/s with the RAID card.

After that I rebuilt the server with HyperV and removed the RAID card from the server. The RAID card hasn't been used in 18mths.

A couple months ago I rebuilt the server again but this time with ESXi 6 Update 2 and ever since I have been having terrible disk speeds/IO. The IBM M5015 has been installed in the server and I have 4 SSD drives connected to the RAID card using a single cable with 4 SATA connections on it. The RAID cards is plugged into a PCIe 3 x8 slot.

I did the same test on my ESXi 6 install as when I had the ESXi 5.5 install for testing disk speed and it went like this:

I have two Samsung Pro 840 128GB drives. Each one is setup as a datastore in ESXi 6. I then have two VMs, one runs on datastore one with disk 1 and the other VM runs on datastore 2 with disk 2. Each VM is using the vmxnet3 vNIC and is showing a speed of 10gb/s.

Yet, when I copy a 5GB ISO from VM1 to VM2 (or vice versa) I am getting well under 80MB/s! I have run out of ideas as to what I can try next but so far I have tried the following in the WebBIOS in the M5015:

1) Ensuring write back cache is enabled (with or without BBU)
2) I have tried enabling and disabling read ahead

(I have also tried wIth and without the battery installed while ensuring write back mode cache is enabled)

So what am I doing wrong here? Is this a problem with ESXi 6? Or have I forgotten to do something or done it incorrectly?

My 4 SSD drives connected to the RAID card are all in RAID 0 and consist of:

2 x Samsung Pro 840 128GB
1 x Samsung Pro 830 256GB
1 x Samsung Pro 850 512GB

Help please ;-) This slow IO is driving me crazy......

I don't have the exact firmware numbers available but I can get them if it'll help. I *vaguely* remember seeing the RAID firmware with the date of June 2013.
 
I removed the battery and set write through mode to always be used even with no battery and the performance was still poor.
 
You say each 128GB disk is its own datastore but then you also say all the disks are in a raid 0? Which is it?

Also, have you tried loading hdtach or any other hard drive test utility inside the VM to see what the performance is within the VM?
 
in some versions of the firmware for that card it will not use the memory without the battery no matter what the setting. It will also in some cases use verify writes(i forget the correct term atm) which is dog slow.
Using it without is a very bad idea anyway.
 
Eickst said:
You say each 128GB disk is its own datastore but then you also say all the disks are in a raid 0? Which is it?

Also, have you tried loading hdtach or any other hard drive test utility inside the VM to see what the performance is within the VM?
Click to expand...

In the WebBIOS, each disk is set in a RAID0 configuration on its own (ie: one disk is in the RAID0). I don't think theres any other way?

I did run ATTODiskBenchmark in the two VMs and the read/write speed seemed fine after running the benchmark but real world speeds were terrible (around 20MB/s).

I'm baffled because I have used this card before (even without the battery) and was able to get 400MB/s speeds. The only difference since I last used it is the version of ESXi (6.0 vs 5.5).
 
in some versions of the firmware for that card it will not use the memory without the battery no matter what the setting. It will also in some cases use verify writes(i forget the correct term atm) which is dog slow.
Using it without is a very bad idea anyway.
Click to expand...

I haven't changed the version of the firmware on the card since I last used it. Currently the only thing that has changed is the version of ESXi I am using (6.0 vs 5.5). When running ESXi 5.5 I did (temporarily) run it without a battery as when the RAID card arrived the battery was dead. So I set the cache mode to use the write back mode even if a battery wasn't present. I was still able to get around 400MB/s speeds when copying an ISO between VMs. I know not using a battery on a RAID card is a bad idea (my server is on a UPS) but right now all I care about is getting my storage speeds back to +-400MB/s. I'm willing to take the risk without running with a battery.

Could the firmware need updating? I'd also be interested to hear what other IBM M5015 owners set their WebBIOS settings to when using this card with SSD drives. I'm clearly doing something stupid/wrong!
 
I temporarily plugged the M5015 into my Win10 desktop and upgraded the firmware using MSM. I am now running the following updated version(s):

Firmware version: 2.130.403-4660
FW Package Version: 12.15.0.-0239

After updating the firmware I checked the BBU status in MSM. It was in a degraded state so I ran a relearn cycle and it FAILED. So I have physically unplugged the battery from the card (althought the board the battery was connected to is still connected to the card).

After doing all this I installed the M5015 in my server again and connected the two Samsung Pro 840 128GB drives to the card. I setup two virtual drives...each one using RAID0 and I first tried the following virtual drive settings/policies:

Read Policy: Normal
Write Policy: Always write back
IO Policy: Direct
Disk cache: disabled

I then setup two datastores in vSphere and, again, performance was poor/slow after testing.

I then rebooted and changed the virtual drive settings again in the WebBIOS to the following:

Read Policy: Ahead
Write Policy: Always write back
IO Policy: cached
Disk cache: enabled

and like deja vu, poor performance again.

So even though I have set the write policy to "Always write back" and I have no BBU, it looks like it is still running in write through mode (hence the poor write performance)? I'm just guessing here! Is there a way to FORCE the card to ALWAYS use write back mode even if there is no BBU/battery?

I'd like to avoid buying another (expensive) battery if I can as I have a UPS. Yes I know its not best practice to run a RAID card without a BBU but this is a lab and I am willing to take the risk. I am just after good write speeds at this point with my SSDs.

Thanks!
 
Why not just use the two drives individually and not use the raid card
 
BSDMan said:
Because the write perforamce with ESXi is awful without a RAID card and cache.
Click to expand...

Never been a problem for any I've set up. And it seems like your kind of running without cache now anyway...
 
Sorry I missed your earlier post somehow about getting speed before upgrading to esxi 6. Did you load the driver for the raid card manually or was it included in the esx iso? It's possible that the driver for the card is a different version, maybe see if there's an updated vib for it
 
I have installed the latest ESXi vib driver for the card and in health status its all showing "normal" in the vSphere client (the drives are showing as RAID0 and optimal).

I think the card is running in "write through" mode without the BBU so is there a way to FORCE the card to use "write back" (even without a BBU)? I'm assuming its this!

Other than that I am out of ideas...
 
Is it using the ibm vib or the lsi one? I think that's just a rebranded lsi card, maybe reflash it with their firmware and use their vib for it. Sorry just running through the gamut of stuff to try.
 
Eickst said:
Is it using the ibm vib or the lsi one? I think that's just a rebranded lsi card, maybe reflash it with their firmware and use their vib for it. Sorry just running through the gamut of stuff to try.
Click to expand...

I pretty much have given up on my RAID card so I removed it and connected the drives back to the LSI 2308 onboard controller.

I noticed that the ESXi driver being used was version 19 for the LSI-2308. After visiting:

VMware driver for LSI 2308

I noticed that version 20 was available so I downloaded it and installed it on my host. Interstingly the speeds have gone from about 20MB/s to 80MB/s! Still not fantastic considering its an all SSD setup but an improvement nonetheless.

So I'm guessing the RAID card needs a new battery/BBU for the write cache to work.

So my options at this stage are:

  1. Leave it as is and accept 80MB/s (nah!)
  2. Buy another RAID card
  3. Replace the battery
  4. Start using VSAN (no RAID card needed but you need enterprise drives that are on the HCL)
I may go option 4...still deciding. I love ESXi/VMware but I have yet to see my host perform well from a storage point of view on ESXi 6.0!
 
BSDMan said:
I pretty much have given up on my RAID card so I removed it and connected the drives back to the LSI 2308 onboard controller.

I noticed that the ESXi driver being used was version 19 for the LSI-2308. After visiting:

VMware driver for LSI 2308

I noticed that version 20 was available so I downloaded it and installed it on my host. Interstingly the speeds have gone from about 20MB/s to 80MB/s! Still not fantastic considering its an all SSD setup but an improvement nonetheless.

So I'm guessing the RAID card needs a new battery/BBU for the write cache to work.

So my options at this stage are:

  1. Leave it as is and accept 80MB/s (nah!)
  2. Buy another RAID card
  3. Replace the battery
  4. Start using VSAN (no RAID card needed but you need enterprise drives that are on the HCL)
I may go option 4...still deciding. I love ESXi/VMware but I have yet to see my host perform well from a storage point of view on ESXi 6.0!
Click to expand...

Option 5, build a battery using nimh cells and a holder. If you're comfortable doing that sort of thing, I'm assuming your card is already out of warranty.
 
Eickst said:
Option 5, build a battery using nimh cells and a holder. If you're comfortable doing that sort of thing, I'm assuming your card is already out of warranty.
Click to expand...

I've got one of these with a dead battery I set up with ESXi6 a few days ago and so far all it's done is crash the UnityVSA VM I'm trying to run on it due to latency.

I'd be curious how feasible it'd be to make a homemade battery replacement. The battery has four leads, not sure what they all are. One's probably a thermistor. Maybe I'll rip off the battey cover and see what's inside.

Haven't done a lot of work yet troubleshooting the M5015 or using it outside of Unity yet, I'll see if I can get some throughput numbers.
 
I cut open the battery pack on my M5015. There's a Sony LiIon battery that would be easily replaced for about 10 bucks, but there's also a circuit board with ten or so ICs. Might be worth trying a new battery, but I'd guess the ICs store some info on the battery state and it's not going to be as easy as just replacing the battery itself. They might determine that the battery is "expired" before it actually is, or do it based on a timer rather than condition of the actual battery.
 
You must log in or register to reply here.
Back
Top