New ESXI server - RAID 10 disk performance (benchmarks included)

T

Transition

Limp Gawd
Joined
Jan 18, 2005
Messages
181
I got my first ESXi server configured (thanks to those on here that gave suggestions) and just wanted to verify the HD performance looks ok. I'm a little concerned about how the read performance is so sporadic.

wqMFN.png


k7R26.png


eaw8A.png


Do these numbers look acceptable? It's a RAID 10 setup with a dell PERC 6i controller - 64kb stripe.
 
Last edited:
Did you format the FS with regards to the chunk/stripe size? How many disks are you in your raid10 ?
 
Disks are just 7200 RPM SATA drives - nothing too special. I was impressed how much the disk speed blows away my desktop Velociraptor drive. Makes that thing look slow.
 
Yeah, you're doing JUST fine then.

100 IOPS per disk, 400 iops, a bit more with cache. Nice numbers.
 
Transition said:
Using 4 disks in the RAID 10 setup. The datastore block size is 1mb.
Click to expand...

What is the allocation unit size of your guest FS? The default for NTFS is 4k. However, I've read that you'll get slightly better performance with a higher size, like 32k or even 64k. I'm sure lopoetve can confirm or deny that.

Also, if your guest OS is Windows XP/2003 or older, you should align your partitions with the VMFS datastore which will also improve performance.

http://www.vmware.com/pdf/esx3_partition_align.pdf

The alignment can't be done with diskpart on an existing partition (although tools do exist to perform this such as mbralign from Netapp). You can create a new VM, then give its hard disk to another VM. then use diskpart within the second VM to create an aligned partition, then add it back to the new VM and install your OS from there.
 
So the alignment doc you link to... not as important anymore. That was primarily designed for the Clariion series of arrays. You may see 1-2% for most configs (DS series of arrays seem to like it too), and then of course NetApp (WAFL design architecture), but most of the time it doesn't matter that much.

Fire up IOmeter - 1024 outstanding IO, 32k and 128k random mixed, and see what kind of IOPS you get :)

As for guest- definitely try it, it all depends on the disks/controller, and I'm not familiar enough with those to know for sure.
 
I just came from the Manage for Performance class a couple weeks ago and disk alignment is still being touted.
 
*snort*...

BWAHAHA

*snort*

Some arrays, sure (especially NetApp - use MBRalign!), but other than that... >_<

I haven't had a customer worry about alignment anywhere else in over a year.
 
lopoetve said:
*snort*...

BWAHAHA

*snort*

Some arrays, sure (especially NetApp - use MBRalign!), but other than that... >_<

I haven't had a customer worry about alignment anywhere else in over a year.
Click to expand...

This is just a snippet of the results of running SQLIO on an Oracle RAC array. The SANS allocated to us is using 8 300 gig FC 15K RPM disks short-stroked to half capacity in RAID 5.

The first result was with a default 63K offset and 4096 byte File Allocation Unit Size
The second result was using 1024K offset and 64K File Allocation Unit Size. This was using a 3 gig test file and the following parameters:

Snippet from the batch file I made for this particular test:

IF '%CHOICE%'=='3' SET /A TESTLENGTH=300& SET /A TESTSIZE=3000&GOTO CONFIRM

@ECHO RANDOM WRITE 8K
SQLIO -kW -s%TESTLENGTH% -frandom -o8 -b8 -LS -F"%WD%\PARAM.TXT" >> "%WD%\SQLIORESULTS%COMPUTERNAME%\RANDOMWRITE8K.TXT"

(sorry for the formatting, maybe I'll upload an image later)

RANDOM WRITE 8K RANDOM WRITE 8K
************************** **************************
IOs/sec: 1032.30 IOs/sec: 8890.70
MBs/sec: 8.06 MBs/sec: 69.45
Min_Latency(ms): 0 Min_Latency(ms): 0
Avg_Latency(ms): 30 Avg_Latency(ms): 3
Max_Latency(ms): 2404 Max_Latency(ms): 957

Simple partition alignment and 64K formatting increased I/O and throughput nearly 900%

Want to see a RAID 10 parition-aligned array using 4 WD 500 gig SATA drives vs 3 15K RPM SAS drives in RAID 5 that isn't partition-aligned and using 4096 byte FAUS?

(I know there some apples/oranges stuff in this comparison but you would think 15K SAS drives would blow away 7200 RPM SATA)

SEQUENTIAL READ 8K SEQUENTIAL READ 8K
************************** **************************
IOs/sec: 21478.00 IOs/sec: 32896.72
MBs/sec: 167.79 MBs/sec: 257.00
Min_Latency(ms): 0 Min_Latency(ms): 0
Avg_Latency(ms): 0 Avg_Latency(ms): 0
Max_Latency(ms): 131 Max_Latency(ms): 77
************************** **************************
SEQUENTIAL READ 64K SEQUENTIAL READ 64K
************************** **************************
IOs/sec: 3757.00 IOs/sec: 4137.59
MBs/sec: 234.81 MBs/sec: 258.59
Min_Latency(ms): 0 Min_Latency(ms): 0
Avg_Latency(ms): 8 Avg_Latency(ms): 7
Max_Latency(ms): 2021 Max_Latency(ms): 2478
************************** **************************
SEQUENTIAL READ 128K SEQUENTIAL READ 128K
************************** **************************
IOs/sec: 1911.56 IOs/sec: 2070.15
MBs/sec: 238.94 MBs/sec: 258.76
Min_Latency(ms): 0 Min_Latency(ms): 0
Avg_Latency(ms): 16 Avg_Latency(ms): 14
Max_Latency(ms): 2039 Max_Latency(ms): 2494
************************** **************************
SEQUENTIAL READ 256K SEQUENTIAL READ 256K
************************** **************************
IOs/sec: 947.43 IOs/sec: 1035.79
MBs/sec: 236.85 MBs/sec: 258.94
Min_Latency(ms): 0 Min_Latency(ms): 0
Avg_Latency(ms): 33 Avg_Latency(ms): 30
Max_Latency(ms): 2133 Max_Latency(ms): 2537
************************** **************************
SEQUENTIAL WRITE 8K SEQUENTIAL WRITE 8K
************************** **************************
IOs/sec: 28208.33 IOs/sec: 30405.51
MBs/sec: 220.37 MBs/sec: 237.54
Min_Latency(ms): 0 Min_Latency(ms): 0
Avg_Latency(ms): 0 Avg_Latency(ms): 0
Max_Latency(ms): 6 Max_Latency(ms): 6
************************** **************************
SEQUENTIAL WRITE 64K SEQUENTIAL WRITE 64K
************************** **************************
IOs/sec: 3528.25 IOs/sec: 3940.41
MBs/sec: 220.51 MBs/sec: 246.27
Min_Latency(ms): 0 Min_Latency(ms): 0
Avg_Latency(ms): 8 Avg_Latency(ms): 7
Max_Latency(ms): 24 Max_Latency(ms): 68
************************** **************************
SEQUENTIAL WRITE 128K SEQUENTIAL WRITE 128K
************************** **************************
IOs/sec: 1764.64 IOs/sec: 1968.44
MBs/sec: 220.58 MBs/sec: 246.05
Min_Latency(ms): 0 Min_Latency(ms): 1
Avg_Latency(ms): 17 Avg_Latency(ms): 15
Max_Latency(ms): 43 Max_Latency(ms): 62
************************** **************************
SEQUENTIAL WRITE 256K SEQUENTIAL WRITE 256K
************************** **************************
IOs/sec: 883.54 IOs/sec: 984.26
MBs/sec: 220.88 MBs/sec: 246.06
Min_Latency(ms): 1 Min_Latency(ms): 2
Avg_Latency(ms): 35 Avg_Latency(ms): 32
Max_Latency(ms): 74 Max_Latency(ms): 70
************************** **************************
RANDOM READ 8K RANDOM READ 8K
************************** **************************
IOs/sec: 1386.62 IOs/sec: 1949.84
MBs/sec: 10.83 MBs/sec: 15.23
Min_Latency(ms): 0 Min_Latency(ms): 0
Avg_Latency(ms): 22 Avg_Latency(ms): 15
Max_Latency(ms): 983 Max_Latency(ms): 2130
************************** **************************
RANDOM READ 64K RANDOM READ 64K
************************** **************************
IOs/sec: 1174.90 IOs/sec: 1055.92
MBs/sec: 73.43 MBs/sec: 65.99
Min_Latency(ms): 0 Min_Latency(ms): 0
Avg_Latency(ms): 26 Avg_Latency(ms): 29
Max_Latency(ms): 448 Max_Latency(ms): 2119
************************** **************************
RANDOM READ 128K RANDOM READ 128K
************************** **************************
IOs/sec: 640.73 IOs/sec: 696.15
MBs/sec: 80.09 MBs/sec: 87.01
Min_Latency(ms): 0 Min_Latency(ms): 0
Avg_Latency(ms): 49 Avg_Latency(ms): 45
Max_Latency(ms): 551 Max_Latency(ms): 1191
************************** **************************
RANDOM READ 256K RANDOM READ 256K
************************** **************************
IOs/sec: 457.39 IOs/sec: 329.05
MBs/sec: 114.34 MBs/sec: 82.26
Min_Latency(ms): 0 Min_Latency(ms): 0
Avg_Latency(ms): 69 Avg_Latency(ms): 96
Max_Latency(ms): 511 Max_Latency(ms): 960
************************** **************************
RANDOM WRITE 8K RANDOM WRITE 8K
************************** **************************
IOs/sec: 1140.86 IOs/sec: 656.10
MBs/sec: 8.91 MBs/sec: 5.12
Min_Latency(ms): 0 Min_Latency(ms): 0
Avg_Latency(ms): 27 Avg_Latency(ms): 48
Max_Latency(ms): 241 Max_Latency(ms): 1753
************************** **************************
RANDOM WRITE 64K RANDOM WRITE 64K
************************** **************************
IOs/sec: 978.47 IOs/sec: 372.36
MBs/sec: 61.15 MBs/sec: 23.27
Min_Latency(ms): 0 Min_Latency(ms): 0
Avg_Latency(ms): 32 Avg_Latency(ms): 85
Max_Latency(ms): 278 Max_Latency(ms): 2241
************************** **************************
RANDOM WRITE 128K RANDOM WRITE 128K
************************** **************************
IOs/sec: 488.13 IOs/sec: 240.92
MBs/sec: 61.01 MBs/sec: 30.11
Min_Latency(ms): 0 Min_Latency(ms): 1
Avg_Latency(ms): 65 Avg_Latency(ms): 132
Max_Latency(ms): 285 Max_Latency(ms): 3498
************************** **************************
RANDOM WRITE 256K RANDOM WRITE 256K
************************** **************************
IOs/sec: 340.93 IOs/sec: 173.26
MBs/sec: 85.23 MBs/sec: 43.31
Min_Latency(ms): 2 Min_Latency(ms): 2
Avg_Latency(ms): 93 Avg_Latency(ms): 183
Max_Latency(ms): 297 Max_Latency(ms): 2997
************************** **************************
 
Did you do any with just the 64k formatting? I expect that's most of where your difference came from, given what we work with. And what's the actual array behind it?
 
The Oracle RAC is a RAID 5, but I've done tests with just 64K formatting and not partition-aligning on our own low-end servers with the following results (I'm concentrating on Random Write 8K for SQL .mdf performance results)

3x7200RPM 500 gig SATA in RAID5 not aligned, 4096 FAUS

RANDOM WRITE 8K
**************************
IOs/sec: 215.86
MBs/sec: 1.68
Min_Latency(ms): 0
Avg_Latency(ms): 147
Max_Latency(ms): 4356
**************************

3x7200RPM 500 gig SATA in RAID5 not aligned, 64K FAUS

RANDOM WRITE 8K
**************************
IOs/sec: 217.77
MBs/sec: 1.70
Min_Latency(ms): 0
Avg_Latency(ms): 146
Max_Latency(ms): 4393
**************************

3x7200RPM 500 gig SATA in RAID5 1024K alignment, 4096 Byte FAUS

RANDOM WRITE 8K
**************************
IOs/sec: 217.18
MBs/sec: 1.69
Min_Latency(ms): 0
Avg_Latency(ms): 146
Max_Latency(ms): 4381
**************************

3x7200RPM 500 gig SATA in RAID5 1024K alignment, 64K FAUS

RANDOM WRITE 8K
**************************
IOs/sec: 231.46
MBs/sec: 1.80
Min_Latency(ms): 0
Avg_Latency(ms): 137
Max_Latency(ms): 5020
**************************

We no longer use RAID5, I'm a SAME and BAARF believer now.
 
RAID 5 on ~what~? PERC cards? The family dog? NetApp? 3 rabid monkeys planning an assault on the Eifel tower? ;)

We notice no where near that kind of difference on ESX VMs on almost all the arrays out there, and I've done a ton and a half of performance tuning.
 
It was an EMC Oracle RAC setup.............probably pretty standard stuff, but I wasn't given any more granularity than that. Storage guys were pretty hush-hush: "It's a black box.........you tell us the I/O you need an we'll produce it" to paraphrase their response.

The point is: Partition alignment is important STILL whether you're talking real storage or virtual.
 
I'd still beg to disagree from the recent testing I have, especially since RAC does things very significantly differently than a virtual setup (although closer than a single machine, by all means) of multiple hosts and VMs.

For those numbers I'd hazard at a CX series array with good front end cache, or perhaps a smaller DMX3. I will bet my salary that you will never see those levels of performance difference on ESX aligned vs. unaligned VMs. Good arrays :)

I'll ask the question around again, see what real world numbers people are seeing on 4.1 so far, but I doubt they've changed from the 2-4% that we normally see on 4.0 for most arrays (again, NetApp does things different and gains a great deal more).
 
So the biggest results I've seen were around 40% on a saturated NetApp 3000 series with NFS (150 VMs). Also the IBM DS series we helped SpaceHockey with in here, which was around 30% give or take. Most of the rest (clariion tends to gain about 4%, Symmetrix gains almost nothing). HDS gains very little, unless you've got a really goofy config (again, in my experience working with it). MSA series/MD3000 you're going to run out of IOPS at a point well before you'd gain anything usable with alignment for most things, but I haven't run a lot of tests on them. I'll inquire if we have one locally and test it out. I can also run tests on a CX3, FAS 2000, and a DMX-3 as well. :)
 
http://www.vmware.com/pdf/Perf_Best_Practices_vSphere4.0.pdf, page 26

http://www.vmware.com/pdf/esx3_partition_align.pdf

I can find no supporting documentation that states that alignment does not need to be ensured for data volumes. The fact that VSphere by default now aligns all partitiions created makes things simpler, but you still have to partition align VM's where the Guest OS does not automagically do it such as XP and Server 2003.

If your clients are streaming video or doing large block-size transfers you're not going to see much of a difference since sequential performance at those sizes isn't affected much. For MS SQL applications or others that do small flle random I/0 you will see a tremendous difference.
 
Slow response. The above benchmarks reflect 1 bad disk in that array. I've replaced the bad disk and will be re-running the tests.
 
MassiveOverkill said:
http://www.vmware.com/pdf/Perf_Best_Practices_vSphere4.0.pdf, page 26

http://www.vmware.com/pdf/esx3_partition_align.pdf

I can find no supporting documentation that states that alignment does not need to be ensured for data volumes. The fact that VSphere by default now aligns all partitiions created makes things simpler, but you still have to partition align VM's where the Guest OS does not automagically do it such as XP and Server 2003.

If your clients are streaming video or doing large block-size transfers you're not going to see much of a difference since sequential performance at those sizes isn't affected much. For MS SQL applications or others that do small flle random I/0 you will see a tremendous difference.
Click to expand...

Yes, I'm well aware of those docs - the first one is just talking about VMFS, and the second one was done by EMC for ESX 3.0 and has been considered depreciated for some time. There are an infinite number of things that you do not need to ensure - we can't document them all ;)

All VMFS partitions have been aligned since 3.0.1, if you used the GUI to create them. That bug was fixed a long long time ago, to an 8k boundary (3.0.0 had a bug where it would align to a 63 offset to start with). There hasn't been an array that required differently, or recommended differently, since that time either. All current array vendors support the 8k (we use a 128 starting offset) offset just fine. I will have that document corrected, as there are other issues than performance for an unaligned VMFS volume. :)

Please note that the 4.X document does not mention guest alignment, only VMFS, as the vmkernel datamovers have issues even internally with unaligned VMFS volumes.
 
lopoetve said:
All VMFS partitions have been aligned since 3.0.1, if you used the GUI to create them. That bug was fixed a long long time ago, to an 8k boundary (3.0.0 had a bug where it would align to a 63 offset to start with). There hasn't been an array that required differently, or recommended differently, since that time either. All current array vendors support the 8k (we use a 128 starting offset) offset just fine. I will have that document corrected, as there are other issues than performance for an unaligned VMFS volume. :)
.
Click to expand...

MassiveOverkill said:
. The fact that VSphere by default now aligns all partitions created makes things simpler, but you still have to partition align VM's where the Guest OS does not automagically do it such as XP and Server 2003.
Click to expand...

I believe we covered this so I'll agree that we agree.
 
lopoetve said:
Correct. Only NetApp has current documentation out supporting that procedure, using their MBRscan and MBRalign tools. The rest have remained effectively silent in terms of published recommendations :)
Click to expand...

You're saying Microsoft has remained silent on partition-alignment on it's OS flavors?
 
MassiveOverkill said:
http://vmware.in/pdf/ws7_performance.pdf

Page 30, 6th bullet point. Practice is recommended for Workstation and ESXi (Vsphere) guests.
Click to expand...

lopoetve said:
I know the doc well - it's only for workstation. :)
Click to expand...

Now you're just getting picky. My previous statements (and proof thereof) already established that you need to do this on VM guests in ESXi/VSphere. My statement:

"Practice is recommended for Workstation and ESXi (Vsphere) guests" was me pointing out that VMware recommends that you partition-align no matter whether you're using Workstation or Vpshere.......it wasn't to specify that that particular document referred to both as I've already made references to VSphere documentation that recommends the same practice.
 
Just because something is best practice doesn't mean you have to do it. It's recommended. My engineers align partitions on guests...but unless the customer requests it we never re-align partitions and I usually sway my customers from bothering to do that unless they really hit a problem.

I'm with lopoetve on this one.
 
NetJunkie said:
Just because something is best practice doesn't mean you have to do it. It's recommended. My engineers align partitions on guests...but unless the customer requests it we never re-align partitions and I usually sway my customers from bothering to do that unless they really hit a problem.

I'm with lopoetve on this one.
Click to expand...

If your guest is an XP operating system, then yes I would tend to agree, if your guest is Server 2003 and the MSSQL database resides on the guest VM, then you would have a performance problem.

I disagree with the blanket statement that it doesn't need to be done and implementing best practices before you deploy something is far easier than doing the "Oh shit" later on.

This is [H]ardForum is it not?
 
MassiveOverkill said:
If your guest is an XP operating system, then yes I would tend to agree, if your guest is Server 2003 and the MSSQL database resides on the guest VM, then you would have a performance problem.

I disagree with the blanket statement that it doesn't need to be done and implementing best practices before you deploy something is far easier than doing the "Oh shit" later on.

This is [H]ardForum is it not?
Click to expand...

No doubt...that's why I said we do it..we do it before we deploy and while we deploy. But I've had customers take on a massive project to realign a couple hundred VMs and I thought it was a massive waste of resources for lightly utilized VMs.
 
NetJunkie said:
No doubt...that's why I said we do it..we do it before we deploy and while we deploy. But I've had customers take on a massive project to realign a couple hundred VMs and I thought it was a massive waste of resources for lightly utilized VMs.
Click to expand...

So once again I'll agree that we agree.

Lopoetve discounts the whole premise of partition alignment, unless you're using Netapp. He then slightly backs off this statement stating it can have a noticeable impact in other apps. You say you agree with, but then state your own peeps back up what I'm saying, for properly deploying VMs PA'd.

I'm not trying to discount Lopoetve's VM knowledge. He's obviously seasoned. I'm sticking with my guns on this issue and I'll be happy to take the discussion directly to VM's own forums to discuss it. I'll gladly recant when VMware and Microsoft publish something that says mis-aligned volumes no longer affect disk performance. I'm talking about the misaligned volume itself, not the fact that all of MS latest OS offerings now properly partition-align.
 
Last edited:
MassiveOverkill said:
So once again I'll agree that we agree.

Lopoetve discounts the whole premise of partition alignment, unless you're using Netapp. He then slightly backs off this statement stating it can have a noticeable impact in other apps. You say you agree with, but then state your own peeps back up what I'm saying, for properly deploying VMs PA'd.

I'm not trying to discount Lopoetve's VM knowledge. He's obviously seasoned. I'm sticking with my guns on this issue and I'll be happy to take the discussion directly to VM's own forums to discuss it. I'll gladly recant when VMware and Microsoft publish something that says mis-aligned volumes are no longer affect disk performance. I'm talking about the misaligned volume itself, not the fact that all of MS latest OS offerings now properly partition-align.
Click to expand...

Just wanted to state that I am not an expert in any way shape or form on VMware or any storage platform.

I think what Lopoetve is trying to say is that VMware is leaving it up to the OS and storage vendors on partition alignment and that apparently all except for NetApp have been silent on this issue. Also I do believe that Lopoetve works for VMware and that he works with many storage platforms in his job so he may have a bit more insight into this than most on this forum.
 
mct said:
Just wanted to state that I am not an expert in any way shape or form on VMware or any storage platform.

I think what Lopoetve is trying to say is that VMware is leaving it up to the OS and storage vendors on partition alignment and that apparently all except for NetApp have been silent on this issue. Also I do believe that Lopoetve works for VMware and that he works with many storage platforms in his job so he may have a bit more insight into this than most on this forum.
Click to expand...

Fair-enough, but there's a good reason why there was no response to my rhetorical question:

"You're saying Microsoft has remained silent on partition-alignment on it's OS flavors?"

because this is heavily documented.
 
NetJunkie said:
No doubt...that's why I said we do it..we do it before we deploy and while we deploy. But I've had customers take on a massive project to realign a couple hundred VMs and I thought it was a massive waste of resources for lightly utilized VMs.
Click to expand...

This is my company's stance as well.

We're working with a client now on moving VMs from ESX 3 servers to vSphere and P2Ving physical servers. We told them that since there will be downtime anyway (reboots due to VMware Tools updates, updating Virtual hardware, P2Ving, etc.) we may as well use mbralign to align the guest partitions.

Unless there was a sign of degraded storage performance due to misaligned guest partitions and a noticeable benefit would be expected by doing so, I wouldn't make a project of re-aligning a slew of VMs all by itself.

We also stress any new VMs should be properly aligned.
 
Last edited:
Child of Wonder said:
This is my company's stance as well.

We're working with a client now on moving VMs from ESX 3 servers to vSphere and P2Ving physical servers. We told them that since there will be downtime anyway (reboots due to VMware Tools updates, updating Virtual hardware, P2Ving, etc.) we may as well use mbralign to align the guest partitions.

Unless there was a sign of degraded storage performance due to misaligned guest partitions and a noticeable benefit would be expected by doing so, I wouldn't make a project of re-aligning a slew of VMs all by itself.

We also stress any new VMs should be properly aligned.
Click to expand...

Exactly what we do. To me, there is no downside to doing it on new or P2V'd VMs due to downtime anyway...so why not? Even if it's 5%..over a large number of VMs it can add up.
 
MassiveOverkill said:
Now you're just getting picky. My previous statements (and proof thereof) already established that you need to do this on VM guests in ESXi/VSphere. My statement:

"Practice is recommended for Workstation and ESXi (Vsphere) guests" was me pointing out that VMware recommends that you partition-align no matter whether you're using Workstation or Vpshere.......it wasn't to specify that that particular document referred to both as I've already made references to VSphere documentation that recommends the same practice.
Click to expand...

And I'm saying VMware does not recommend that anymore.

It recommends you ask your array vendor what they recommend, and the only one making that recommendation right now is NetApp.

YGPM.
 
MassiveOverkill said:
If your guest is an XP operating system, then yes I would tend to agree, if your guest is Server 2003 and the MSSQL database resides on the guest VM, then you would have a performance problem.

I disagree with the blanket statement that it doesn't need to be done and implementing best practices before you deploy something is far easier than doing the "Oh shit" later on.

This is [H]ardForum is it not?
Click to expand...

Our recent performance testing shows that it isn't really required anymore.

You tell me what SQL test to run, and I'll run it on a variety of arrays, and send the results here, and we'll let them speak for themselves (IN a VMware VM).
 
You must log in or register to reply here.
Back
Top