DS3400 SAN settings in VMware

[geriksen]

Was following a earlier thread this year (http://www.hardforum.com/showthread.php?t=1401841), and was hoping someone could clear some things up regarding best performance tuning for DS3400 with VMware. We have the following:

2 pcs. IBM DS3400 w/dual controller and 512Mb cache. 300Gb 15K SAS spindles and 450Gb 15K SAS spindles connected to IBM Total Storage SAN16B-2 switch and 4 ESX hosts with each Qlogic 4Gb HBA single channel. Are running ESX 3.5 U4 today, but are soon to migrate to clean new vSphere 4 setup.

Here are a couple of questions:

1. Best spindle setup, RAID5 450Gb 15K (4+1) and RAID 10 300Gb 15K (2+2) leaving two 300Gb HotSpares and one 450Gb Hotspare. Configured as 2 arrays with a LUN each???

2. Best setup for an RAID 5 Array means 4+1 as one big LUN, or use a couple of LUNs inside that array?

3. Best speed with 1 HBA or Dual HBA towards the DS3400 on each ESX host?

4. Qlogic HBA Que Depth Length... any tuning to do here?

5. Also have a physical VCBProxy Server with Veeam Backup with Qlogic Dual HBA, one channel connected to IBM SAN switch and the other directly connected to Backup IBM DS3400 SAN controller. We are getting slow performance with Veeam Backup here. Anyone have a best practice setup for simular environment?

 

[sabregen]

I am going to break this up into parts:

Was following a earlier thread this year (http://www.hardforum.com/showthread.php?t=1401841), and was hoping someone could clear some things up regarding best performance tuning for DS3400 with VMware. We have the following:

2 pcs. IBM DS3400 w/dual controller and 512Mb cache. 300Gb 15K SAS spindles and 450Gb 15K SAS spindles connected to IBM Total Storage SAN16B-2 switch and 4 ESX hosts with each Qlogic 4Gb HBA single channel. Are running ESX 3.5 U4 today, but are soon to migrate to clean new vSphere 4 setup.

Here are a couple of questions:

1. Best spindle setup, RAID5 450Gb 15K (4+1) and RAID 10 300Gb 15K (2+2) leaving two 300Gb HotSpares and one 450Gb Hotspare. Configured as 2 arrays with a LUN each???

How many of each size do you have installed? You can have more than one LUN per array. More commonly, your configuration looks like this:

Array 1 > Volume 1 > mapped as LUN0 to ESX1 only (ESX1 Boot LUN)
Array 1 > Volume 2 > mapped as LUN1 to ESX Host Group (VMFS3 LUN)
Array 1 > Volume 3 > mapped as LUN0 to ESX2 only (ESX2 Boot LUN)
Array 1 > Volume 4 > mapped as LUN0 to ESX3 only (ESX3 Boot LUN)
Array 1 > Volume 5 > mapped as LUN0 to ESX4 only (ESX4 Boot LUN)
...etc...

You usually don't want to create an unnecessarily large number of arrays, in Storage Manager. Best practices recommend that you create an array per Host type. If you have a bunch of ESX hosts that you want to have one LUN, or many LUNs, you'd just make one array. Inside of the array, you can create multiple LUNs. Say, one for each ESX host to boot from, and several VMFS3 stores. Boot LUNS are mapped only to the host that they are for (keep your naming conventions standardized, it helps a lot), but the VMFS3 LUNs get mapped to the Host Group in the mappings view.

The number of LUNs (or volumes) per array is unlimited by IBM (or LSI, who makes the array), but obviously, you're going to have issues if you make a 3drive RAID-5 and start beating the crap out of it. Make sure you have enough spindles to overcome contention issues. Also, FYI, each array uses a partition license. I believe the DS3400 currently comes with 4, for free. Use them wisely... they are not terribly cheap.

2. Best setup for an RAID 5 Array means 4+1 as one big LUN, or use a couple of LUNs inside that array?

That depends on the purpose of that particular LUN, really. The only real-world usage (that I have seen) for 1-to-1 ratios on array to LUN are for high performance VMs, and usually ones that you're going to run high transaction database-like applications on. Furthermore, these applications are usually (when run in a VM) also mapped directly to the disk via RDM, for performance concerns. the 1-to-1 ratio, in addition to RDM access from the VM also requires HBAs capable of NPI-v, as well as switches that support it.

3. Best speed with 1 HBA or Dual HBA towards the DS3400 on each ESX host?

To be honest, I'm not sure what the question is, here, but I'm going to guess that you're asking about additional links from the hosts to the storage system, presumably for better throughput. The DS3400 is an active/passive array. Only one controller ever has control of the LUN at any given time. Since the DS3400 cannot operate in Active/Active mode, the addition of links to the array from the switch, or directly from the hosts, are only to facilitate better availability through multiple paths.

4. Qlogic HBA Que Depth Length... any tuning to do here?

No, you can leave this alone as long as you are within 500m of the storage system. This setting is for buffer credits, and should only be configured for distances over 500m.

5. Also have a physical VCBProxy Server with Veeam Backup with Qlogic Dual HBA, one channel connected to IBM SAN switch and the other directly connected to Backup IBM DS3400 SAN controller. We are getting slow performance with Veeam Backup here. Anyone have a best practice setup for simular environment?

I would suggest that if you have open ports on your switch, that you attach all of your VCB backup proxy machine FC ports to the switch, and invest some time in proper zoning, for starters. You may have a zoning problem on your switch, if you are both connected to the switch, and to the DS3400, directly. I would suspect that your throughput issues could be because the VCB backup proxy can see multiple valid paths to the DS3400, and is flop-flopping in between them. This could be causing LUN thrashing on the DS3400, as well, as the VCB backup kicks off, the non-preferred path could be activated by the proxy server attempting to access a given LUN from what should be the failover path, forcing a controller bounce from the LUN being accessed. I could go on and on with all of the potentials on this configuration, but I won't You have a switch, please save yourself the headache and use it.

I hope I don't come off sounding like a dick, it's not my intent. You paid good money for this gear, and I can see design flaws in the design, and likely ones that will seriously impact your operation and performance in the environment. I suggest that you either commit some time to learning about the hardware and proper implementation (LSI has great classes on DS3k/4k/5k gear in Wichita...don't go to the ones in California... talk to your IBM business partner for more information, or PM me and I can get you some information to point you in the right direction). It will save you a lot of time down the road. You bought good gear, time to put it to good use.

 

[SpaceHonkey]

1. Best spindle setup, RAID5 450Gb 15K (4+1) and RAID 10 300Gb 15K (2+2) leaving two 300Gb HotSpares and one 450Gb Hotspare. Configured as 2 arrays with a LUN each???

2. Best setup for an RAID 5 Array means 4+1 as one big LUN, or use a couple of LUNs inside that array?

3. Best speed with 1 HBA or Dual HBA towards the DS3400 on each ESX host?

4. Qlogic HBA Que Depth Length... any tuning to do here?

5. Also have a physical VCBProxy Server with Veeam Backup with Qlogic Dual HBA, one channel connected to IBM SAN switch and the other directly connected to Backup IBM DS3400 SAN controller. We are getting slow performance with Veeam Backup here. Anyone have a best practice setup for simular environment?

I'll ad my two cents here as I think I'm the one that opened the original DS3400 can-o-worms

1. In addition to the partition limitations (I thought it was per host group - not per array), this will primarily be a performance thing. For RAID5, stick to an odd number of drives. RAID10 is ok if you absolutely have to have it, but you are wasting a hot spare because of this. Just depends on the performance you need.

2. Use multiple LUNs as necessary. I wouldn't use one big LUN as every VM will be in the same contention domain, but at the same time you don't need one lun per VM. Also - budget for 5% free space. That is mandatory on any LUN - do not let your VMDKs exceed 95% capactity of the LUN! I've tested it, it's bad (loss of the datastore in my case). Also choose the appropriate block size for the datastore. That means remember the file size limitations of the VMDK - 1MB = 256GB max. If you need a 500GB disk on a datastore that is using 1MB block size, you are SOL. The penalty of using a larger block size does not matter, so use the size that is appropriate.

3. If you only have one switch, I would think one HBA is enough - but I'll defer this to the SAN experts.

4. no - watch it using esxtop. You'll probably never see it queue much.

5. Like sabregen said - don't do that! Add the VCB to the zone for your hosts, disconnect the direct link to the SP.

The most important thing that came out the perfomance thread I created is to align your partitions (in the VMs). Also, since it sounds like everything you have is redunant except your switch, I'd get another if you can. This will also allow you to use round robin in vSphere...

 

[sabregen]

balls. you're right. it is per host group (partitions, that is), not per array. sorry about that. I would also second the recommendation for the additional switch. to prevent any availability issues on the host side, though, you will need to add an additional single port HBA to each host, as well. current your single points of failure are the single HBAs and the single switch.

 

[geriksen]

Thanks guys for excellent input and thoughts!

But, I'm still confused.. or exceptional slow... ;-) Here are some more input and questions:

1. Maybe I was not clear enough... We have 2 DS3400 w/dual controllers, that I want to have exact same setup regarding spindles, because of load and vMotion/vStorage with vSphere som I easily can do maintenance on one of the SANs without downtime. What I'm wondering about is if there are any penalty with setting up RAID5 (4+1) and RAID10 (2+2) on the same box? Or should I use 2 RAID5 arrays? Also I think that using multiple LUNs on the same Array would give some decrease in performance? And finally, NO we don't use any LUNs for BOOT. We have fast RAID1 controller with SAS locally in each ESX, so I think thats easier to maintain...

SpaceHonkey; I got the part with sticking to odd numbers for RAID5 and even numbers for RAID10! ;-)

2. To clearify; I want to have multiple VMFS3 stores for VMs, mostly Windows 2003/2008 for DC/FILE/PRINT/APP and CITRIX services... We are a small Service Provider, that runs every customer isolated through VLAN and their own Domain and VMs...

SpaceHonkey; Thanks for the free space tip! ;-) I guess you mean vDisk file size limit, and not the VMFS3 store limit?

3. I know about the single point of failiure for the switch and HBA, but that's on the budget for next year... ;-)

Do any of you know how the VMs will perform if I split some vDisk on SAN1 and some on SAN2, all connected to same switch. Would that be a good ide or should I keep it to same SAN, but different LUNs? Have read that more LUNs would perform better against same SAN in VMworld...

Lastly; I really appreciate all the input guys! Since we are migrating to a new setup, I want the design to be flawless... some hope you guys can bare with me, so we can work this out! ;-)

 

[geriksen]

SpaceHonkey; You said "align your partitions (in the VMs)" is important, but what's the easiest way to do this for Windows Servers? Do this also apply to the System vDisk? Read some place that this wasn't necessary on the system drive, but that's sounds wrong... I usually create 3 vDisk for Windows Server VMs, 1 for System/OS, 1 for Data/Apps and 1 for SWAP.

Further you are talking about "Also choose the appropriate block size for the datastore."... I thought 1MB would be OK? This also raise the questions regarding formatted cluster size on LUN, vDisk (should be equal as far as I know) and VMFS block size. What recommended here? The same as in your initial tread on DS3400? Never found anything about the Read settings on DS3400 Logical drive setting. Also the Write mirror setting have a huge penalty, so how do you approach this?

I know... to many questions, but want to understand what's going on, as well as getting the most out of my hardware...

 

[sabregen]

oh, you have two DS3400's! That makes the 2pcs. part make a lot more sense now (I was confused). I see what you're trying to do now, but given that it doesn't sound like you're beating your controllers too much, I'd also most tempted to recommend that you license ERM on the DS3400s and SRM in VMware for site failover/recovery, but on the DS3400 you have to manage this feature through the CLI (basically, you can't screw with it through the GUI... believe me, I beat the LSI engineers on this point. they never gave me a good reason for their implementation, and requirement on the DS3k series to do ERM through the CLI).

1.) there's no RAID controller penalty (for throughput or I/O capabilities) to run multiple RAID types on the same box with the DS3k line. They are using IBM PowerPC processors in the controllers, and this storage line has been around for years, and is a proven industry leader in performance, and a solid platform to deploy on. That being said, the number of VMs that you run on a given number of spindles can impact performance, and it also depends on your usage needs. You wouldn't want to run 20 Windows Server VMs on a 3disk RAID-5 and then run WSUS update on all of them at the same time, so there is some sizing to consider. On your 15k disks, I'd say max 4VMs per spindle as a rule of thumb, and 3 or 4 for your 10k disks, multiplied by the number of usable disks in the array. As follows:

5disk RAID-5 gives you 4 usable disks worth of space. 15k disks x 4 disks = max 16VMs (or thereabouts)

2.) regarding SpaceHonkey's best practices note - you never want the usage level of your VMFS3 volumes to encroach on the 5% overhead limit. Example: 1TB VMFS3 volume and current files int he datastore = 951GB = potential loss of the VMFS3 volume and all data in it. Keep 10% overhead on the VMFS3 volumes to be safe (in other words, don't fill them all the way up).

3.) I think you are mixing and matching terminology here in your question, but I am sure that you know the difference between your SAN and a LUN on the SAN, so I'll skip the explanation on the difference. You may see some better disk throughput and I/O if you are splitting the workload between two independant boxes. It only makes sense. However, as previously stated, these boxes are very robust, and should have no issues in handling your throughput or I/O requirements, given your stated usage. That being said, I think that it is likely that you will hit disk contention issues on the LUN before you hit any issues on the controller. These boxes are designed for a maximum of 48 disks on each controller (the DS3400 itself, and 3x EXPs attached with 12 drives/enclosure). I am almost tempted to suggest that if you are concerned about LUN contention of anything else that you populate more disks in one chassis and expand the array and then add more LUNs, or to create a new array from the newly added disk...but then we get into the DACSTORE issues, and controller personalities...it could get ugly. Spread your workload if you want. Givent he number of spindles that you say you have in each box, I am sure you will get to LUN contention before ever saturating the controllers or the FC link.

regarding VMFS3 block sizes - just make sure that you choose the block size that is appropriate when you create the VMFS3 volumes. there's sizing guidelines, and radio buttons to click on. As long as you clicked the one that is closest to your VMFS3 volume, you should be okay. the block size depends on your VMFS3 size. IIRC, the guidelines are as follows:

256GB -1MB blocks
512GB - 2MB blocks
1024GB+ - 4MB blocks
(I think - someone else to confirm)

And yes, align your vm partitions.

 

[lopoetve]

2048=8mb blocks.

 

[lopoetve]

SpaceHonkey; You said "align your partitions (in the VMs)" is important, but what's the easiest way to do this for Windows Servers? Do this also apply to the System vDisk? Read some place that this wasn't necessary on the system drive, but that's sounds wrong... I usually create 3 vDisk for Windows Server VMs, 1 for System/OS, 1 for Data/Apps and 1 for SWAP.

Further you are talking about "Also choose the appropriate block size for the datastore."... I thought 1MB would be OK? This also raise the questions regarding formatted cluster size on LUN, vDisk (should be equal as far as I know) and VMFS block size. What recommended here? The same as in your initial tread on DS3400? Never found anything about the Read settings on DS3400 Logical drive setting. Also the Write mirror setting have a huge penalty, so how do you approach this?

I know... to many questions, but want to understand what's going on, as well as getting the most out of my hardware...

Don't do it to the system disk - there have been concerns for VSS functionality for some reason with it, but I don't know if they still apply. Performance concerns there are minimal. Do it to a data disk for sure

 

[sabregen]

there ya go!

 

[geriksen]

Thanks, guys!

Seems like a good role of thumb regarding 4 VMs pr spindle, but today I have a RAID5 (6+1) + Hotspare on 1 LUN with 43 VMs spread out on 3 ESX hosts... and seems like it's no problem, but maybe not optimal. But, Sabregen, could it be abit low number pr spindle? Do you know why many ppl suggest creating a couple of smaller LUNs on an Array instead of 1 big in VMware cases?

Also agree on not using alignment on the system disk. If we make a template, do you know if the vDisk needs to be realign afterwards? Think I've read something about that being a necessary every time you create a new VM, even if the Template's vDisks are aligned...

What settings are you using / recommending on the DS3400 for READS and WRITE MIRROR on the Arrays?

Also what do you recommend for vDisk formated cluster size or what's it called for the system, data and SWAP vDisks...

 

[lopoetve]

it's all about spindle contention. If it performs, there's no concern (other than the fact that you're concentrating stuff on a single device, with the limitations of hardware failure/etc hitting a larger target instead of a smaller one). Each machine has to be aligned, template deployment or not