Need drive setup recommendations for a VMWare box.

F

fibroptikl

Supreme [H]ardness
Joined
Mar 9, 2000
Messages
7,339
In the next couple of weeks or month, I plan to build a Dual 3.4GHz Xeon Nocona system. I was able to get 2 CPU's and Intel board for free.

I plan to use this system as a machine used to run VMWare. It'll host between 3 - 10 images, if not 15. They'll mainly be used for testing stuff out, etc. This way I can try whatever I'd like to, without having to use another box, more power, etc.

While I have most of the system figured out, I can not decide upon what drive setup to get. I was thinking of an 80GB 7200RPM 8MB SATA drive as my boot drive, and creating a RAID0 using 200GB SATA drives (or 120's for that matter).

Does anyone here have any suggestions that may guide me in the best direction so I can get the best disk performance?
 
Sata is pretty much Right Out, in Monty Python terms. You need a scsi disk, and preferrably two, for this machine. Sure, Vmware makes mostly desktop-like disk access patterns, but with even five running at the same time it looks awfully server-like. 15k is not out of the question. A single scsi disk will handle requests a heck of a lot faster than a pair of sata disks, and I wouldn't even consider raid 0 for this. It'll increase latencies, and with multiple outstanding requests more latency is a killer. Try raid 1 or JBOD (and not raid 0!) if you must use sata disks, but as far as I'm concerned it's a sub-par setup. Buy it nice or buy it twice.

 
I recomend the Maxtor Atlas series either the 10kV or the 15kII both are excellent for their class. They are a little loud and hot but the performance and price is tough to beat for that sector.
 
or you could consider using a single disk for each OS image.. or at least a couple of independent disks running only a few OS images each.
 
I've got a VM box running 18 or 19 VM's right now. If you ever plan on moving to ESX, you have to use SCSI drives.

Every single aspect of this box get's maxed at one time or another. It's truly a endless resource pit.

Ram, Ram and then some more Ram.

Processors, belive it or not depending on what the box is doing, are almost secondary.

Drives:- I've got Maxtor something 10K Scsi drives in raid 1. On bootup of 1 or two of the VM's it's maxed. Reboot a bunch of them and it's a mess, but all works in the end just takes a little longer.

Mage, no raid 5 either? JBOD huh, there must be someing faster. Sooner or later I need to address increasing storage in this box.

Ram, drives, processors (Depending on your application). ESX is nice, spendy. It's also Linux dumbed down for the most part. Only bad part I found was porting the VM .vdk over to a Win box for backup. There are a few methods, and third party backup programs.
 
Raid 5 is indeed a bad idea - anything where the data is striped across disks means that every disk has to seek every time any VM asks for a block. I think that'd add up pretty dang fast, and you'd leave your disks seeking 24/7.

I really don't know what to recommend for this, because in an administrative sense (doing backups, creating new VMs (where do I put the disk for this one?), etc) it's a lot simpler to keep all the virtual disks on the same disk or volume, but you don't want to end up with all the disks seeking.

Raid 1 still forces that on writes, but on reads you gain the advantage that the two disks can process independent reads in parallel and you don't have to seek every drive for any request.

JBOD has the capacity advantage, and it shouldn't force too many seeks on you. Depending on the filesystem, you may be able to get performance like what raid 0 theoretically does out of this - SGI's xfs, for example, is designed to treat every X-gb chunk as its own filesystem and thus you can get multi-GB/s I/O from a large enough disk system. The downside is if you're using a dumb filesystem (one that doesn't notice that spreading out the data is faster), you could end up with all the data on one disk, and it gets hammered and the other one sits idle.

Marty: JBODing another raid1 onto that one might be your best choice for expansion, if your controller will do it. Some controllers effectively do that when you have 4 disks in raid 1 - they don't mirror it 4 ways, but instead mirror pairs and then concatenate them. The idea here is maximize the number of places you can read from, but minimize the number of times you need to seek. Like I said, JBOD can be really friggin fast if you happen to be optimized for it and your problem matches the answer ;)

 
Well mage I dunno. Just spent some time over at VM forums, did not really get any place, but the only things they are talking about is either Raid 5 and how many disks to use or the SAN/NAS/LUN thingies.

SAN/NAS are generally in raid 5 I think, no talk of JBOD over there. Yea, you take a hit on seek time, but xX times the through put should make up for it.

Maybe on smaller systems, raid 1+0 or 10 might be a good choice.
 
Raid 5 is just a bad idea. They seem to be under the impression that throwing more hardware at a problem will fix the problem - and it will, don't get me wrong - but if you can fix it with less hardware, that'd be the best solution, right? You could fix your car by upgrading the engine, or you could try changing the oil... ;)

 
At the moment I have a dual Xeon 2.4 Ghz box, 4 GB of RAM, and 8 SCSI disks setup as a Vmware server with 9 VMs

Disks configured as follows:
2x18GB - Raid 1- OS and Vmware Server Software
3x73GB - Raid 5 - VM Images
3X73GB - Raid 5 - VM Images

I find the above configuration works better than a straight up 6x73GB Raid 5 since the controller can access both sets of Raid simultaneously. Boot times on the VMs from a cold startup are much faster with the load split across both arrays as well.

I haven't had the time yet, but I am considering Raid 50 or Raid 10 to see what if any impact that will have overall on the system.

Also it really depends on what role or function the VM will perform as to how you spec out your storage and speed needs. IOW, a test Exchange or SQL environment will have different layout of disks than file/print/dc VMs.

VM environments have a tendency fragment very quickly when running multiple VMs on a single disk or array, spread the VMs out, and the fragmentation lowers a bit. A decent defrag utility is definitely needed and can only take place when the VM images are not in use.
 
I run MS Virtual Server (company mandate, not mine) and at home I use an iSCSI SAN (Linux 2*PIII 800 1.5Gig ram using IET for the target software running on ATA disks) hosting 1 VM server with 3-6 VM’s, my media server and scratch space for my desktop. At work they have a FC EMC/Dell SAN hosting 3 VM Servers running ~10 VM each (some more, some less). With those configurations most throughput issues due to RAID 5 or ATA are mitigated thanks to the large disk caches.

If you can offload your disk processing to another system this is the best way to go. iSCSI costs me some thorughput, but the resulting responsiveness combined with using LVM on the Linux side to be able to expand my disks makes it all worthwhile. Otherwise SCSI would probably give you some help in this area. You will not have too many problems with ATA, but you may find yourself waiting more than you like under heavy load.

I second what other here have said, memory is #1 (get lots and leave room for expansion), disk space is #2 (architect your space so you can expand it easily) and processor speed is #3. I have thrown in an extra NIC for just VM use and I also keep my VM OS VHD files and my VM Temp/Swap VHD files on different physical drives.

If you can architect your system to keep your VM drives on raw disks then that is great, but I am not able to do that due to my work patterns (and also MS VS did not support it until R2).

I also second the defrag comments. If you keep your drive files defragmented from the host side it helps (unless you run Linux as a host using a filesystem that does not suffer from fragmenting as much)
 
With regard to fragmentation, why not allocate the files all at once? There's an option to do so, use it. It should help out quite a bit, I imagine.

Off-host disks may make sense, but I would have thought 100 mbit ethernet would be painful to run VMware over. I do admit, though, LVM makes many things easy. I'd tend toward a single linux box with LVM and VMware on the same machine, though. There's no reason for a seperate Windows host that I can see (as long as you're not stuck with MSVS...).

SJConsultant: Since moving to 2 raid 5 arrays from a single helped, why not move to six individual disks? Raid 50 or 10, I think, would bring you back to the original problem - only one array. The problem, of course, is 9 VMs and 6 disks... are some of them lower load than others?

 
If you have the space to allocate your drives all at once go for it. I have to much stuff to allocate them all at once. Doing so would probably cut down on fragmentation. You would only need to worry about it when you allocate new drive files. This is one of the recommendations I have read in the past for performance tuning.

My iSCSI SAN is running Gigabit with jumbo frames on a dedicated network. So I get about 50MegaByte on sequential reads via iSCSI. My random read/write mix is around 30MB. This will probably get better when I get better NICs. I also altered the target software to take better advantage of Linux’s write-back scheme, but that is another can of worms. The other reason I moved to a SAN was to make more efficient use of my drives. I had about 90% of my free drives in use, but only about 60% of the space was in use and the free space was not accessible by the systems that needed it , hence the SAN approach (plus I am a big geek, so it sounded like fun and it has been).
 
unhappy_mage said:
SJConsultant: Since moving to 2 raid 5 arrays from a single helped, why not move to six individual disks? Raid 50 or 10, I think, would bring you back to the original problem - only one array. The problem, of course, is 9 VMs and 6 disks... are some of them lower load than others?
Click to expand...

By using one VM per disk, it would increase the risk of losing quite a bit of time and work if the disk failed. I have often thought of using a SCSI DAS, but currently not able to budget for such a device.

The "trick" to running multiple VMs is to determine the load factor of each VM and spread them out in a "balanced" fashion, thus two high loads on two arrays and the rest split as evenly as possible.

I am also exploring the option of using *nix, ISCSI, Gigabit ethernet, and Sata Raid as a lower cost repository of VMs.
 
Well, I didn't think I would require SCSI drives for this endeavor so I thought it'd be cheaper.

I was thinking of using two Western Digital RE2 WD4000YR in a RAID0. Or just keeping them as seperate drives and putting a few per drive.

I could look at using a bunch of smaller drives (ie: 40 or 80GB) and putting one or two on a drive, but I don't consider that a viable option.
 
OP- Most anything will work, VMware products are most resilient. To get the best possible performance then begins the mess.

:)
 
You must log in or register to reply here.
Back
Top