Need some help in relieving disk contention - Virtualized Environment in a box

sabregen

Fully [H]
Joined
Jun 6, 2005
Messages
19,501
It's been a while since I have posted much in here, but a new issue is forcing me to ask for outside opinions. As part of my job, and in keeping our company on the edge of virtualization technologies as a potential revenue stream, I have built an ESX4 environment in a box. Since it will bear significance to the potential solution, the hardware platform for this rolling demo unit is as follows:

Sager NP9262 laptop: http://www.sagernotebook.com/product_customed.php?pid=29175
Specs:
Intel Core 2 Quad Q6600 @ 3.0GHz (pinmod)
Intel 945 chipset
Intel ICH7R RAID controller (0, 1 and 5)
3x Fujitsu 5400RPM 2.5 SATA w/8MB cache
4GB PC2-6400 DDR2
17" 1920x1200
Expresscard 34/52 shot
4x USB 2.0 ports

I will leave out the rest of the specs, as they are not going to impact this current situation. The idea is that I have a rolling demo machine, complete with the following VMs for a customer loaner or demo box (for presentations). The machine runs Vista x64 Ultimate as the host OS, and is running VMWare Workstation 6.5:

ESX1 - 1vCPU, 2048MB of RAM, 10GB boot LUN (default partitions), 3 NICs
ESX2 - 1vCPU, 2048MB of RAM, 10GB boot LUN (default partitions), 3 NICs
OpenFiler iSCSI target - 1vCPU, 384MB of RAM, 2GB boot LUN, 40GB iSCSI served LUN, 2 NICS
2k3 AD PDC - 1vCPU, 384MB of RAM, 10GB LUN, 2 NICs, running DNS, RRAS (NAT between interfaces)
2k3 vCenter - 1vCPU, 384MB of RAM, 10GB boot LUN, 1 NIC, running vCenter services

All of the above are running under Vista x64 Ultimate, inside of Workstation 6.5. The disk system for the physical host machine are configured in a RAID-5 set, as the machine was previously my desktop/laptop machine for work. I have since moved off to a Mac, and now the machine is being re purposed. As you might imagine, having that many levels of disk abstraction, and that the host machine (OS and drivers) calculating RAID-5 parity for every write to disk...is killing the performance. I can get to about 3 launched VMs before the machine just goes to it's knees. RAM usage is very high (this can also be attributed to latent software installations, which have been cleaned up, as much as possible), but I think would be usable if the disk contention could be relieved.

So, here are the potential configurations for the already internal disks, and potential expansion options:

1) Set all internal disks to operate independantly. Install the host OS on the first drive, and split the remaining VM load between the two remaining drives (run a 3 disk config, no RAID). This may cause issues though, as the ICH7R is tied into the PCI-E bus on this machine, and all 3 available ports take the same path (as far as the block diagram for the chipset is concerned)

2) Try a varying array of RAID-0 configurations. 2 or 3 drive RAID-0, and the lack of calculated parity by the host OS drivers and CPU/RAM resources required to perform the calculations could lighten the overall system load. Also, this may potentially relieve the disk contention issues, as any data reads/writes will be spanned across 2 or 3 spindles. However, this does not remove the potential bottleneck of all of the disk access going through the same path, with regard to the chipset block diagram

3) Offload the VMs in a sensible layout to high speed USB 2.0 thumb drives or hard drives. Retain the host OS load as a RAID-5 set, but move some or all of the Workstation 6.5 VMs off to other data storage devices. With hard drives and thumb drives being cheap, this could help offset the current disk contention situation, with a reasonable cost incurred. The potential bottleneck here is the USB 2.0 bus, itself.

4) Offload the VMs in a sensible fashion to an ExpressCard connected (or based) storage solution. This could be any type of storage device that attaches to the ExpressCard slot. As far as the clock diagram of the internal chipsets are concerned, this method may prove the most beneficial for relieving disk contention/bus contention issues, as the ExpressCard slot is attached via it's own PCI-E lanes to the northbridge, and is not coupled to the SATA controller, at all.

5.) Replace current spinning disks with cheap SSDs. This might prove useful, as well. The IOPS are not high for the environment, when fully running, and the throughput is not either. The major issue is the heads bouncing from one sector of the disks to another, and the time that it is taking to go back and forth. With all of the running VMs paging out RAM to disk, which is actually being used by the host system, as well, it's causing a lot of read/write delays to happen, and the disk queue is getting hit hard. I wouldn't need any fast SSDs, because the latency in the reads/writes is the issue, not the throughput from the disks.

6.) Some mixture of the above.
 
I don't know if testing this on a laptop will give you the ideal solution for a test or demonstration.
 
Not ideal, but meant to be an instant "take a look at this" type of demonstration that can be brought along for demos. And by instant, I don't mean instant in the sense that the environment launches instantly, just that it can be brought out of a backpack.
 
I'm leaning towards a few SSDs personally. Using some external storage solution through the expresscard slot is going to be expensive and bulky. Why don't you also put 8gb of RAM in the thing too? Should set you back only about $300. I don't use ESX all that much though, so I'm not completely certain how effective all this would be.
 
8GB would not relieve the disk contention issue that I have. Also, I can't install 8GB of RAM inside this machine, as the BIOS can only handle 4GB.
 
I would stop using the raid 5 and use individual disks. In my experience, when running lots of VM's you get better performance with individual disks. It's not the PCI-E bus contention that is getting you it's the seek times on the disks, and you end up with long disk queues. With individual disks the VM's don't compete with each other as much. Also don't be afraid to put a VM on the host OS drive, once it's all booted up not much will be happening there (expect maybe for swap, but I always set VMware to 'fit all vm's in physical memory option.') Some SSD's would also help out, in the case of SSD's I would put them all in one big stripe -- generally a demo machine like that, at least in my experience, won't have any really critical data on it, and with SSD's you might as well gain the throughput from the stripe, since the seek penalties are nearly non-existent.
 
Yeah, I know it doesn't fix the disk issue, I just thought it might help a bit too. The Sager page said it supports 8gb, so I figured it might be worth a shot. I guess you have an older version?
 
Shit...how did I miss that? 6.8GB usable out of 8GB...might be worth it, but it would have to be in addition to the disk solution, as well. Good catch!
 
I would stop using the raid 5 and use individual disks. In my experience, when running lots of VM's you get better performance with individual disks. It's not the PCI-E bus contention that is getting you it's the seek times on the disks, and you end up with long disk queues. With individual disks the VM's don't compete with each other as much. Also don't be afraid to put a VM on the host OS drive, once it's all booted up not much will be happening there (expect maybe for swap, but I always set VMware to 'fit all vm's in physical memory option.') Some SSD's would also help out, in the case of SSD's I would put them all in one big stripe -- generally a demo machine like that, at least in my experience, won't have any really critical data on it, and with SSD's you might as well gain the throughput from the stripe, since the seek penalties are nearly non-existent.

I second this, a stripe with ssd's will be the ultimate in performance.
 
I would say get three of 30gb Vertex's(reason for the Vertexes is that they are the best SSD for the $$, ya the Intel is *better* but is outrageously more expensive, and I know this is for a business so $$ may not be that big of a factor but it will make somebody happy that you can do it for not much) ...this even if you are still on R5 will speed things up, its not the RAID thats killing you but 5400rpm drives the seek times on those is prolly in the 20's.....BUT considering that its a demo I cant imagine that any data other than the Vmdk's and some test data will be on here so Raid0 would be in order for optimal performance(that is what you want when your showing something off to potential customers). I would use the highest stripe size, i believe its 128kb on the ich8, and enable the cache.

And like someone said above pack it with 8gb. I know it says only 6.8 is usable, but according to intel the p965 chipset should support all 8gb, so maybe a driver or BIOS update will allow you to fully utilize all the RAM. Even if not 6.8 is better than 4.
 
I use Virtual Box the only way to go is to have independent disks. I have a M50VM Asus.
I run 3 VM's one is of the Esata port 2 on USB when I test. When I was doing it from my single drive the disk contention was horrible.
 
Thanks for all of the replies guys. In diagramming this all out (the block diagrams for the chipsets/port connectivity) yesterday, it was deemed likely that we could upgrade at least one of the two DIMM slots to a 4GB module. At the very least, that would give 6GB total system RAM. 8 would be prefferrable, but I'm not sure if we can swing it.

extide, you hit the nail on the head, as did nitrobass24 and Ockie. The seek times are killing me, and that the entire system is using a single larger drive, which is using host system resources to calculate parity, and increasing contention, is killing me. I will look into the SSD potential, and see what I can get approved, by the boss.

Thanks again, guys.
 
This
The disk system for the physical host machine are configured in a RAID-5 set
and this are your biggest problems. Get an SSD or two, don't do raid 5.
RAM usage is very high (this can also be attributed to latent software installations, which have been cleaned up, as much as possible), but I think would be usable if the disk contention could be relieved.
Why not do a clean install of Windows?
1) Set all internal disks to operate independantly. Install the host OS on the first drive, and split the remaining VM load between the two remaining drives (run a 3 disk config, no RAID). This may cause issues though, as the ICH7R is tied into the PCI-E bus on this machine, and all 3 available ports take the same path (as far as the block diagram for the chipset is concerned)
The contention for the PCI express bus isn't happening. You're not hitting throughput caps, but seek times.
3) Offload the VMs in a sensible layout to high speed USB 2.0 thumb drives or hard drives.
This could improve latency, true, but you'll lose any potential for throughput that you once had. PCI express is way, way faster than USB. It won't make things faster to use USB.
 
Back
Top