sabregen
Fully [H]
- Joined
- Jun 6, 2005
- Messages
- 19,501
It's been a while since I have posted much in here, but a new issue is forcing me to ask for outside opinions. As part of my job, and in keeping our company on the edge of virtualization technologies as a potential revenue stream, I have built an ESX4 environment in a box. Since it will bear significance to the potential solution, the hardware platform for this rolling demo unit is as follows:
Sager NP9262 laptop: http://www.sagernotebook.com/product_customed.php?pid=29175
Specs:
Intel Core 2 Quad Q6600 @ 3.0GHz (pinmod)
Intel 945 chipset
Intel ICH7R RAID controller (0, 1 and 5)
3x Fujitsu 5400RPM 2.5 SATA w/8MB cache
4GB PC2-6400 DDR2
17" 1920x1200
Expresscard 34/52 shot
4x USB 2.0 ports
I will leave out the rest of the specs, as they are not going to impact this current situation. The idea is that I have a rolling demo machine, complete with the following VMs for a customer loaner or demo box (for presentations). The machine runs Vista x64 Ultimate as the host OS, and is running VMWare Workstation 6.5:
ESX1 - 1vCPU, 2048MB of RAM, 10GB boot LUN (default partitions), 3 NICs
ESX2 - 1vCPU, 2048MB of RAM, 10GB boot LUN (default partitions), 3 NICs
OpenFiler iSCSI target - 1vCPU, 384MB of RAM, 2GB boot LUN, 40GB iSCSI served LUN, 2 NICS
2k3 AD PDC - 1vCPU, 384MB of RAM, 10GB LUN, 2 NICs, running DNS, RRAS (NAT between interfaces)
2k3 vCenter - 1vCPU, 384MB of RAM, 10GB boot LUN, 1 NIC, running vCenter services
All of the above are running under Vista x64 Ultimate, inside of Workstation 6.5. The disk system for the physical host machine are configured in a RAID-5 set, as the machine was previously my desktop/laptop machine for work. I have since moved off to a Mac, and now the machine is being re purposed. As you might imagine, having that many levels of disk abstraction, and that the host machine (OS and drivers) calculating RAID-5 parity for every write to disk...is killing the performance. I can get to about 3 launched VMs before the machine just goes to it's knees. RAM usage is very high (this can also be attributed to latent software installations, which have been cleaned up, as much as possible), but I think would be usable if the disk contention could be relieved.
So, here are the potential configurations for the already internal disks, and potential expansion options:
1) Set all internal disks to operate independantly. Install the host OS on the first drive, and split the remaining VM load between the two remaining drives (run a 3 disk config, no RAID). This may cause issues though, as the ICH7R is tied into the PCI-E bus on this machine, and all 3 available ports take the same path (as far as the block diagram for the chipset is concerned)
2) Try a varying array of RAID-0 configurations. 2 or 3 drive RAID-0, and the lack of calculated parity by the host OS drivers and CPU/RAM resources required to perform the calculations could lighten the overall system load. Also, this may potentially relieve the disk contention issues, as any data reads/writes will be spanned across 2 or 3 spindles. However, this does not remove the potential bottleneck of all of the disk access going through the same path, with regard to the chipset block diagram
3) Offload the VMs in a sensible layout to high speed USB 2.0 thumb drives or hard drives. Retain the host OS load as a RAID-5 set, but move some or all of the Workstation 6.5 VMs off to other data storage devices. With hard drives and thumb drives being cheap, this could help offset the current disk contention situation, with a reasonable cost incurred. The potential bottleneck here is the USB 2.0 bus, itself.
4) Offload the VMs in a sensible fashion to an ExpressCard connected (or based) storage solution. This could be any type of storage device that attaches to the ExpressCard slot. As far as the clock diagram of the internal chipsets are concerned, this method may prove the most beneficial for relieving disk contention/bus contention issues, as the ExpressCard slot is attached via it's own PCI-E lanes to the northbridge, and is not coupled to the SATA controller, at all.
5.) Replace current spinning disks with cheap SSDs. This might prove useful, as well. The IOPS are not high for the environment, when fully running, and the throughput is not either. The major issue is the heads bouncing from one sector of the disks to another, and the time that it is taking to go back and forth. With all of the running VMs paging out RAM to disk, which is actually being used by the host system, as well, it's causing a lot of read/write delays to happen, and the disk queue is getting hit hard. I wouldn't need any fast SSDs, because the latency in the reads/writes is the issue, not the throughput from the disks.
6.) Some mixture of the above.
Sager NP9262 laptop: http://www.sagernotebook.com/product_customed.php?pid=29175
Specs:
Intel Core 2 Quad Q6600 @ 3.0GHz (pinmod)
Intel 945 chipset
Intel ICH7R RAID controller (0, 1 and 5)
3x Fujitsu 5400RPM 2.5 SATA w/8MB cache
4GB PC2-6400 DDR2
17" 1920x1200
Expresscard 34/52 shot
4x USB 2.0 ports
I will leave out the rest of the specs, as they are not going to impact this current situation. The idea is that I have a rolling demo machine, complete with the following VMs for a customer loaner or demo box (for presentations). The machine runs Vista x64 Ultimate as the host OS, and is running VMWare Workstation 6.5:
ESX1 - 1vCPU, 2048MB of RAM, 10GB boot LUN (default partitions), 3 NICs
ESX2 - 1vCPU, 2048MB of RAM, 10GB boot LUN (default partitions), 3 NICs
OpenFiler iSCSI target - 1vCPU, 384MB of RAM, 2GB boot LUN, 40GB iSCSI served LUN, 2 NICS
2k3 AD PDC - 1vCPU, 384MB of RAM, 10GB LUN, 2 NICs, running DNS, RRAS (NAT between interfaces)
2k3 vCenter - 1vCPU, 384MB of RAM, 10GB boot LUN, 1 NIC, running vCenter services
All of the above are running under Vista x64 Ultimate, inside of Workstation 6.5. The disk system for the physical host machine are configured in a RAID-5 set, as the machine was previously my desktop/laptop machine for work. I have since moved off to a Mac, and now the machine is being re purposed. As you might imagine, having that many levels of disk abstraction, and that the host machine (OS and drivers) calculating RAID-5 parity for every write to disk...is killing the performance. I can get to about 3 launched VMs before the machine just goes to it's knees. RAM usage is very high (this can also be attributed to latent software installations, which have been cleaned up, as much as possible), but I think would be usable if the disk contention could be relieved.
So, here are the potential configurations for the already internal disks, and potential expansion options:
1) Set all internal disks to operate independantly. Install the host OS on the first drive, and split the remaining VM load between the two remaining drives (run a 3 disk config, no RAID). This may cause issues though, as the ICH7R is tied into the PCI-E bus on this machine, and all 3 available ports take the same path (as far as the block diagram for the chipset is concerned)
2) Try a varying array of RAID-0 configurations. 2 or 3 drive RAID-0, and the lack of calculated parity by the host OS drivers and CPU/RAM resources required to perform the calculations could lighten the overall system load. Also, this may potentially relieve the disk contention issues, as any data reads/writes will be spanned across 2 or 3 spindles. However, this does not remove the potential bottleneck of all of the disk access going through the same path, with regard to the chipset block diagram
3) Offload the VMs in a sensible layout to high speed USB 2.0 thumb drives or hard drives. Retain the host OS load as a RAID-5 set, but move some or all of the Workstation 6.5 VMs off to other data storage devices. With hard drives and thumb drives being cheap, this could help offset the current disk contention situation, with a reasonable cost incurred. The potential bottleneck here is the USB 2.0 bus, itself.
4) Offload the VMs in a sensible fashion to an ExpressCard connected (or based) storage solution. This could be any type of storage device that attaches to the ExpressCard slot. As far as the clock diagram of the internal chipsets are concerned, this method may prove the most beneficial for relieving disk contention/bus contention issues, as the ExpressCard slot is attached via it's own PCI-E lanes to the northbridge, and is not coupled to the SATA controller, at all.
5.) Replace current spinning disks with cheap SSDs. This might prove useful, as well. The IOPS are not high for the environment, when fully running, and the throughput is not either. The major issue is the heads bouncing from one sector of the disks to another, and the time that it is taking to go back and forth. With all of the running VMs paging out RAM to disk, which is actually being used by the host system, as well, it's causing a lot of read/write delays to happen, and the disk queue is getting hit hard. I wouldn't need any fast SSDs, because the latency in the reads/writes is the issue, not the throughput from the disks.
6.) Some mixture of the above.