HDD I/O Bottleneck for virtual's

[TechIsCool]

Hey everyone I have been working with such a small budget it not funny. Every year I try to upgrade our so called Hard Drive array. Right now the specs are as follows.

Distro Name - Openfiler NAS/SAN
Kernel Version - 2.6.29.6-0.24.smp.gcc3.4.x86_64 (SMP)
Processors Model - Pentium(R) Dual-Core CPU E5200 @ 2.50GHz
Memory - (2x) 512MB Dimm's
PCI Devices - (2x) Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller
- Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller
- Ethernet controller: Intel Corporation 82573V Gigabit Ethernet Controller
- RAID bus controller: Dell PowerEdge Expandable RAID controller 5
SCSI Devices - ATA ST910021AS (Direct-Access)
- DELL PERC 5/i (Direct-Access)
- DELL PERC 5/i (Direct-Access)

/dev/sda SCSI ATA ST910021AS 93.16 GB msdos 3 (view) 1x 100GB
/dev/sdb SCSI DELL PERC 5/i 2792.99 GB gpt 1 (view) 4x1TB in Raid 5
/dev/sdc SCSI DELL PERC 5/i 2792.99 GB gpt 1 (view) 4x1TB in Raid 5

I have all four network cards running in 802.3ad which seemed to help a small amount but not much. I know that my bottleneck is in the R/W too the HDD's but I would like some help from everyone on deciding a plan of attach. Do I buy 12 more spindles (consumer drives) or do I buy some SSD's (consumer grade). Right now I have 8 virtual machines hammering these raid 5's they are split equally across the sdb and sdc but I still only get about 10MB/s throughput on the virtual's.

The virtual's are running on two ESXi 4.1 devices connected two this OI device.

The other question I have is how to improve the bandwidth between the OI device and the virtual host's. I have a switch that can handle fiber at 1Gb/s or Ethernet at 1Gb/s but not a switch that can handle 10Gb. I can not financially pay for a 10Gb switch so I am not sure how to get the bandwidth between the two virtual hosts and the one NAS/SAN.

Advice would be greatly appreciated. I can supply more information about all hardware upon request.

 

[hotcrandel]

I'd say your RAID card is more of a limitation than anything else.

The PERC 5/i is okay, pretty reliable, but not terribly fast, especially in raid5. Changing to RAID10 might net you more speed, since there is less [no] parity calculations, though at a cost of capacity.

Are you using iSCSI or NFS? I could see either capping a single gig link R or W, but not more than one.

Do you feel like your VM's aren't fast enough? What's the real problem that you're trying to address?

 

[TechIsCool]

I'd say your RAID card is more of a limitation than anything else.

The PERC 5/i is okay, pretty reliable, but not terribly fast, especially in raid5. Changing to RAID10 might net you more speed, since there is less [no] parity calculations, though at a cost of capacity.

Are you using iSCSI or NFS? I could see either capping a single gig link R or W, but not more than one.

Do you feel like your VM's aren't fast enough? What's the real problem that you're trying to address?

I have about 1,200$ that I can spend to make them go faster.

I don't really like how slow some of them respond when I/O bound I should say that I have a 4 database servers on these most of the time they have minimal load but when someone pulls a large amount against one of the databases it slow downs significantly. I would like to alleviate this issue.

System is using NFS because I had two esxi hosts connected to a single iscsi initiator and it was crashing my virtual's.I can't even cap a single 1Gbe card with the array.

 

[hotcrandel]

Have you looked at ZFS builds at all?

Using pass through with a reasonably fast CPU, and a cheap IT firmware card a ZFS setup with z-mirrors would probably have better performance than a perc card running R5. I would speculate with 7200 RPM drives you're running out of IOPS, especially with the perc slowing it down.

Are your VM's disk intensive? How big are they, how many GB?

 

[Red Falcon]

If I were you, I would upgrade your memory. 1GB isn't very much for that kind of configuration, even with light usage. Try upgrading to at least 2GB if you can, that way the swap space won't be getting hit on your drive(s).

 

[hotcrandel]

Yeah I missed that.

Openfiler calls for minimum 1GB memory and recommends 2GB for bare metal.

It would be pretty cheap to move to 2-4GB and see how performance is.

http://www.openfiler.com/products/system-requirements

Have you thought about virtualizing your Openfiler installation(s), and having it live with the same box(es) as your databases?

This would eliminate any networking bottlenecks with 10G virtual NIC's, and give you a chance do some disk i/o load balancing as you move them off this mediocre disk back-end.

If you can fit your databases on SSD's that you can afford, I think you know what to do.

 

[TechIsCool]

Have you looked at ZFS builds at all?

Using pass through with a reasonably fast CPU, and a cheap IT firmware card a ZFS setup with z-mirrors would probably have better performance than a perc card running R5. I would speculate with 7200 RPM drives you're running out of IOPS, especially with the perc slowing it down.

Are your VM's disk intensive? How big are they, how many GB?

Yes I have looked at it actually have another server running on zfs but this one was created before I knew about it. The problem is where to store all the data and downtime to change the OS over to zfs.

I would like it with a slog device but right now OI with raid is all I have.

If I were you, I would upgrade your memory. 1GB isn't very much for that kind of configuration, even with light usage. Try upgrading to at least 2GB if you can, that way the swap space won't be getting hit on your drive(s).

RAM is a very good suggestion does anyone know if OI is 32bit or 64 bit if its 64 bit will I see a benefit from maxing out the RAM?

Yeah I missed that.
Have you thought about virtualizing your Openfiler installation(s), and having it live with the same box(es) as your databases?

This would eliminate any networking bottlenecks with 10G virtual NIC's, and give you a chance do some disk i/o load balancing as you move them off this mediocre disk back-end.

If you can fit your databases on SSD's that you can afford, I think you know what to do.

The reason I have not created an all in one box is because I have two virtual hosts and HA is running on both of them for fail over. I can't find a simple solution on how to achieve the virtual networking for both devices and also still be able to back it up quickly.

SSD's are a viable option I am just curious how everyone thinks they should work the problem with running two virtual host with HA fail over is that I can't just mount the SSD's on the virtual host internally I have to run them via the network for fail over purposes.


On another similar topic would I benefit drastically by running a zfs group of M4 SSD's? something around 1TB worth of them?

 

[Red Falcon]

^ OI itself will not necessarily benefit from having more RAM as it alone only eats about 400-600MB depending on your setup, perhaps even less.

However, running that many arrays in hardware RAID + daemons + network resources + users means the more RAM the better, and 1GB is not a lot to get by with.

If I were you, at least upgrade your RAM to 2GB and if you can afford it, 4GB+ at least on a VM host machine.

 

[patrickdk]

I would attempt to hange your nics also. I have the same nics onboard my system, 82573L, connected via pcie x1. I could rarely get it to go over 60MB/sec. I installed a 82576, and now I an get a full 112MB/sec. I thought it was my zpool being slow, but it was just the nic being annoying. Also with you using 802.3ad, that doesn't help nfs, unless you have a cluster of clients, 2 will benifit some, though.

I also gained the usage of jumboframes, that was my origainal beef with the 82573, but then I found out I didn't need jumboframes to max out the card.

 

[TechIsCool]

I would attempt to hange your nics also. I have the same nics onboard my system, 82573L, connected via pcie x1. I could rarely get it to go over 60MB/sec. I installed a 82576, and now I an get a full 112MB/sec. I thought it was my zpool being slow, but it was just the nic being annoying. Also with you using 802.3ad, that doesn't help nfs, unless you have a cluster of clients, 2 will benifit some, though.

I also gained the usage of jumboframes, that was my origainal beef with the 82573, but then I found out I didn't need jumboframes to max out the card.

So your suggestion is to set up a RAM disk and see if I can max out the network card. If I can then its not that. NFS is shared between two different servers but its only two NFS Shares. Yah I wish that 802.3ad works like a real set of bonded copper...

 

[TechIsCool]

^ OI itself will not necessarily benefit from having more RAM as it alone only eats about 400-600MB depending on your setup, perhaps even less.

However, running that many arrays in hardware RAID + daemons + network resources + users means the more RAM the better, and 1GB is not a lot to get by with.

If I were you, at least upgrade your RAM to 2GB and if you can afford it, 4GB+ at least on a VM host machine.

This is for Both of them

VMware HA: On
VMware EVC Mode Intel Xeon 45nm Core 2
Total CPU Resources 28GHz
Total Memory 63.98GB
Number of Hosts 2
Total Proccessors 12
Virtual Machines 11
Current Failover Capactiy 1 host

 

[TechIsCool]

Bump