[GUIDE] Choosing HDDs for your VM Host

G

Gigas-VII

Limp Gawd
Joined
May 2, 2005
Messages
240
This post contains general advice. There will be an exception for every piece of advice, a caveat for any recommendation. Nothing here is absolute.
Disk performance is a huge concern for hosting multiple VMs on a single system. To simplify the process of selecting disks, I will lay out a few classes of disks, and weigh their relative strengths / weaknesses.

NOTE: For the sake of clarity, I'm talking about whitebox / homebrew equipment here. Enterprise servers will include lengthy and expensive support contracts, and be installed in datacenters, where power draw, noise level, and drive replacements will be largely irrelevant. If you are considering such a server, talk to your engineers or a salesperson at a major OEM. However, small business owners and home-lab users may benefit from the information I provide below.

Classes of Disks
  1. Home-Grade Disks
  2. Workstation-Grade Disks
  3. High-Performance Disks
  4. Solid-State Disks

Home-Grade Disks
Examples: Western Digital Caviar Green, Samsung(Seagate) Spinpoint F4, Seagate Barracuda Green
RPM: 5400
Size: 2.5" (7-9mm height), 3.5"
Interface: SATA2, SATA3
Cache: 16-32MB
IOPS: Generally less than 100
Strengths:
  • Inexpensive
  • Always available locally (Walmart, Target)
  • Low power footprint (3-5W peak)
  • High capacities available
Weaknesses:
  • Low throughput
  • Low IOPS
  • High latency
  • RAID-issues (TLER)
  • Firmware-mandated automatic idle shutdown behaviors w/o override
  • Warranty period: 1 to 3 years
Usage Scenario: Storage servers: mirrored pool of a large number of cheap disks can mitigate many of the drawbacks.
Recommendation: Avoid RAID5 or RAID6. Consider RAID10, RAID50, RAID60, or ZFS. Store VHDs on a dedicated host packed full of as many disks as possible, and rely on a dedicated fast link between VM Host and storage (10gig+ ideally)
Price: $0.03 - $0.05 per GB

Workstation-Grade Disks
Examples: Western Digital Caviar Blue, Seagate Barracuda XT
RPM: 7200
Size: 3.5"
Interface: SATA3
Cache: 32-64MB
IOPS: 80-120
Strengths:
  • Balanced performance
  • Generally available (electronics stores)
  • Warranty period: 3 - 5 years
Weaknesses:
  • RAID-issues (TLER)
  • Noisier than Home-Grade drives (but not by much)
  • High power footprint (5-7W peak)
  • Higher cost for performance gain (diminishing returns on investment).
Usage Scenario: Mid-size pool of disks. Still need a high spindle count to achieve IOPS, but all-around better performance than Home-Grade disks.
Recommendation: Avoid this class of disks for VM Hosts. They have similar drawbacks to lower-end disks, and similar failure rates, but much higher costs. Instead, consider using a larger number of cheaper disks.
Price: $0.06 - $0.08 per GB

High-Performance Disks
Examples: Western Digital RAID-Edition and Velociraptor, Hitachi Deskstar 7K3000, Seagate Barracuda ES, Anything SAS2
RPM: 7200, 10000, 15000
Size: 3.5", 2.5" (12-15mm height)
Interface: SATA3, SAS, SAS2
Cache: 16-64MB
IOPS: 150-200
Strengths:
  • Very low latency (4ms)
  • RAID-compatible (TLER configurable)
  • Warranty period: 3 - 5 years
  • Fewer disks needed to achieve similar IOPS (esp. when short-stroked)
Weaknesses:
  • Very expensive per unit capacity
  • Very noisy.
  • Very high power footprint (15W peak for some models)
  • Hard to locate replacements locally (carried only at specialty stores, not always in stock)
Usage Scenario: IOPS-intensive applications (databases, high-volume webservers), severe over-provisioning of VM Hosts (please don't do this intentionally). If you need the IOPS, cost won't be the issue. SAN may be a better option.
Recommendations: Heavy use of cache functions can further increase IOPS. Consider SSDs as an alternative to HDDs. Kiss dollars goodbye. If you choose to use these drives, play it smart and order several spares.
Price: $0.12 - $0.30 per GB, but sometimes more than $1.00

Solid-State Disks (SSDs)
Examples: FusionIO ioDrive, OCZ Z-Drive and Revo Drive, Crucial M4, Samsung 830
RPM: :rolleyes:
Size: almost universally 2.5" (9mm height) for SATA/SAS type drives, but some are available as PCIe cards.
Interface: SATA2, SATA3, SAS, SAS2, PCIe
Cache: varies considerably per design. usually 64 - 512 MB
IOPS: 10,000+ (sometimes in excess of 100,000, often bottlenecked by the disk controller)
Strengths:
  • IOPS off the charts, hundreds of times more than mechanical disks
  • Often RAID-compatible.
  • High R/W throughput
  • Silent (zero vibration)
  • Power footprint can be very low (3W peak).
  • Can be used to augment a disk array via a few methods
Weaknesses:
  • Extremely expensive per unit capacity
  • SLC NAND is ridiculously expensive
  • MLC NAND is prone to failure from certain types of workloads. 10% the durability of SLC NAND, but 200% capacity.
  • TLC NAND (upcoming) is prone to failure. Not a good choice for a VM Host. 1% the durability of SLC NAND, but 300% capacity.
  • Firmware issues. Firmware updates are common, and sometimes require the disk to be completely erased.
  • Steady performance degradation through utilization, fixed only via TRIM
  • Performance is limited by SATA/SAS controller
Usage Scenarios:
  • SATA / SAS controller-based R/W-cache to improve IOPS of an array. Intel offers "Rapid Storage Technology" as part of their on-board controllers, but it only works under Windows. LSI offers a hardware-based solution.
  • Filesystem-level R/W-cache. ZFS offers the option to use an SSD in two different ways (L2ARC, ZIL). Windows has several software-based solutions to cache NTFS to an SSD.
  • Array of SSDs not for OS or storage, but to hold any high-IOPS VHDs / filesystems.
  • Spendy Option: SSDs for primary storage. As prices drop, this option becomes more viable. IOPS of a RAID10 of SSDs could saturate the controller's IOPS :cool:
Recommendations: Use only where IOPS are paramount. 2x SSDs in RAID1. Alternatively, use SSDs only as a cache.
Price: $0.65 - $2.50 per GB
 
Last edited:
IOPS are often limited by the controller rather than the disks, other than that, sweeping generalizations are sweeping. Plenty of consumer grade drives carry a 5 year warranty while there are lots of Seagate SAS drives that only come with 3 years. Modern SAS drives use very little power, as low as 5-7W, that's not a very high power footprint at all.

Honestly, some of the weaknesses you list such as "noisy" are really irrelevant in a production environment. "Hard to find in an emergency situation" is contrived at best considering that in production you have spares sitting on the shelf, you potentially have 4 hour replacement maintenance contracts, and newegg offers overnight shipping.

As a side note; folks who know what they are doing don't severely overprovision storage on VM hosts, even listing that as a usage scenario really doesn't do much for credibility of the post.
 
Thuleman said:
IOPS are often limited by the controller rather than the disks, other than that, sweeping generalizations are sweeping. Plenty of consumer grade drives carry a 5 year warranty while there are lots of Seagate SAS drives that only come with 3 years. Modern SAS drives use very little power, as low as 5-7W, that's not a very high power footprint at all.
Click to expand...
I'm not going to let you antagonize me :)
Most 5400 RPM drives intended for OEMs, which are the standard for Home-Grade machines, do not include 5 year warranties. You'll be lucky if you find more than 1 year on those drives. I've seen as short as 90 days.
5-7W is a lot higher than 3-4W, math. If you have 10 disks, that wattage gap becomes significant.

Thuleman said:
Honestly, some of the weaknesses you list such as "noisy" are really irrelevant in a production environment. "Hard to find in an emergency situation" is contrived at best considering that in production you have spares sitting on the shelf, you potentially have 4 hour replacement maintenance contracts, and newegg offers overnight shipping.
Click to expand...
Who said anything about "production environment"s? Those cases are better serviced by an OEM that provides on-site service. I'm talking about whitebox equipment here. That should be obvious.

VM hosts don't have to be run in racks, in proper datacenters. If you run one in a bedroom, media rack, library, or lab-room, then the noise level can be extremely important. Plus, the vibration and noise of 10-20 disks combined is a lot worse than just 1 or 2.
Emergency replacement means being able to have a replacement drive installed within a few hours. The fastest shipping you can find will still be longer than driving to a store and buying a comparable drive. For low-end drives, this is sufficient.

Thuleman said:
As a side note; folks who know what they are doing don't severely overprovision storage on VM hosts, even listing that as a usage scenario really doesn't do much for credibility of the post.
Click to expand...
Folks who know what they're doing don't read threads like this. Virtualization is becoming more mainstream and I'm trying to help create a more welcoming point-of-entry for newbies. Hardened enterprise engineers / admins need not apply.

Also, in some cases, for spatial or power-consumption reasons, intentional over-provisioning is necessary. It can be done right. Many webhosting companies do it, for example.
 
Either this thread is going to die a quick death, or it will be chock full of information. Either way, I'm along for the ride.
popcorn.gif
 
BigJayDogg3 said:
Either this thread is going to die a quick death, or it will be chock full of information. Either way, I'm along for the ride.
popcorn.gif
Click to expand...

I'm really just trying to be helpful for newbies. I'm looking for constructive criticism. If you have ideas, I'm all ears.
 
Performance wise for 5-10 VMs which SSD setup would be best? 1 x 256GB? 2 x 128GB in RAID0? 2 x 128GB with VMs spread between them?
 
I wouldn't bother with raid when using 2 ssd's, it's not a gaming machine, even a single ssd is going to deliver better performance than an entry level SAN. Assuming the price is similar, I'd go with 2x 128gb over 1 256. You get the benefit of a 2nd port, so you are less likely to be port limited on performance (this is probably really appropriate to the crucial m4's, where the 256 is actually a touch slower than the 128 for some reason), and if 1 dies, you only lose the vm's on that ssd. The only downside is you have to plan a little bit more on disk space usage when using 2 drives.
 
Using an SSD on WS2012 HV is amazing. You can actually dedup VHDs on WS2012 on a VM host, but you have to shut the VMs down and run the dedup manually. It's not a supported configuration but the main reasons MS says it's not supported, one being increased seek time on spinning media, which is non-existant on SSDs, and balooning, which does happen, but I can live with a bit of balooning when I can fit 400GB+ worth of VMs onto a 180GB partition with 100GB free space left. I can simultaniously boot 20 deduped VMs off a SSD faster than a single VM off spinning media, much less if you try to simultaniously boot more than one VM off a spinner.
 
Honestly a single 5400 rpm drive will be fine for 95% of the folks here running white box home VM labs. If you arent looking to run a home lab, you already are going to be looking at higher end SAN solutions. While i can appreciate the OPs intentions to deliver information to noobs, most of it is pretty obvious.
 
5400RPM drives are so horrible for everything esp any sort of VM host. The IOPS on a 5400 drive is abysmal, and anything you stick in a VM is probably some sort of server and servers are pretty much always doing something with the drive such as writing log files. One 5400RPM drive per VM might be ok, it's just awful trying to put more than one per drive.
 
4saken said:
Honestly a single 5400 rpm drive will be fine for 95% of the folks here running white box home VM labs. If you arent looking to run a home lab, you already are going to be looking at higher end SAN solutions. While i can appreciate the OPs intentions to deliver information to noobs, most of it is pretty obvious.
Click to expand...

a 5400 rpm drive runs 1 instance of windows somewhat poorly, let alone several in a virtual environment. If someone has the need for a home lab, then the advice above is horrible.
 
Regarding the noise in a production system as not an issue.... Wow. Really? Have you ever been in a data center next to some craptacular blade chassis whose fans are going nuts? It's much more pleasant to work next to equipment that isn't loud. That said, disk drives typically aren't your major source of noise. It is the fans on poorly designed hardware.... Again, if you know data centers you know hot isle cold isle.... This is what separates cheap hardware (super micro) from well... The not cheap stuff.

As for choosing disks, depends on your use, how many VMs, etc... Ive typically deployed sas-based SANs running RAID5 volumes. VMs just aren't the platform for maximum performance.
 
dave99 said:
a 5400 rpm drive runs 1 instance of windows somewhat poorly, let alone several in a virtual environment. If someone has the need for a home lab, then the advice above is horrible.
Click to expand...

I hear that. I had to spend a week (on 2 occasions now) using a training facility's VM host, where the hypervisor and all VMs lived on a single 5400 RPM disk. The performance was staggeringly bad when running just 2 VMs. When I had to run 4 at once, 2 of which were light duty and 1 was SQL server, it was basically unusable for 15-20 minutes after committing a serious operation.

HDD I/O is the biggest crippling factor in these systems, especially if you aren't (yet) trained in working with the equipment. If you want to learn, you need a lab, and picking bad components for your lab will make you miserable.

My recommendation for a cheapo VM host array, for a lab or small business, is this: 6+ 5400RPM Home-Grade disks in a RAID10. Striping across mirrored pairs of disks will keep rebuild times short (just a few hours), and have several times the IOPS of a RAID5/6.

Kelter said:
Regarding the noise in a production system as not an issue.... Wow. Really? Have you ever been in a data center next to some craptacular blade chassis whose fans are going nuts? It's much more pleasant to work next to equipment that isn't loud. That said, disk drives typically aren't your major source of noise. It is the fans on poorly designed hardware.... Again, if you know data centers you know hot isle cold isle.... This is what separates cheap hardware (super micro) from well... The not cheap stuff.
Click to expand...
They aren't the main source of noise in a mostly-modern rack environment with tons of 1U and 2U boxes. Those centrifugal fans are loud as hell. However, in 3U and 4U environments, you can actually have 16-36 disks per server, in which case the noise and vibration can be considerable. I can usually hear the sound of 16+ 5400 RPM HDDs over any fans at less than 4-5kRPM.

Kelter said:
As for choosing disks, depends on your use, how many VMs, etc... Ive typically deployed sas-based SANs running RAID5 volumes. VMs just aren't the platform for maximum performance.
Click to expand...
Agreed on the last part. VMs are for maximizing utilization of hardware, decreasing square-footage of the stack, and compartmentalization of roles. If you want performance, look into HPC solutions rather than VM solutions.
 
Last edited:
No way I'd buy 6 or more spinning disks for a home lab now. Buy a couple SSDs since they are so cheap. You'll get 30x the performance and you don't use as much disk space as you'd think..just thin provision.
 
Hmmm... I have a bunch of 4u high capacity San shelves... Hold like 70 3 1/2" disks in 4u. While they are noticeable, worse has always been inefficient servers with their fans at 100% because they aren't set to pull cold from the cold isle and exhaust to the hot. One of our colos has gone through a power savings retrofit and installed wind curtains in every customers cages... Equipment that it just not up to par goes nuts.

I guess I'm just kind of used to the sounds created by storage arrays. But at least you can still have a conversation around them. I guess before I saw a super micro blade server in action, they were the loudest things... But those things make storage arrays look silent. I use hitachi and hp blade chassis.... Night and day.
 
NetJunkie said:
No way I'd buy 6 or more spinning disks for a home lab now. Buy a couple SSDs since they are so cheap. You'll get 30x the performance and you don't use as much disk space as you'd think..just thin provision.
Click to expand...
I think I agree with this. However, I imagine that running a database that performs tons of transactions per minute on an SSD would murder that SSD pretty quickly, unlike the disks. Then again, the disks wouldn't have a snowball's chance in hell at keeping up with the SSD. Still, 300 IOPs is sufficient for running a small pool of VMs where no two of them are performing disk-heavy tasks concurrently, so 30,000 IOPs would be overkill by 100x.

If you can, keep your storage on a proper storage array (NAS / SAN) rather than on your VM-host. That way you can get away with only running 1-2 SSDs. 128-256GB would probably suffice for running 6-10 VMs if at least half are non-windows environments.

Kelter said:
Hmmm... I have a bunch of 4u high capacity San shelves... Hold like 70 3 1/2" disks in 4u. While they are noticeable, worse has always been inefficient servers with their fans at 100% because they aren't set to pull cold from the cold isle and exhaust to the hot. One of our colos has gone through a power savings retrofit and installed wind curtains in every customers cages... Equipment that it just not up to par goes nuts.

I guess I'm just kind of used to the sounds created by storage arrays. But at least you can still have a conversation around them. I guess before I saw a super micro blade server in action, they were the loudest things... But those things make storage arrays look silent. I use hitachi and hp blade chassis.... Night and day.
Click to expand...
Blade boxes are crazy. They pack tons of tiny fans (sometimes redundant fan configs even, for extra turbulence) into those little blades, and then usually a couple big, extremely high RPM exhaust fans out the back. If you want lots of CPUs per U, then blades rule. If you want lots of RAM per CPU, then blades suck. If you want lots of RAM and lots of CPU, then we're talking VM hosts.

Then again, with a blade box, you can host a small handful of VMs on each blade and distribute the VMs across more blades. This makes for a better environment for live-migration to distribute VMs according to current utilization or to work around a hardware issue. It's a very economical way to improve uptime, but that only matters where uptime is important (financial institutions, large e-tailers, media networks I'd guess).

I normally just run a single beefy box (8-16 cores) and toss VMs on it. This usually allows you to achieve more DIMMs per core (usually 8-16 DIMM slots per CPU socket), but fewer cores per U (max 4 CPUs in 1U @ 8-16 cores per CPU, which is incredibly cramped at best. 2U would make thermals a lot more manageable), and probably fewer DIMMs per U as well. My environment isn't U-scarce, though, nor would be a home/small-business lab. Uptime also isn't really an issue in smaller-scale environments. 20 minutes downtime means very little to a small shop.

I'm old-fashioned; I really prefer working with 3U and 4U boxes to 1U or 2U. I hate dealing with higher RPM fans, ducts, risers, and bizarrely tight clearances for components. Then again, density is the name of the game in a colo. I just don't have to colo anything, we have our own datacenters :cool:
 
Last edited:
Are we talking production or a lab? Very different. Anyone doing prod on SSDs should be using "Enterprise" SSDs that are spec'd to last as long as spinning disks, even on I/O heavy work. Most database I/O is read, BTW, except for the log drives which don't need to be on SSD.

As for blades..it's not 10 years ago. I can give you 512GB of RAM and 20 physical cores on a single half-width UCS blades. Other vendors are similar. That exceeds 99% of what most vSphere hosts have. If you want more I can do better than that on a full width blade.
 
NetJunkie said:
Are we talking production or a lab? Very different. Anyone doing prod on SSDs should be using "Enterprise" SSDs that are spec'd to last as long as spinning disks, even on I/O heavy work. Most database I/O is read, BTW, except for the log drives which don't need to be on SSD.
Click to expand...
If by most, we mean the majority, then sure. There are applications which are going to write to the database probably 30-45% as much as read, though, and that's a considerable amount of writing. Agreed, logging is probably the highest transaction count, which really shouldn't be on an SSD. If we remove logging and consider 9R:1W, then yeah, you're absolutely right.

NetJunkie said:
As for blades..it's not 10 years ago. I can give you 512GB of RAM and 20 physical cores on a single half-width UCS blades. Other vendors are similar. That exceeds 99% of what most vSphere hosts have. If you want more I can do better than that on a full width blade.
Click to expand...
I'm not familiar with most blade layouts, but UCS are 1U in height and 0.5U in width... oh that's what you meant. How many DIMMs are we talking about? Also, 20 cores is a really strange number. Do they have blades with odd numbers of CPU sockets or something? What does such a blade setup cost? What's the point of having a blade setup if the modules are each 1U full-width? Wouldn't you be better off just using a pile of 1U servers so they're all fully independent?

Just to recap: the purpose I had in mind for this thread was to discuss and recommend disk configurations for lab / small-business grade VM Hosts. I want to help people get hands-on with virtualization by removing some challenges or points of frustration from the equation. Enterprise-grade blade solutions, where the expected load is 100+ VMs and the system cost exceeds that of a family sedan, are going to play by entirely different rules, for good reason.
 
The UCS 5108 blade chassis holds 8 blades in 6U of rack space so less than 1U per compute node. The 20-core blades are not an odd config..they are dual-socket E7 CPUs each doing 10 physical cores. Those blades (B230M3) have 32 DIMM slots. You'll be seeing 1TB of RAM in half-width blades before too long. The benefits over standalone 1U boxes are: Far better I/O capability, single point of management, easy server migration to new hardware with UCS Service Profiles, faster expansion. Not all blade configs (most don't) have these benefits but that's why I don't sell those others. ;)

Unless you need a lot of space or have a very tight budget no reason not to go SSD these days in lab or SOHO hosts. Buy.com has 128GB Crucial M4s for $79 right now. I've seen 256GB drives for under $200 lately. If you do need more space split it... That's what I do in my big Synology DS3611xs. 4 x SSD and 8 x SATA 7200RPM.

Disk performance is no longer a serious problem thanks to SSDs.
 
NetJunkie said:
Unless you need a lot of space or have a very tight budget no reason not to go SSD these days in lab or SOHO hosts. Buy.com has 128GB Crucial M4s for $79 right now. I've seen 256GB drives for under $200 lately. If you do need more space split it... That's what I do in my big Synology DS3611xs. 4 x SSD and 8 x SATA 7200RPM.

Disk performance is no longer a serious problem thanks to SSDs.
Click to expand...
$200-ish per 256GB SSD, and you want a pair of them for RAID1 for fault-tolerance, so that's $400. Would you be better served with 4x 2TB disks in a RAID10 for the same $400? That depends on your workload, but I tend to agree with NetJunkie on this one: 256GB of storage for VHDs is probably sufficient if you can move your mass-storage to a different resource.

However, if it isn't enough capacity, or not enough fault-tolerance (maybe you need 3-way mirrors?), or just costs too much, then disks become better options. I don't like the idea of using a pile of 64GB SSDs, personally (maybe, if they were really incredibly cheap). I worry about their durability.
 
Last edited:
Lol.. funny thing is, I think the folks left here all have a fairly deep understanding of VMware deployments.

Yep, as netjunkie said, you can get quite a bit of memory in blades... but, the real restraint comes to VMware licensing. With ESX5, it is licensed by memory footprint and not core count. 96GB per license, so this is where we typically bring our hosts up to. We are using 4 racks of EMC Clarion's for our backend storage with the UCS blades for the hosts and Nexus switching.. including the Nexus 1000 virtual switching for VMware.

Regarding SSDs for DB logs though.. I suppose for MSSQL the quorum doesn't need much performance, but on Oracle Enterprise (RAC to a much lesser extent), the archive logs don't require much performance, but the redo logs are the highest IOPS usage of the system. One of the main reasons we went SSDs was to handle the IOPS requirements. The drives are about $6k ea for 200GB SSDs, and I kind of went overboard with a RAID10 on these.. but.. when you are throwing down $500k for Oracle licensing, what is another 12k? We were pushing about 4000 IOPS for just the redo logs. The equivalent of that for traditional HDDs would be about 10 4-disk RAID 10 volumes... 40 disks. (We stuck with RAID10 since we were using ASM). That's almost a full shelf of storage.. or as I like to guestimate.. another 80k.. vs 24k of the SSDs which nets you about 25k IOPS real world usable.
 
For HOME USAGE ONLY I don't mind not having any sort of redundancy on my SSDs, all my VMs get backed up frequently to raided HDDs so all I'd have to do to recover is restore from them, and SSDs seem a bit more reliable than HDDs as far as random failures go (as long as you avoid OCZ like the plague), Intel and other high quality drives seem to rarely fail as long as they're not DoA.

And yeah 30k IOPS may be super overkill but the problem is the gap between spinners and SSD IOPS is so wide that what's super overkill on the SSD side may be "requires half a dozen spinners to be sufficient" on the HDD side.
 
I think that the fundamental problem with this guide is that it is attempting to generalize and the reality of it is that there simply is no generalization possible.

Even for "home use" people have very very different requirements. I run up to 8 VMs off of a single SSD at home and it works just fine. Similarly it would also work fine if I were to run it off of a single 5,400 RPM drive because my VMs do very little I/O period.

The guide is too full of assumptions, some of which are simply incorrect, to be actually useful to anyone from where I am looking. I also continue to take exception to 3-5W being a low power footprint whereas 5-7W is a high footprint. In home use this doesn't matter at all, so what if you have 100 disks and you burn 300W more during peak load. You drives in a home use scenario will be idle 99% of the time anyway. In a business environment most premises you seem to base this guide off of simply do not apply at all.

From where I am looking most of the benefits/weaknesses are contrived at best. I googled for it and was not able to find a new 5,400 drive that came with anything other than 3 year warranty. I spend about 5 minutes looking. The 90 day warranties are for refurbished drives.

Still, the general issue is that you just can't have a guide that uses a "one size fits all" approach. I would bet good money on that most home users will be perfectly fine with slow, cheap, high latency (all the way up to 8ms, omgwtfbbq!) drives for their home labs.

The need for IOPS is often very overestimated. I always like to point to this study our VAR did which logged the IOPS for an entire school district at ~200 because most of what they do is open excel files. There is absolutely no point in having a 30k IOPS drive if you only need 200, or even 2000.

Again, in any kind of raid setup your IOPS bottleneck is often the controller, not the individual disk.

Lastly, people who deploy enterprise grade systems will (hopefully) not use the Internet at large to design their system but will have actual professionals do it for them.

I commend you on spending your time to try and help folks out, it's just that this particular way just isn't all that useful. Not for the lack of trying but because literally every case is different and there are as many cases as they are users. This large amount of variation just can't be captured in a single short guide.
 
Thuleman, while I tend to agree with you on this, I've never seen a decent controller with a BBWC be an IOPS bottleneck - it's always been the disks. A plain, local controller - sure; is that what you're talking about? :confused:
 
Thuleman said:
The need for IOPS is often very overestimated. I always like to point to this study our VAR did which logged the IOPS for an entire school district at ~200 because most of what they do is open excel files. There is absolutely no point in having a 30k IOPS drive if you only need 200, or even 2000.
Click to expand...

For 200 IOPS sure but 2000? 2000 IOPS is a lot for HDDs, you'd need quite a few to achieve that and if all you need is a few hundred GB of medium IOPS storage, SSDs are way better, since there's no such thing as storage with a medium amount of IOPS, it's either a HDD with a few hundred IOPS or SSD with tens of thousands, and if you need or want more IOPS than a HDD can give your only option is overkill.
 
I do agree with Thuleman that most people really do not understand IOPS... we run some pretty large DB's here, and like I said, we get to about 4000 IOPS. A whole school district being about 200 IOPS to cover sounds about right.

At 2000 IOPS, ya, you are kind of up in the air as to handle with SSD or HDD, but SSD buys you so much additional overhead that it just makes sense from a cost perspective. Most things causing high IOPS will also not require that much storage.. it is just a TON of small read/writes (typically DB related). People tend to confuse this with filers which need high throughput. If you are just serving media files (pictures/audio/video), IOP requirements are usually pretty low... probably your bottleneck would be the number of inodes in your filesystem.

Regarding controllers being bottlenecked.. ya, if you used some sort of storage array and filled it with all SSDs, you would definitely bottleneck the controllers. In which case, you either need more controllers or less SSDs. With a lot of the carrier grade equipment, they specify how many SSDs you can have per controller.. from what I have seen on Hitachi VSPs and 3Pars, you can get about 4 SSDs per controller (and that is all they will sell you as putting more in could cause controllers to have issues). But on the enterprise grade equipment side Hitachi HUS/AMS, Netapps, etc, they don't tend to tell you this and just let you order whatever you want.
 
"Pretty large DB" at 4000 IOPS is actually a "medium size DB".

I count large DBs at around 10k IOPS, going up from there. But, I also have a somewhat different perspective on it.
 
lopoetve said:
"Pretty large DB" at 4000 IOPS is actually a "medium size DB".

I count large DBs at around 10k IOPS, going up from there. But, I also have a somewhat different perspective on it.
Click to expand...

Well, I was talking purely about the redo logs as that does not take up much space and eats the most IO, so it lends itself to being put on SSD. Are you talking about just that or the DB as a whole.. because I'm not including any parts of the actual dataset... which while it can eat up a lot of IO, also requires storage capacity.

If the whole DB, then ya.. well over 10k IOPS, but you usually augment that in traditional HDD.

But yes, large/medium.. it's all relative. 3 of our customers are responsible for 3 of the largest DBs in the world... which absolutely dwarf our DBs.. but I would still consider ours large.
 
It is unfortunate to see that apparently the OP is full of misinformation as this is exactly the type of thread I needed.

My current set up is an ESX whitebox and a QNAP. The QNAP is serving as both a file server and NFS datastore. I will take my current drives out of the QNAP and use them in a virtualized openindiana/napp-it machine in conjuncture with a M1015 I picked up.

What drives should I purchase to stick in the QNAP ? I'm thinking ~500 GB drives in either RAID0 or RAID1. In terms of use, I will be creating machines fairly often as the purpose of the esx box is to aid in certification study. I won't be doing anything too crazy with my 'production' VMs, probably just a domain controller, instance of Ubuntu for sabnzbd/sickbeard, pfsense and a box to run something like spiceworks on.
 
Buy whatever...good 7200 RPM drives will do the job. My lab runs on WD Black 1TBs. Or a pair of SSDs and a pair of SATAs..if you have 4 bays.
 
I wouldn't say the OP was full of misinformation.. he was generalizing, but it's hard to generalize since VM use is very specific per user.

As to the disk for your QNAP array.. It's a NAS, so you are going to be limited regardless to 1Gb/s theoretical.. probably closer to 700Mb/s practical.. and more realistically given it is a SOHO device, MAYBE 200Mb/s. A single drive is 3-6Gb/s.. so throughputwise.. anything will do... ethernet latency is going to be much higher than your disk access time. So you can probably go with anything and you won't really notice too much of a difference. The question would then be IOPS needed/desired. The more IOPS you plan to use, the faster disk you would need.
 
Kelter said:
I wouldn't say the OP was full of misinformation.. he was generalizing, but it's hard to generalize since VM use is very specific per user.
Click to expand...
Thanks :) it is really difficult to generalize when describing the myriad of usage scenarios for a VM Host.

Kelter said:
As to the disk for your QNAP array.. It's a NAS, so you are going to be limited regardless to 1Gb/s theoretical.. probably closer to 700Mb/s practical.. and more realistically given it is a SOHO device, MAYBE 200Mb/s. A single drive is 3-6Gb/s.. so throughputwise.. anything will do... ethernet latency is going to be much higher than your disk access time. So you can probably go with anything and you won't really notice too much of a difference. The question would then be IOPS needed/desired. The more IOPS you plan to use, the faster disk you would need.
Click to expand...
ethernet latency inside your subnet should be 1ms or less. Disk latencies can be as long as 8ms but average 4ms as I recall. You can reduce these using short-stroking if necessary. SSDs have near zero latencies.

When trying to calculate the number of IOPS needed, I assume a peak of 25 IOPS for each light role (domain controller, email). This estimate is intentionally low: most of the time, if IOPS are insufficient, these roles don't suffer very much. I assume 50 IOPS per NAS, if only for copying large numbers (>5000) of small files (<5MB), but you may need to halve or double that number according to your usage. I assume a minimum of 200 IOPS peak per actively-used database, but it's much safer to measure your own setup than to make an assumption.
 
dave99 said:
a 5400 rpm drive runs 1 instance of windows somewhat poorly, let alone several in a virtual environment. If someone has the need for a home lab, then the advice above is horrible.
Click to expand...

you are doing it wrong then. 1 5400 rpm drive is more than capable of running multiple low i/o windows/*nix instances in a home lab.

But i was merely making the statement to argue that there isnt necessarily a cut and dry rule to what disks you should use when, as the OP would suggest.
 
4saken said:
you are doing it wrong then. 1 5400 rpm drive is more than capable of running multiple low i/o windows/*nix instances in a home lab.

But i was merely making the statement to argue that there isnt necessarily a cut and dry rule to what disks you should use when, as the OP would suggest.
Click to expand...
whoa whoa whoa, I'm not making rules of any kind here. I'm just offering a starting point to newbies who want to put together VM hosts. If you have a suggestion for what to change in my post, please say so. I'm trying to make this a constructive thread.

The general consensus seems to be that SSDs are the endgame for all disk technology, and I don't necessarily agree. They have their advantages and their disadvantages. With that said, I'm happy to start the post off with a heartfelt recommendation that admins consider hosting their VMs using SSDs (single or array/pool).
 
There is too much to disagree with to make a reasonable point by point post. Perhaps, it would be better to break this down into classes of vm hosts including small with local file system, small with shared file system, etc.
 
I am not new to vmware but I am new to it in production environments.

What are peoples thoughts on Raid configurations? I was toying with one setup here with the virtual machines running on a raid 5 setup and it moved at the speed of a snail.
 
depends on what you're doing, read/write workload numbers, disk types, etc- that's not a question that can be answered in short.
 
piker28 said:
I am not new to vmware but I am new to it in production environments.

What are peoples thoughts on Raid configurations? I was toying with one setup here with the virtual machines running on a raid 5 setup and it moved at the speed of a snail.
Click to expand...

jimh425 said:
I prefer Raid 10.
Click to expand...
RAID5 has some often-ignored side effects. While it can achieve the throughput of n-1 disks, it will achieve the IOPS of 1 disk (namely the slowest disk). Adding more disks will not increase the IOPS. This principle extends to RAID6 (n-2) as well. RAID50 and RAID60 improve upon this, to the number of RAID5s striped across. Ex: If you have 6 disks in RAID50, you'd have a RAID0 of 2 RAID5s with 3 disks each. You'd achieve the IOPS of the two slowest HDDs in the total RAID50, roughly.

RAID10 yields the throughput, capacity, and IOPS of n/2 disks (roughly). Some implementations of RAID10 will allow read operations to occur on single disks in the RAID1s within the RAID10, so you'll have Read IOPS of n disks, which is as fast as a RAID0 of those disks. In all cases, Write IOPS will be slightly less than n/2. Ex: if you have 6 disks in a RAID10, you'd have the capacity of 3 disks, and the IOPS of 3 disks. Your IOPS will be significantly greater than if you had used RAID5 or RAID50: you'd have 300% the IOPS of the RAID5, or 133% the IOPS of the RAID50.

RAID5,6,50,60 all have very long rebuild times. RAID10 does not.

For this reason, I prefer RAID10 to any other RAID layer.
 
Last edited:
I had that feeling but was curious to see what peoples thoughts are. On my personal setups I do not run any raid at all but at work my managers love backups backups backups!
 
You must log in or register to reply here.
Back
Top