RAID6: what is a good ratio for usable space to space used for parity?

Cerulean

Cerulean

[H]F Junkie
Joined
Jul 27, 2006
Messages
9,476
I will be building a RAID6 with five 2TB drives, which would give me 6TB usable space and 4TB for parity. If I want to add more 2TB drives, how far should I not go / at what point would RAID6 be equal to RAID5?
 
First, let me preface this by saying you should ALWAYS have good backups because no single RAID system should be considered "Safe". The maximum members I recommend in a raidset is dependent on a number of factors, including member size, activity and the kind of data that lives on the array. With 3TB/4TB drives, For enterprise mail/file/print/database I generally recommend 8-10 members (including the parity members). For home media servers I generally recommend a maximum of 12 drives (up to 14 for arrays with excellent backups and mostly RO static info) not including hot spares per raidset.
 
Again, you should make sure to have a reliable automated backup. For home use, what you should do if a drive fails is to stop using the array, backup any new data that is not in your backup storage yet and then replace the failed drive and rebuild. It does not really matter if another drive fails during rebuild because you have everything in your backup. And for that reason RAID5 with such small array sizes is enough most of the time.

If you cannot live with such a scenario and absolutely need access to your array even if a drive fails then RAID6 is for you. I personally would not start with RAID6 with less than maybe 8 drives today.
 
Cerulean said:
Is there a particular reason you have for this? :?
Click to expand...

Wasted space. Even if harddisks are not particularly expensive, they are also not extremely cheap. I would even say most home users should not use RAID at all and do backups instead (because most people I know do not care to backup their data). Best is both, but it is also the most expensive. With your harddrives you are wasting 40% of space for parity. And that just for the unlikely condition that two drives fail with the very low IO rate of a home setup? You did not specify your backup solution, how valuable your data is and how your workload looks like. It is not really possible to deduce the best setup for you.
 
Last edited:
omniscence said:
Wasted space. Even if harddisks are not particularly expensive, they are also not extremely cheap. I would even say most home users should not use RAID at all and do backups instead (because most people I know do not care to backup their data). Best is both, but it is also the most expensive. With your harddrives you are wasting 40% of space for parity. And that just for the unlikely condition that two drives fail with the very low IO rate of a home setup? You did not specify your backup solution, how valuable your data is and how your workload looks like. It is not really possible to deduce the best setup for you.
Click to expand...

I have to disagree with you here. The space is not wasted, it is being used by parity information to keep your array up in case of failure. As to the "low I/O" rate in a home server, this has exceptionally little bearing n the chances of failure. The drives are still powered on (even if sleeping) but all external failure points (power, temp etc) are still in play.
As to not using RAID at all, again i must disagree. Most people doing home servers want their data in a central location with pooled storage of some kind, not having to worry about what files are on what drive letters, volumes etc and with at least some type of uptime adjunct (parity) so they don't need to restore their entire array for a single (or even dual) failure.
Above all this, as many of my posts suggest, Nothing beats a robust backup strategy!
 
It's not wasted space, but does take a disproportionate amount of the arrays raw size. At least with 8 disks in a raid6, you are getting 12TB usable in exchange for your 4tb of parity.
 
8 Drive max for RAID5 (six disks is the usual cap in the workplace)
12 Drive max for RAID6 (a fair amount of parity considering the disk size and rebuild time)
(16 drives would be OK in RAID6 if you are a SOHO/enthusiast, but dangerous considering at least 90hours of rebuild on any top end card like the 1882)

Less disks if you are more worried, there is always RAID50 and 60 when you need to go big with safely sized RAID's.
 
Greetings,

The specs in my signature are mostly accurate. I no longer have a 750W PSU, but now have a Corsair AX1200 (1.2kW). In addition, I have a Cosmos II chassis. At this time, I haven't had the opportunity to determine a wind tunnel fan setup scheme (thus have not moved over my 252cfm Delta fans). The HDDs I still have, but will not be powering them until I get a RAID in place. I no longer trust Green drives for my activities due to the tier/scale of work I have advanced.

The purpose of using RAID6 over RAID5 is entirely to delay total storage failure, like having two warnings or second chances. I intend to setup a RAID1 consisting of two 4TB disks dedicated for housing Acronis backup images (totally sufficient for 6TB usable factoring in archive compression ratios, and considering that some of this space is used temporarily for mid-term archiving before permanent deletion). I will be purchasing WD Black drives entirely; this particular choice is for the warranty if not also its greater endurance / closer to being an enterprise-grade product (thus reliability). I will be getting an Adaptec 5805 (though overkill, precisely why I chose it -- to extend life of the overall system without burdening particular components all the time), which features two SAS ports supporting 8 SATA drives each. The Cosmos II has 13 drive bays, so there is a lot of room and potential for growth. I will probably be running with a chassis like this for more than a decade (or even longer!).

If I setup the two RAIDs correctly, having the blessing of multiple chances allows me to utilize the 5-year warranty on the disks more effectively and without any grief. This in itself will also lower long-term costs and prevent emotional and mental headaches that I have increasingly experienced with my Green drives the past 6-10 months. Unrelated, I once accidentally threw out a 32GB SDXC card loaded 100% to capacity with unprocessed, unpublished, new photos at a fast-food restaurant, and realized what I had done days later. I have also lost a huge collection of photos I meant to archive in backups to a Green drive that finally decided to kick the bucket on me; I believe I also lost quite a number of unpublished photos in this scenario too. Recently, about 2-3 weeks ago, I encountered a scare that my new France photos (processed and ready for publishing, but not yet published!!) were gone forever to a dead Green drive (it was brand new too!). Fortunately, WD Diagnostic Tools made successful repairs and the drive is considered fully operational. All three incidents occurred in the past year. Before I recontinue my attempt to get my photos published to Flickr, I will wait until I have a more reliable storage system in place so that I don't open my HDDs to opportunity to die at inconvenient timing. :| All this, the stress, emotional and mental experience, is too much for me. It's either move up or go home, and I can't go home with the amount of money I have already spent on photography equipment and supplementing components for my computer (such as the 1.2kW PSU, Cosmos II chassis, i7-3930K, and a sickeningly gut-wrenching amount for photography equipment so far).

What do I use my system for? Mostly for processing plain, HDR, and panoramic photos. I do use it for gaming once in a blue moon. I still will need to upgrade my 128GB SSD to a 512GB, my RAM to 64GB total (vs current 32GB), a 1950W UPS ($1500 D: ), and lastly the video card (last thing on my mind right now, and I have patience to wait for Adobe and AMD to finish implementing OpenCL GPU support for AMD cards into After Effects and Premiere Pro so that when the time comes I could make a more fair judgment on whether to go NVIDIA for its CUDA or stay with AMD). I think my priority here, after implementing the storage system (including redundant storage for backups), I will attain a 2U 1950W UPS as an additional counter-measure to overall system + storage/RAID failure (even though I will have a dedicated battery install on the RAID card). Then, it will either be 64GB RAM or a 512GB SSD. :D

I greatly appreciate the responses to this thread, as they have provided me with much direction and thought. :)
 
Last edited:
You should consider placing your drives in a separate system. Harddrives seem to like a constant temperature environment the best without being powered up and down often.

terayon1515 said:
... at least 90hours of rebuild on any top end card like the 1882 ...
Click to expand...

Does it really take 90 hours? Even for an idle array? Linux software RAID usually rebuilds within 6-8 hours for 3 TB drives.
 
Does it really take 90 hours? Even for an idle array? Linux software RAID usually rebuilds within 6-8 hours for 3 TB drives.
Click to expand...

I was going to say similar. With the same 16 drives I expect my linux software arrays to rebuild in less than 12 hours with modern 7200 RPM 2TB or larger drives. 90 hours is crazy long. Is that without a BBU?
 
omniscence said:
You should consider placing your drives in a separate system. Harddrives seem to like a constant temperature environment the best without being powered up and down often.
Click to expand...
I don't have that kind of ability. I live in an apartment.
 
terayon1515 said:
8 Drive max for RAID5 (six disks is the usual cap in the workplace)
12 Drive max for RAID6 (a fair amount of parity considering the disk size and rebuild time)
(16 drives would be OK in RAID6 if you are a SOHO/enthusiast, but dangerous considering at least 90hours of rebuild on any top end card like the 1882)

Less disks if you are more worried, there is always RAID50 and 60 when you need to go big with safely sized RAID's.
Click to expand...

Raid5 should not be used with today's 1T+ size drives, the chance of failure during rebuilding is too high and has been shown often with desktop class drives, even enterprise.

Raid 6 would never be like raid 5, ever, because it has added parity just make sure you have a good raid card for raid 6.

Adaptec 5805 while a "decent" card, i have 2 at work, and in the last month, BOTH have started to drop drives from raid arrays, this was a past issues Adaptec always had and i thought it was fixed but apparently not, get an LSI or Areca card, also performance for me was.. meh! for a raid 6 with 10 hitachi 2T drives with 2 hot spares.

Question, what on earth do you need a 1950W PSU for..... talk about over kill.

if these pictures are THAT important you need a proper backup, not 2 raid arrays.
 
MrGuvernment said:
Raid5 should not be used with today's 1T+ size drives, the chance of failure during rebuilding is too high and has been shown often with desktop class drives, even enterprise.

Raid 6 would never be like raid 5, ever, because it has added parity just make sure you have a good raid card for raid 6.

Adaptec 5805 while a "decent" card, i have 2 at work, and in the last month, BOTH have started to drop drives from raid arrays, this was a past issues Adaptec always had and i thought it was fixed but apparently not, get an LSI or Areca card, also performance for me was.. meh! for a raid 6 with 10 hitachi 2T drives with 2 hot spares.

Question, what on earth do you need a 1950W PSU for..... talk about over kill.

if these pictures are THAT important you need a proper backup, not 2 raid arrays.
Click to expand...
I'm not sure I agree with your statement about "Raid 6 would never be like raid 5, ever, because it has added parity ..." http://www.zdnet.com/blog/storage/why-raid-6-stops-working-in-2019/805

Do you have recommendations for any particular LSI or Areca RAID controllers that support at least 8 SATA drives?

APC Smart-UPS SUA2200RM2U is the UPS I intend to get later in the future (or something about the same).
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
I say go for 6 or 10 total drives with RAID-6. 5 drives with RAID-5. I try to stick to the rule of thumb of the number of DATA drives being a power of 2. As for controllers, any 6Gbps LSI SAS card that supports the RAID level you want should be fine. Unfortunately if you want hardware RAID-5/6, avoid the inexpensive IBM M1015 that many of us recommend. I think the M5015/M5016 should support those, though I'm not sure if they are similarly inexpensive compared to the LSI-branded version.

For the UPS, I prefer Powerware these days, but the APC SUA you linked is fine. Do check out www.refurbups.com either way though.
 
Last edited:
You might take a look through my posts over the past year or so. I help my wife run her wedding photography business of 9 years. I manage all her data. I *fully* understand initual photography expenses (just as an example she personally carries two camera on her person at weddings. One D3 and one D4) and I just ordered a D600 for the 5th backup camera! ;)

I'll cover it a little here but go back through my posts for some idea's on how I've done it for 9 years now to have a 100% perfect retention/no loss record.

The images start in camera (and all of our cameras have dual cards with the cards always set to backup). only one card comes out of the camera at a single time. Never pull the backup card until you have made another copy of the first card. The first copy goes onto the wifes workstation. Immediately a second copy is made to my workstation (now it goes to the ZFS SAN build *and* my PC). So at this point we have 3 copies on 3 different file systems in 3 different machines. She will usually retain those CF/SD cards in a lockbox until she has processed the photos for the client and uploaded the processed JPGs to smugmug (smugmug retains an unlimited amount of JPGs for a single yearly fee. Its a near unbeatable price if you need to sell directly to clients and also want another backup point). In our case the client usually gets a DVD/Memstick with their wedding photos in JPEG form on it as well (and our legal duty to retain those photos ends, though I still keep them anyway for our uses).

I also add two other forms of data protection, bluray and LTO3. We seem to grow data at a rate of 1-1.5TB a year so I'm using 40 or so bluray's per year and 3 LTO3 tapes. Good bluray disks are $1 each and brand new LTO3 tapes are 10-12$ each (per 400GB uncompressed, which is all you get with RAW images btw). So I spend another 70$ a year on "archive" media. I don't test them often, but should. Considering that I'd probably double up on the LTO3 tapes and skip the blurays in the future to be honest. I'm not sure (I really like having them on a non-magnetic media in some form). Regardless, in the end those photos are located in 6 different locations not counting the client (or the SD/CF cards). My biggest concern is fire and the need for offsite storage of the actual RAW images. I'm seriously considering using amazon s3 glacier for a years worth of data. I'll have to see what it ends up costing.

Cerulean said:
Greetings,

The specs in my signature are mostly accurate. I no longer have a 750W PSU, but now have a Corsair AX1200 (1.2kW). In addition, I have a Cosmos II chassis. At this time, I haven't had the opportunity to determine a wind tunnel fan setup scheme (thus have not moved over my 252cfm Delta fans). The HDDs I still have, but will not be powering them until I get a RAID in place. I no longer trust Green drives for my activities due to the tier/scale of work I have advanced.

The purpose of using RAID6 over RAID5 is entirely to delay total storage failure, like having two warnings or second chances. I intend to setup a RAID1 consisting of two 4TB disks dedicated for housing Acronis backup images (totally sufficient for 6TB usable factoring in archive compression ratios, and considering that some of this space is used temporarily for mid-term archiving before permanent deletion). I will be purchasing WD Black drives entirely; this particular choice is for the warranty if not also its greater endurance / closer to being an enterprise-grade product (thus reliability). I will be getting an Adaptec 5805 (though overkill, precisely why I chose it -- to extend life of the overall system without burdening particular components all the time), which features two SAS ports supporting 8 SATA drives each. The Cosmos II has 13 drive bays, so there is a lot of room and potential for growth. I will probably be running with a chassis like this for more than a decade (or even longer!).

If I setup the two RAIDs correctly, having the blessing of multiple chances allows me to utilize the 5-year warranty on the disks more effectively and without any grief. This in itself will also lower long-term costs and prevent emotional and mental headaches that I have increasingly experienced with my Green drives the past 6-10 months. Unrelated, I once accidentally threw out a 32GB SDXC card loaded 100% to capacity with unprocessed, unpublished, new photos at a fast-food restaurant, and realized what I had done days later. I have also lost a huge collection of photos I meant to archive in backups to a Green drive that finally decided to kick the bucket on me; I believe I also lost quite a number of unpublished photos in this scenario too. Recently, about 2-3 weeks ago, I encountered a scare that my new France photos (processed and ready for publishing, but not yet published!!) were gone forever to a dead Green drive (it was brand new too!). Fortunately, WD Diagnostic Tools made successful repairs and the drive is considered fully operational. All three incidents occurred in the past year. Before I recontinue my attempt to get my photos published to Flickr, I will wait until I have a more reliable storage system in place so that I don't open my HDDs to opportunity to die at inconvenient timing. :| All this, the stress, emotional and mental experience, is too much for me. It's either move up or go home, and I can't go home with the amount of money I have already spent on photography equipment and supplementing components for my computer (such as the 1.2kW PSU, Cosmos II chassis, i7-3930K, and a sickeningly gut-wrenching amount for photography equipment so far).

What do I use my system for? Mostly for processing plain, HDR, and panoramic photos. I do use it for gaming once in a blue moon. I still will need to upgrade my 128GB SSD to a 512GB, my RAM to 64GB total (vs current 32GB), a 1950W UPS ($1500 D: ), and lastly the video card (last thing on my mind right now, and I have patience to wait for Adobe and AMD to finish implementing OpenCL GPU support for AMD cards into After Effects and Premiere Pro so that when the time comes I could make a more fair judgment on whether to go NVIDIA for its CUDA or stay with AMD). I think my priority here, after implementing the storage system (including redundant storage for backups), I will attain a 2U 1950W UPS as an additional counter-measure to overall system + storage/RAID failure (even though I will have a dedicated battery install on the RAID card). Then, it will either be 64GB RAM or a 512GB SSD. :D

I greatly appreciate the responses to this thread, as they have provided me with much direction and thought. :)
Click to expand...
 
@bp_968

!

I don't have that much money to do exactly the same as you are doing. Other differences are that I have a very poor internet connection of 50-60 KB/s upload speed, one DSLR body with no backup slots and uses SD, I use Flickr, RAW-only shooting, convert RAWs to 24/32-bit TIFF, process TIFFs through NoiseNinja, and then do further processing with Lightroom and Efex software suite, and I don't do photography for money or business or as an organization. :X
 
NeghVar said:
What happens if the RAID controller fails? Is all the data lost?
Click to expand...

Pain in the ass... you can try sourcing another of the same kind of RAID controller (or in some cases same brand from similar time period MIGHT work but do NOT just assume it will). If that's not possible or if you don't want to try it, then you take clones (using dd) of each drive, hook them up to a machine running software like R-Studio (or similar), tell it as much about the array as possible, and cross your fingers that it recovers your stuff.

R-Studio has bailed me out twice back when I used a piece of crap Adaptec 3085 RAID card plus once with Intel Matrix RAID. Now I use ZFS software arrays which are inherently safer, but should something go awry I won't be able to use R Studio to fix it (unless they add ZFS support someday).
 
The 9260 is a good card and I would take it over an Adaptec 5805 100% of the time. Make sure to update the controller with the latest firmware before adding any arrays if the card isn't provided with the latest out of the box (do this regardless of which controller you get).
 
Alright, I purchased the following:
Grand total: $1448.5 shipped

I am looking at the downloads for the RAID controller. Which software do I need to download? I imagine that at minimum I need to install MegaRAID Storage Manager - Windows - 5.6, but do I need Windows SMIS Provider VIB - MR 5.6 or StorCLI 5.6?
 
Last edited:
As an Amazon Associate, HardForum may earn from qualifying purchases.
NeghVar said:
What happens if the RAID controller fails? Is all the data lost?
Click to expand...

solid answer: it depends.

Some of the LSI controllers can read and import arrays from other LSI controllers. I went from a dell perc5 (lsi 8408 clone) to a IBM M5015 (9260-8i clone), which is basically a 2 (maybe 3) generation jump with no problems. Plugged in my drives, the controller read the config off the drives, and away it went.
 
So using five WD Black drives with the intention of doing a RAID6 configuration, how bad will my overall effective IOPS going to be? Is it going to increase or decrease? :X From what I read, read speeds should increase by approximately 3x but the write speeds should be approximately the same as what a single drive can do.
 
The cache on the controller should really help with the writes, assuming you aren't writing a massive stream that fills the cache before the drives can get it written.
 
Are you sure about using the WD Black drives in a RAID array? I thought this was troublesome now that TLER isn't available on anything other than the Red and RE drives.
 
Pultzar said:
Are you sure about using the WD Black drives in a RAID array? I thought this was troublesome now that TLER isn't available on anything other than the Red and RE drives.
Click to expand...
*pout* I already ordered five WD Black drives. :| Plus they have 5-year warranty, whereas RE3 drives only have 3-year warranty. :( Am I in the wrong for getting Black drives?

dave99 said:
The cache on the controller should really help with the writes, assuming you aren't writing a massive stream that fills the cache before the drives can get it written.
Click to expand...
Going to ask this question now since I know it will come later, but what is an appropriate ratio between read vs write when setting up the RAID6? I had also heard something about a block size or something -- is that supposed to match the block size used to format for NTFS on the RAID array? And is there a recommendation for a different cluster size than the default 4096 since this is a RAID array rather than a single physical+logical drive?
 
From my research the Black drives are at risk of being dropped by the controller card. Unfortunately the only drives that WD officially supports with raid are Red and RE. Hopefully some others will chime in on their experience.

For software raid, I don't think that it much matters.
 
WD Black drives haven't shipped yet, but are being prepared. I've submitted a cancellation for those, so hopefully that will go through before they get loaded onto a truck.
 
OK. WD Black drives shipped, so I will have to issue a cancellation and ship them back. I did some reading on WD Black drives & RAID today, and learned that the reason it isn't a good idea to use them in RAID is largely due to their lack of TLER. RAID controllers will drop out HDDs that don't respond after about 8 seconds, and with TLER this can be prevented. RE and Enterprise drives have TLER.

So, I will pay the premium of WD Enterprise drives once I get the WD Black drives shipped back and refund processed. -_- I like the idea of having a 5-year warranty. Also, I read that Enterprise-grade drives feature heavier/bulkier chassis to handle local nearby vibrations better (consumer-grade drives aren't meant to be running together in adjacent bays/close physical proximity) and supposedly higher quality build material and meant to be able to operate 24/7/365.
 
You must log in or register to reply here.
Back
Top