So I'm expanding the storage at work soon, i got a 24bay enclosure, that will be either hooked up to the server/servers via fibre or scsi. I'm thinking of using 2tb drives. Should i do 1 volume across all 24drives in raid6 with 2 set to hot spares? or should I go with something else? There will be no databases, just storage, accessed by 10-30users, but nothing too heavy of a load. What are your thoughts [H]?
![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
Should i set 24bay system to Raid6 or not?
- Thread starter Pitbull
- Start date
Concentric
[H]ard|Gawd
- Joined
- Oct 15, 2007
- Messages
- 1,028
Basically check out some of the other threads in this section and you'll realise that the popular storage setup at the moment is to use ZFS.
Suggest you check out some of the systems other people have been building and follow the recommendations. You can then come back with more specific questions about your particular requirements.
Suggest you check out some of the systems other people have been building and follow the recommendations. You can then come back with more specific questions about your particular requirements.
There was a good discussion on this issue recently here.
24 disks would be right at my limit in RAID6 with two hot spares and doing good back ups. I would also consider two RAID6 arrays using 3tb drives if your budget can handle it -- but drive prices have really gone up lately.
24 disks would be right at my limit in RAID6 with two hot spares and doing good back ups. I would also consider two RAID6 arrays using 3tb drives if your budget can handle it -- but drive prices have really gone up lately.
What controller are you planning on using? I personally wouldn't go over 14 drives for a single RAID6. I don't know how much space you are willing to give to spares and parity, but to be very safe I would do 2 11 drive RAID6 arrays with 2 global spares shared among the arrays (in a 24 bay enclosure). What kind of file storage/usage patterns do you envision? What kind of backup strategy do you have in mind?
Yeah i think the 2 raid 6 with global hot spares is good. Backup is a LTO super loader that runs each night incremental, with full backup on day 1 for each set. I currently have 12bay raid 5 with 2 hot spares, that might become a disk backup for before LTO gets updated, and 2 smaller 6 drive raid 5 arrays that need to be fazed out due to age.
What LTO level backup do you have, and how many drives are in the changer. Here is where Disk to disk to tape comes into play. Lets say you go with the 2 11 drive RAID6 arrays. That yields you 27TB per array if you go with 3TB drives. Lets make some assumptions here (no jokes about what happens when you assume please). Since this is a business based array, lets assume you work 9-5, which allows you a maximum backup window of 16 hours. I will also assume max usage here, since you should be able to handle full (or near so) arrays. That means you would need to backup 54 TB in 16 hours, which means you need to backup at a rate of 3.375TB/hour, or a rate of 56.25 GB/minute. LTO5 (the latest and greatest) has a theoretical throughput of 140MB/Second (native without compression). That gives you 8.4GB/Minute (Native) or up to 16.8GB/Minute (Compressed, if your data is compressible). You see the problem here if you are doing full backups every night. Even doing incremental backups, you are still way short of time each time you start a new master for the increments. If you have 2, 3 or 4 drives in your changer and the necessary bandwidth between the array and the LTO device(s), then you come closer to achieving what you would need depending on the compressibility of your data. Also, lets assume you get 1.25:1 (a valid real world estimate if your data isn't all incompressible audio/video files). LTO5 is 1.6TB Native, yielding you 2TB/Cartridge. That would mean your changer would need to be at least 27 Slots just for the master backup, and if you did weekly incremental backups you would need a few more slots to handle new/changed files from the initial master. To be safe, with a cleaning cartridge in a slot as well, you should have at least 32 slots in your backup changer.
Last edited:
Well our usage is below 7-8TB now, i use LTO 3 with 7 tapes +1 cleaning which gives me 2.8tb uncompressed, full backups start on Friday last till Monday morning. I change tapes over the weekend, we do not have alot of new data coming in so incremental always finishes waaay ahead of schedule. Disk - Disk to LTO works great sometimes i use the smaller arrays to offload before and let the LTO run from those during work hours. I will consider LTO5 with dual magazines if our usage spikes.
It is in my view (and many others here), the best combination of performance, economics, and reliability for an array of that size. With 24 drives you are starting to push the limits of what is generally considered safe for a RAID 6 array. Stripe two 12-drive RAID 6s, and the math behind the reliability starts to approach the "near impossible to fail" realm (barring the usual weak points like controller failure). It is also a more economical use of space compared to going with a RAID 10.
Why would you worry? Rebuild times? Data Density? For the 100,000th time (not specifically to you). RAID IS NOT A BACKUP! RAID allows for high availability. It allows for your array to continue to operate (albeit slower in a degraded state) and for people to continue to read/write to the drives without downtime. If you do not have a backup plan, actually backup your data and make that backup just as important (ie don't forget to check your backups), you are setting yourself up for sorrow.
Trepidati0n
[H]F Junkie
- Joined
- Oct 26, 2004
- Messages
- 9,269
This...UGH. I think we probably get a good dozen threads every year saying "my raid died and I have no backups, save me". Furthermore, we get a few threads every year that a RAID6 (or ZFS equivalent) died with no backups. I understand you play games to maximize your chance of a success on a rebuild...but the chance of failure is never zero unfortunately.
I often get calls from people that decided the cost of backup was too much, usually begging me to get their data back. Many of these are also customers that balked on the proper backup systems due to cost or expediency etc.. It is sad, but too often will cost 5x (in dollars, 100x in worry) in recovery costs what a proper backup system would cost in the first place. I actually turned down a job recently where they wanted to reuse a shelf they had and do a 14 drive RAID5 with no backup plan.
@OP
One more thing to consider when using parity RAIDs like levels 5 and 6 is that best performance is had when the number of effective data-bearing drives (EBD) is a number which fits neatly into the binary system.
For RAID 5, where the number of effective data-bearing drives is (N - 1), the best total number of drives is 3 (2 EBD), 5 (4 EBD), 9 (8 EBD), 17 (16 EBD), etc. For RAID 6, the model is (N - 2), so the total numbers of drives for best performance are 4, 6, 10, 18, 34 etc.
For 24 bays, I would do two RAID 6s of 10 drives each, plus at least 2 global hot spares. I do realise however, that in the real world, such neat fittung isn't always possible.
If you're going to do two RAID 6s, what are you going to pool the arrays with? Or will you keep them separate?
One more thing to consider when using parity RAIDs like levels 5 and 6 is that best performance is had when the number of effective data-bearing drives (EBD) is a number which fits neatly into the binary system.
For RAID 5, where the number of effective data-bearing drives is (N - 1), the best total number of drives is 3 (2 EBD), 5 (4 EBD), 9 (8 EBD), 17 (16 EBD), etc. For RAID 6, the model is (N - 2), so the total numbers of drives for best performance are 4, 6, 10, 18, 34 etc.
For 24 bays, I would do two RAID 6s of 10 drives each, plus at least 2 global hot spares. I do realise however, that in the real world, such neat fittung isn't always possible.
If you're going to do two RAID 6s, what are you going to pool the arrays with? Or will you keep them separate?
Trepidati0n
[H]F Junkie
- Joined
- Oct 26, 2004
- Messages
- 9,269
Unless you are trying to squeeze out performance..."EBD" has marginal if not negligable benefit to the average user. In the case of the OP, throwing this wrench into the works will not benefit him in any significant way.
What the? I'm not saying RAID is a substitute for backup. I'm actually quite an avid believer in backup, it's what I do for a living, and I have backup of every single important file on my home computers.
Thing is, this guy probably does not. And if it's no problem just restoring the data, why not just run RAID-0 ? I mean, it doesn't matter, its all backed up anyway, right? RAID is just about availability! Well, no. Not just availability. In a perfect world you'd never get to the point that you'd need a backup, except for version handling. Obviously, there are other ways around this like snapshops and shadow copies, but this might not be used or even supported on your particular system.
Stop viewing everything so black and white, please. I don't, which is why I don't believe it's necessary to start every post I make in a thread about RAID with "RAID IS NOT BACKUP!!!".
The guy asks which RAID level he should use. Running a RAID-5 with this many disks makes less sense than RAID-0 since the chances of another disk dying while rebuilding is actually pretty huge. And yes, I know he didn't mention RAID-5.
I specifically said I didn't mean you. I have just read too many people here (and dealt with too many people as a consultant) who lose their arrays and then bitch that they lost their data because they never considered (even after being told) that RAID is not a backup. One question, why would you say that running a RAID5 makes less sense than running a RAID0? At least the RAID5 could survive one failed disk, the RAID0 could not. And the chances of a drive dying during rebuild isn't "pretty huge", it is just elevated because of the increased stress.
The reason a raid-5 with this many/this large disks does not make more sense than raid-0 is that the chance that the array dies while rebuilding is actually rather large, and you don't get the speed benefit of raid-0. So, in reality, you're probably going to restore everything from backup with either raid level, so why not use raid-0 and get the speed benefit?
The above is in no way a suggestion though, just my way of showing that both raid levels are stupid with such large disks.
The above is in no way a suggestion though, just my way of showing that both raid levels are stupid with such large disks.
The advantage of using a redundant RAID level is that is (should) improves both your recovery point objective and your recovery time objective.
Backups typically are 24 hour based - though of course this depends on the data set etc. So in the example timescale given, if you have a RAID-0 and it dies, you potentially lose a day's worth of data. Depending on the data amount, you are obviously without accesss to that data until you restore it, so having the redundant data platform improves the availability of your data. I'd also note that whilst a RAID rebuild is underway, at the loss of performance one still has access to the data and potentially further backups can be generated to meet the desired RPO.
Whenever you design a computer system it must be aligned with business requirements. So I would advise the OP to discuss the value of the service and data, agree a RPO / RTO and then design / develop / purchase in accordance.
Backups typically are 24 hour based - though of course this depends on the data set etc. So in the example timescale given, if you have a RAID-0 and it dies, you potentially lose a day's worth of data. Depending on the data amount, you are obviously without accesss to that data until you restore it, so having the redundant data platform improves the availability of your data. I'd also note that whilst a RAID rebuild is underway, at the loss of performance one still has access to the data and potentially further backups can be generated to meet the desired RPO.
Whenever you design a computer system it must be aligned with business requirements. So I would advise the OP to discuss the value of the service and data, agree a RPO / RTO and then design / develop / purchase in accordance.
I think you're missing my point, haileris. My point is, that you need to choose the proper raid level for the disk setup you have. We all know the reason to use raid levels with redundancy (which all of them should have, if you look at what the abbreviation "RAID" actually stands for).
Thing is, if you choose a raid5 for your setup of 20 disks of 2 TB, you're basically not gaining any benefit from the parity data, since you'll have a very big risk of losing another disk during the rebuild.
Thing is, if you choose a raid5 for your setup of 20 disks of 2 TB, you're basically not gaining any benefit from the parity data, since you'll have a very big risk of losing another disk during the rebuild.
I wasn't directly addressing your post, though in truth I do disagree with some of what you say to a mild degree. For example, I wouldn't advovate 20 disks in a RAID5 set myself but again I'd point out if you lost one then you have a chance to backup any changed data since the last backup, something you wouldn't (easily) be able to do under a non-redundant system. There's also no quantification or justification for the claim that you'll "have a very big risk of losing another disk during the rebuild". I have 48 2 TB disks in 4 x RAID-6. I've had to rebuild due to drive dropouts, never lost another disk though during the rebuild process. So what is a "very big risk"? Perhaps you can use some of the MTTF data lying around the web to quantify what that risk is because until then it is largely heresay?
Finally, I think you are missing my point (but apologies if you are not). It all starts with the importance of the data. You design according to the customer requirements, not to how many disks you happen to have or what RAID controller you happen to have lying around.
I would never recommend a RAID5 of 20 disks. I generally recommend RAID6s wit no more than 14. In any case, regardless of your fear of failure you will be better served in a general file server situation with RAID5 over RAID0. Raid0 in certain situations will give you a great sequential speed advantage in a workstation or transactionally for a Database Server. As a server, the added sequential speed boost will be negated unless you have greater than 1Gbit bandwith. Other than the specific cases where you will realize those specific benefits, you will be better served with the extra uptime the parity affords you as well as the additional space (example, 6 drive R1 w/ 2TB drives yields ~6TB, where a 6 drive R5 yields you ~10TB, 6 Drive R6 yields ~8TB). R5 & 6 also offer you increased read speeds, and with the right card very little write penalty.
Last edited:
omegatotal
Gawd
- Joined
- Mar 15, 2002
- Messages
- 672
what ever happened to raid controllers supporting 5ee or 6ee modes, where you have an additional (or more) parity stripes than the normal raid 5 or 6 model.. this meaning that you could have 2 drives fail at the exact same time and still be able to recover ( if my memory serves me well)
There are a number of non-standard raid implementations, 6ee for example as you mentioned essentially has a hot-spare which is part of the block rotation as opposed to a drive which just sits there waiting for failure. Most of these "We Can Do RAID Better" formulas have their own particular limitations, and in some cases are more trouble than they are worth. RAID6 can survive 2 drive failures and can rebuild completely if you have 2 spares available. In any case, the purpose of RAID is the ability to maintain up time while/until repairs can be completed. For a whole host of reasons, it should never be considered a backup of any kind.
It took me a while to figure out how 6x 2TB = 6TB... But then I realized you were referencing a RAID-1 array instead of RAID-0 which we were all discussing. Just felt like mentioning this...
Mistype on my part, my point was to reference a some-kind-of-redundancy array vs the same. We had already discussed why RAID5 is better than RAID0 in a general file sever case. I also wanted to give the benefit of R6 over R1, where for example a 6 drive R6 can survive any 2 drives in that server array set failing , and the wrong 2 drives failing in a R1 is fatal (2 masters in a R1 can fail if the mirrors don't, but if a master and a mirror fail you are SOL)
Last edited: