Bilding large storage with 2TB disc and HighPoint controller

F

Freak1

Limp Gawd
Joined
Sep 9, 2009
Messages
191
Hi there.

After reading a lot of posts in here I decided to register and account and ask my own question.

I'm in the process of building a storage with 24x WD Caviar Green, 2 TB, 32 MB hdwd20eads on a HighPoint RocketRAID 3560 24 ports controller.

I'm going to go with Raid 6.

Is there anything I should be aware off? So far I have only worked with Raid 5, 1 and 2.

Thanks.
 
I'm no expert on this but for a start i'm pretty sure 24 drives is a few too many for one array, even with RAID 6. You'd probably want to split that into two 12 drive or three 8 drive arrays.
 
Why should that be a problem? I can't find anything about that in the manual stating that there is a maximum number of drives in a RAID6 Array.
 
I believe the reason they want you to make separate smaller arrays is in the instance of a disk failure, the amount of time it takes to rebuild a smaller array compared to your one big array is substantially less.
 
#4 Yes that is true. But with 24 in 1 array 2 discs will be used for parity, 2x12 will use 4 3x8 will use 6. I prefere longer rebuild times over that.

I will also need to turn on TLER on all the drives right? Can it be done with my drives? I havent found any info on the hdwd20eads only on 1 TB discs.

http://www.hardforum.com/showthread.php?t=1285254
 
Freak1 said:
Is there anything I should be aware off? So far I have only worked with Raid 5, 1 and 2.

Thanks.
Click to expand...

44TB of data takes a long time to fill up. 24 hard drives take a lot of power to spin up. Your transfer speed is limited by the sloy. RAID takes a long time to rebuild. RAID is not a backup.

I think you are making a bunch of poor decisions.
 
Freak1 said:
Why should that be a problem? I can't find anything about that in the manual stating that there is a maximum number of drives in a RAID6 Array.
Click to expand...

It's not a technological limitation (yes, you can have a 24 disk RAID 6 array), just a sensible measure based on the likelyhood of disk failure.

The likelyhood of more than 2 of the 24 drives failing within a similar time period is fairly high. I'm not saying that they're going to start blowing up a week after you get them, but you're going to have 24 drives that were probably produced around the same time and of the same type so they are likely to start failing at the same time, a little way down the line.
If three or more of the drives were to fail within the time it would take you to buy another drive and start rebuilding the array, you've lost everything.

By using less drives per array, you reduce the likelyhood that an array will be broken since there are fewer drives upon which failure of that array depends. Also, several drives from different arrays could fail without data loss, provided there weren't more than 2 failures in any one array.

Yes, you lose some more of your space to parity, but you're talking about a lot of storage and a lot of disks - if you're taking the RAID thing seriously you need to be a bit more realistic about the amount of storage needed for parity to cover >40TB.
 
Might I suggest using 3 RAID5 arrays and JBODn them together?

This will yeild you higher performance, and faster rebuilds.
 
nitrobass24 That might be a good suggestion. How do i JBODn them together?

Concentric I hear what you are saying. I know RAID is not a backup and all data on the RAID will also be on another RAID 6 100% identical. Yes 3 drives can fail at the same time. But it will have to happen 2 times in 2 different locations at the same time before the data is lost.

GeorgeHR Do you have a link to informations on 24 drives in RAID 6 will be slow, and how slow it will be? And maby a estimate on how long rebuild time will be?
 
Freak1 said:
Concentric I hear what you are saying. I know RAID is not a backup and all data on the RAID will also be on another RAID 6 100% identical. Yes 3 drives can fail at the same time. But it will have to happen 2 times in 2 different locations at the same time before the data is lost.
Click to expand...

Well it's nice to hear that you have a backup (and a large budget by the sound of it) but that shouldn't really be relevant - it's not very sensible to skimp on the fault tolerance of your primary storage simply because there's a backup.

Is this for your business or what? How important is the data?
 
If you do a single array you rebuilds will be measured in Days not hours
 
If you're that corcerned about what ever this data is, why are you using consumer level WD Green drives and not the enterprise version (RE4-GP I think)? You obviously have a large enough budget if you're talking about dual redundent systems. While I haven't ever done an array on the scale you're asking about, the rebuild time of a 44TB array I would think would be weeks.

Nitro, weren't the array's that were taking days to rebuild reasonably smaller than 44TB?
 
I will build the array simulate a failure and see how long it takes to rebuild. Before i make op my mind.
We are 2 guys in this project so one will be at my place and the other at his place. I have the drives that's why I'm not going with RE4 or other drives that are better for RAID.

Im waiting on the controller card to start testing just wanted to get some inputs before. I still believe that 3x hard disk failure is unlikely Especially when it has to happen twice at once. but maybe Im wrong. I guess it all depends on the rebuild time, if its more then 2-3 days maybe i will need to make 2 arrays.

Is there a easy way to put the 2 arrays back in to one partion? OS will be Vista, Win7 or 2003 server.

nitro what is that JBODn can't find much on Google? JBOD (Just a bunch of discs) can that be used? I guess it would be something to do on the raid card right?

It will be a Media server/backup of desktops/laptops with a lot of extra space :)
 
Freak1 said:
I will build the array simulate a failure and see how long it takes to rebuild. Before i make op my mind.
We are 2 guys in this project so one will be at my place and the other at his place. I have the drives that's why I'm not going with RE4 or other drives that are better for RAID.

Im waiting on the controller card to start testing just wanted to get some inputs before. I still believe that 3x hard disk failure is unlikely Especially when it has to happen twice at once. but maybe Im wrong. I guess it all depends on the rebuild time, if its more then 2-3 days maybe i will need to make 2 arrays.

Is there a easy way to put the 2 arrays back in to one partion? OS will be Vista, Win7 or 2003 server.

nitro what is that JBODn can't find much on Google? JBOD (Just a bunch of discs) can that be used? I guess it would be something to do on the raid card right?

It will be a Media server/backup of desktops/laptops with a lot of extra space :)
Click to expand...


Yesw JBOD is what I was refferring to.

JBODn = the act of doing JBOD :)

d00d two or three days??? A 4+1P 5 Drive WD20EADS RAID 5 will take prob take close to two days.

If I were a betting man....which I am....I would say that it will take 12-18 days for a rebuild like that.
 
if its just a media back up server why are you going through so much trouble? raid and everything seems a bit drastic for such a simple purpose doesn't it?
Posted via [H] Mobile Device
 
WHS with DFS-R??

i dont know if that would work, but it might
 
nitrobass24 said:
Yesw JBOD is what I was refferring to.

JBODn = the act of doing JBOD :)

d00d two or three days??? A 4+1P 5 Drive WD20EADS RAID 5 will take prob take close to two days.

If I were a betting man....which I am....I would say that it will take 12-18 days for a rebuild like that.
Click to expand...

?

This thread benches write at least 200MB/s+ which is ~ 2.7 days. For it to take 12 days it'd have to rebuilding at 45MB/s....
 
Well according to your stats of 200MB/s it would actually take 1.45 days....but thats neither here nor there.

When an array is in a degraded state it will not perform as it would under a normal state
Yes you are still running with your data intact, but at a much slower pace.

Not to mention the fact that he will be using GP drives which are slower, and also will be have 20TB to rebuild parity for.....it could take a while.
 
nitrobass24 said:
Well according to your stats of 200MB/s it would actually take 1.45 days....but thats neither here nor there.

When an array is in a degraded state it will not perform as it would under a normal state
Yes you are still running with your data intact, but at a much slower pace.

Not to mention the fact that he will be using GP drives which are slower, and also will be have 20TB to rebuild parity for.....it could take a while.
Click to expand...

Why not just admit you were just making stuff up with 12 days instead of criticizing the numbers I came up with? (http://www.google.com/search?hl=en&...:official&hs=rzS&q=44TB/200MB/s&aq=f&oq=&aqi=)

200MB/s is worst case.. most benches I've seen are more like 500MB/s with good drives. My software raid can manage 80MB/s resync... with 3 disks. You don't think hardware raid would manage more than 45MB/s?

I agree that raid6 using all those disks _is_ a bad idea, but not because resync would take 12 days.
 
Rebuild times is Hardly the biggest issue here.

WD Green drives on a iop341 controller is. Nor are the normal gp drives on the HCL for that controller. So there is no support behind it. GP drives are -NOT- made to be run in raid.
 
What are your performance requirements for this whole system?

I ask because I use unRAID for my backup server and have been very pleased by it. In my instance, I need parity protection, did not want data striping, needed the ability to add drives as needed, be able to mix hard drive sizes, and the ability to turn off drives that are not in use.

In my usage scenario unRAID performs very well. Writes to my array aren't stellar but are okay at about ~50MBps to the cache drive (when writing directly to the array write speeds are only about 18-25MBps) . Reads from the array depend upon the drive it is using (my parity checks will usually run around 60-90MBps.) Also, since it is not RAID in the traditional sense, I have not been affected by TLER issues that plauge some of the WD consumer grade drive. In regards to data recovery, rebuilding my array will not take forever in the event of a drive failure and IF I lose a second drive before I can rebuild the array, I've only lost the data on those drives, not the entire array. Overall, it's not the fastest storage subsystem for reads and writes, but it provides parity for my data, reasonable recovery times for large arrays, the ability to upgrade disks as needed, and if God strikes down multiple drives I'm not completely hosed.
 
It will take ~50 hours to rebuild the array. How do I know? I've done it with a very similar setup.
 
Nacho said:
Rebuild times is Hardly the biggest issue here.

WD Green drives on a iop341 controller is. Nor are the normal gp drives on the HCL for that controller. So there is no support behind it. GP drives are -NOT- made to be run in raid.
Click to expand...

They have added the RE4-GPs to the HCL.

If you enable TLER it would prob work, but if you ever have a problem dont call Highpoint....they wont help you because you are running an unsupported array.
 
Blue Fox- That i think is acceptable. At my work we have a 3 year old server it took 3 days just to add 1 harddisk (140 GB SCSI) to a Raid 5.

Nacho- I would really want to use the drives that i already have and not go buying 24 new enterprice discs. With the TLER tool what is wrong in using the Green drives?

Greatone123- Is there a better way?
 
me? i wouldn't know, im not saying what ur doing isn't the 'right' way like some. I'm just saying it seems like a lot of work and time will be spent.


WHS has spoiled many with ease of use, setup, and better sense of security then most have had in the past.


how much data are storing? u said a lot of extra space.


how are u going to back all that up to the other location?
Posted via [H] Mobile Device
 
That is a lot of space for one array and we all just want you to be aware of the pros and cons as you asked. Since we do get a lot of questions concerning RAID and backups, we just want everyone in the world to know that RAID is not a backup and to carefully evaluate the practical uses of implementing Large Storage Arrays... and the resources that go with it.

Yes you can have one big storage array of all 24 hard drives instead of 2-3 smaller arrays.
Yes you will only experience array failure only if 3 drives fail in the same time period which is not likely. Yes it will take days, possibly weeks to build/rebuild. And yes you will cry when that happens.
But ask any experienced admin who has watched his array die because a 2nd drive dies while rebuilding a degraded array. Or seen all his seagate drives experience issues all at once. Or watch all his IBMs start clicking. or watch the controller card suddenly hiccup and force a rebuild which kills several drives. Or watch one wire on his PSU hiccup and drop all the drives in that row and kill the array. Ask any experienced admin how much time and effort and work they had to do to get everything up and running.
Most admins prefer running several small arrays because it puts less stress on the other drives when one drive/array goes down. I also prefer not losing all my data when a drive array dies and would prefer having 1 small array dead vs 1 large array dead. Drives fail exponentially, like a bad apple in a barrel. Plan for the worst case scenario and you'll be glad when it happens.

Yes it maybe a .00001% chance of happening and it may run fine for 4 years and 364 days. But on the 1 day that something goes wrong, how much time/effort is it going to take you to fix it? And are you ok with the risk and the time/effort to get it working?

I remember it took 3 days to rebuild my 2TB raid5 array of 4 x 500GB drives... and about 4 days for my 4 x 1TB array. I don't remember how long for my 12 x 1TB RAID6 array, but I remember chking on it after 4 days and it was still rebuilding... I started using it a week after it was setup.
 
That many drives on one array, just seems like a lot of risk. Gonna bookmark this thread so when you post the "My huge raid 5 array crashed, how do I recover it" thread I have this one handy to redirect you to.:D
 
jay2472000 said:
That many drives on one array, just seems like a lot of risk. Gonna bookmark this thread so when you post the "My huge raid 5 array crashed, how do I recover it" thread I have this one handy to redirect you to.:D
Click to expand...

QFT
 
longblock454 I already read your thread, i sounds very good. Do you have a time on a rebuild in Raid6?

jay2472000 Actually its Raid 6 ;-)

Greatone123 By external harddrives. Only transporting changes. I have looked into WHS but it does not support RAID and as far as i know it uses twice the space as evey fine is stored in 2 places.
 
Freak1 said:
longblock454 I already read your thread, i sounds very good. Do you have a time on a rebuild in Raid6?
Click to expand...

I never pulled a drive to test, but I'd expect something similar to a background init, depending on how hard your hitting during a rebuild, maybe 10-12 hours on a 13TB array.
 
Freak1 said:
I have looked into WHS but it does not support RAID and as far as i know it uses twice the space as every fine is stored in 2 places.
Click to expand...

yea WHS works like raid1 when it comes to data backup, but you will be having the same amount of hdds with double the servers and they will be still more pround to loose data + you dont need to mirror everything on WHS, just the most crucial data..

if I were in your place i would def go with whs or unraid.. raid is enterprise solution and not home backup/media server option
 
This conversation really ends up being the people that have had a raid fail vs. those that haven't. I personally can say that the only time I am ever going to use a raid subsystem again will be for backing up an important system drive, or for performance reasons. Raid-5 systems worked fine for me for years, but I still have nightmares of the day that the array decided it didn't want to play nice.
 
With that many disks Id suggest 4 arrays of 6 drives in R5. Gives you 40TB of space (pre formatting) and if you lose a drive its much faster to rebuild (and if you lose a second on that array you lose 10TB instead of 20TB, much easier to restore, how many people have 20TB of drives lying around to physically transport to the failed array location?).

Regardless, make sure your power system is up to scratch. Thats quite a bit of power and heat being dumped into the room.

You can assign an empty folder to be the root of each array, but that folder's size will be limited to the size of the array (files dont continue from one to the next). One folder per drive/array in C: gives you easy access.
 
GLSauron said:
With that many disks Id suggest 4 arrays of 6 drives in R5. Gives you 40TB of space (pre formatting) and if you lose a drive its much faster to rebuild (and if you lose a second on that array you lose 10TB instead of 20TB, much easier to restore, how many people have 20TB of drives lying around to physically transport to the failed array location?).

Regardless, make sure your power system is up to scratch. Thats quite a bit of power and heat being dumped into the room.

You can assign an empty folder to be the root of each array, but that folder's size will be limited to the size of the array (files dont continue from one to the next). One folder per drive/array in C: gives you easy access.
Click to expand...

This sounds strikingly similar to something I suggested ;)
O yea but I dont know what I'm talking about:p
 
The issue with a 24 drive array of 2TB discs is that there is about a 1 in 3 chance that there will be a read error somewhere in the data if you were to read the entire array using the drives specified. During a rebuild, those are pretty bad odds, and you don't have multiple parity bits to fall back on at that point. The question then is what does your raid do at the point of the error. Does it just make a mistake and corrupt the file that may be at that point, or does it halt entirely and tell you it can't rebuilt the array?! This becomes much less of an issue if you break it into 2 or 3 arrays, all using RAID6. I have Areca controllers myself, and I tend to split arrays into 2 arrays of 12 drives each using RAID6.

All of the above are just based on the statistics and chances of unrecoverable bits leading to errors or unrecoverable arrays. There will be bit errors with that many drives at that size, so you want to plan accordingly. Don't forget, that when a drive fails in your array, and one will fail at some point, there are also the factors about why it failed to think of. It may be an event that leads to multiple drive failures. Think about not being home when the machine is on and the home air conditioner gives out, and just by chance that last software patch round broke your nifty emergency shutdown script or causes it to hang, letting the drives simply kill themselves with heat. This is just an imaginary example, but my point is don't underestimate Murphy's Law to ruin your day!
 
Last edited:
madcoder 1 in 3 chance for a read error, what are you basing this on? And just for the record i have tryed loosing an entire RAID 5 array losing a lot of data because it got corrupted for some reason. And i am aware of the risks that's is why I'm glad all data will be stored in 2 places. No it won't be funny or easy to recover the data but it is possible.

I think you guys have convinced me to split it up in 2x12 in RAID 6.

I have a 1000Watt power supply I'm planning on using. And I will also be getting a UPS. How many Watts should it be? And a battery for the controller. OS will be on discs in RAID 1. The case will be well ventilated, standing in a rather big and not to hot room.
 
You must log in or register to reply here.
Back
Top