I work for a photo studio, and we would like to catalog our photos on a central file server. I am looking to build a server with 40TB worth of storage using 2tb black Caviar drives. What sata controller for you recommend, and which raid should i use, in order to assure data protection. I hear that raid5 is not ideal for that much storage. Thanks for your help and advice.
![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
-
A Great friend to the HardForum with a great kid that he is trying to get a scholorship to continue his schooling. Please give hime a vote! Only 24 hours left! Thanks.
If you have an VOTE FOR KEENAN!
Building a 40 TB File server. Advice?
- Thread starter nikita
- Start date
Why 2TB Black drives? How many users total? How many concurrent users? What do the access patterns look like? What are budget constraints? What are size/ power/ noise constraints (if any)? What is the network solution that this will be plugged into? How much redundancy is needed? What will the physical server access look like (e.g. someone is around 24x7 and it is in your office or is it in a datacenter somewhere with <7x24 access)? What OS will the server run? What OS will the clients run? Is it only for storing folders or will there be other tasks for the machine?
Just some questions since you probably need to do a bit of research/ reading before asking for advice. General questions get you answers that are not useful.
Just some questions since you probably need to do a bit of research/ reading before asking for advice. General questions get you answers that are not useful.
Krazypoloc
Gawd
- Joined
- Jul 23, 2010
- Messages
- 759
If you have to ask the questions that you have in your original post then you are not ready to undertake such a task. Just throwing a 40TB server together that will be reliable and highly fault tolerant is not as easy as someone rattling off a few part numbers for you. I would do a LOT more research.
Start here.
And give fellow [H]ers more than 84 min to respond before wondering why no one's responding...
Also be nice to know the expected budget minus HDs. You can put together some budget servers as a whole (including almost all of the drives) for practically the price of some single RAID controllers alone, as long as you don't need the mind-numbing performance.
It's only a photo studio, after all. If all that's needed is mass live storage, IMO I wouldn't bother with a hardware RAID card.
TheGreySpectre
Limp Gawd
- Joined
- May 1, 2005
- Messages
- 439
If you are doing this for a buisness I would not use WD black drives; I would use drives rated for RAID application, if you want to use Western digital drives I would use their RE-4 drives. If you are deadset on using consumer level drives then there has been a lot of experience using the 7k2000 hitatchi drives in combination with areca 1680 and 1880 cards. Also consider what operating system you are going to run all of this on and if you have the staff to manage it. We need more information, especially a budget and user quantity.
I have to ask: How did you suddenly wind up with 40TB of data and no place to put it?
---
I would suggest having your originals and 2 backups. At least one backup off site.
RAID might be useful for "recent" projects. But not so much for projects that are finished.
---
---
I would suggest having your originals and 2 backups. At least one backup off site.
RAID might be useful for "recent" projects. But not so much for projects that are finished.
---
You've clearly never dealt with them then. Have fun when your drives start dropping out of the array. If you actually looked into Google's servers, you'd understand why they do what they do.
Is Google using them on a RAID controller?
Answer: Nope
Hence the reason people recommend RAID edition drives. Plus, when it is your job on the line if you lose 40TB of digital assets, a good answer is usually not "But I saved us $2,000 when losing those assets."
Not using a drive with appropriate firmware does increase the likelihood of data loss on RAID controllers.
I've setup and manage a number of RAID setups using consumer level drives that support TLER/ERC/etc (probably should have mentioned that bit). My failure rates haven't been any higher than what Google reports seeing (but then again, ~50 drive sample doesn't really compare to Google's sample of 100,000 drives).
Stereodude
2[H]4U
- Joined
- Oct 20, 2000
- Messages
- 3,285
Even if you don't want to use SATA drives sold for RAID, WD would be the last company on my list for non-RAID drives. Their non-RAID drives don't like to work in RAID whereas non-RAID drives from Hitachi & Samsung have no problems working in RAID.
Was this meant to be funny? My first reaction was to LOL, but then I realized you might be trying to be serious?
Eh, who cares, LOL however you actually meant it...
Would you prefer to make a more constructive statement? This has been my experience. If you've had another, I'd like to hear it (as would the OP, I'm sure).
I think the tone of his post was perfectly appropriate, given that you came into the thread calling "nonsense" about something that actually makes a lot of sense -- companies using RAID-certified HDDs in their RAIDs. Specifically, if a company is building a RAID with hardware RAID cards, in most cases they want to either use an HDD on the RAID card manufacturers HCL or at least a RAID-edition HDD.
Your link to google, which has a completely different storage setup than the OP here mentioned, is more deserving of the comment "nonsense". Ditto for your implication that your having worked with a sample of 50 HDDs is persuasive evidence that companies do not need RAID edition HDDs.
Your link to google, which has a completely different storage setup than the OP here mentioned, is more deserving of the comment "nonsense". Ditto for your implication that your having worked with a sample of 50 HDDs is persuasive evidence that companies do not need RAID edition HDDs.
He has experience using ~50 drives in "a number of RAID setups". Now, let's assume that even "a number" is only four, I'd go as low as three, but three is typically "a few".
That's coming out to roughly four arrays of 12 drives. So, the biggest array he has professional experience with is a 12-drive array.
Simply put, like a lot of us here, I have built personal arrays at home that are 12+ drives big. And I have professional experience with a single array that consisted of more than 50 drives.
I also currently own probably a hundred HDs would be a quick guess. While I admit that stat honestly is irrelevant, it's still more interesting personally to myself that I have more personal experience with HDs as a hobby than he has experience with HDs as a professional.
But I also once built a single machine at home with 32 IDE drives. Three 3ware controllers: one 8-port, two 12-port. So, yeah, a sample of 50 in total to me can mean as little as two machines.
Last edited:
Aye, my apologies. In context of the post, it's definitely horrible advice, especially for mission critical systems. I took TheGreySpecter's post out of the context when responding. I'll try to be more awake (and less stupid) next time I post.
(And yes, I've got experience with 'only' ~50 drives spread across a small number of RAID arrays. I'm primarily a software developer. Don't do much much in the way of IT work and the arrays were built on the cheap more for rather specific development/testing needs rather than anything mission critical. They do get used rather heavily, and haven't had any issues yet.)
(And yes, I've got experience with 'only' ~50 drives spread across a small number of RAID arrays. I'm primarily a software developer. Don't do much much in the way of IT work and the arrays were built on the cheap more for rather specific development/testing needs rather than anything mission critical. They do get used rather heavily, and haven't had any issues yet.)
brutalizer
[H]ard|Gawd
- Joined
- Oct 23, 2010
- Messages
- 1,603
Here is a guy building a 40TB raid.
http://hardforum.com/showthread.php?t=1572188
http://hardforum.com/showthread.php?t=1572188
@OP
Like others have said, do some research first, then come back with some more questions. We'll be happy to help. Additionally, as far as consumer-level drives are concerned, don't bother putting Western Digital's Caviar series in a hardware RAID array; they don't play nice.
A few questions for you:
1) What operating system do you plan to run on this file server? Depending on which OS you choose, you can use a software RAID solution.
2) What O.S. runs on your client machines? (Windows/Mac/Ubuntu)
3) What is your backup and recovery strategy? Where are you going to back up 40TB of data?
@thread
In and of itself, using consumer level drives is not inherently bad AS LONG AS you have the appropriate infrastructure to support such a policy. I've had Western Digital RAID Edition drives literally drop dead and refuse to spin up - and these drives saw hardly any action. By contrast, I've had Western Digital and Hitachi consumer-level drives that have been in 24x7 operation for the past 5 years, and don't have a single SMART error.
Hard drives die. It happens, and you deal with it accordingly. For me personally, enterprise-level drives are worth buying primarily for their better behaviour and compatibility with hardware RAID controllers and their longer warranties which is why I use them for my business.
I certainly don't buy them because I think they're more reliable.
Like others have said, do some research first, then come back with some more questions. We'll be happy to help. Additionally, as far as consumer-level drives are concerned, don't bother putting Western Digital's Caviar series in a hardware RAID array; they don't play nice.
A few questions for you:
1) What operating system do you plan to run on this file server? Depending on which OS you choose, you can use a software RAID solution.
2) What O.S. runs on your client machines? (Windows/Mac/Ubuntu)
3) What is your backup and recovery strategy? Where are you going to back up 40TB of data?
@thread
In and of itself, using consumer level drives is not inherently bad AS LONG AS you have the appropriate infrastructure to support such a policy. I've had Western Digital RAID Edition drives literally drop dead and refuse to spin up - and these drives saw hardly any action. By contrast, I've had Western Digital and Hitachi consumer-level drives that have been in 24x7 operation for the past 5 years, and don't have a single SMART error.
Hard drives die. It happens, and you deal with it accordingly. For me personally, enterprise-level drives are worth buying primarily for their better behaviour and compatibility with hardware RAID controllers and their longer warranties which is why I use them for my business.
I certainly don't buy them because I think they're more reliable.
Rhinohelix
Limp Gawd
- Joined
- Jul 7, 2004
- Messages
- 218
For something like this, why not just buy something off the shelf from Dell or HP? Talk to NetApp and EMC but in the end, at least with one of these companies you have some hardware support in both the configuration and maintenance.
For a personal/small business I guess you could do that but any money you spend (other than the higher end support contracts) is going to be money very well spent.
For a personal/small business I guess you could do that but any money you spend (other than the higher end support contracts) is going to be money very well spent.
Hunterzyph
Limp Gawd
- Joined
- Oct 23, 2005
- Messages
- 159
Sounds to me like this is a small shop with high data storage needs, whether simply the result of anticipated retention, or because they have a hodge-podge setup now....and that they have a limited budget.
All told, I heartily agree with all of the posts here saying 'need more information'. Silly to post here without a budget in the least.
All told, I heartily agree with all of the posts here saying 'need more information'. Silly to post here without a budget in the least.
First of thank you so much for many helpful responses. I really appreciate that. Let me give you a little back ground to clear things up. I have a photo studio that need to have an ability to catalog and view and search through their archive. They would like to have a Digital Asset Management System
. They would like to have 40 Tb storage available to them. And they would like to create backups using blue ray disks.
here is what they already purchased: ASUS Z8NA-D6C Dual LGA 1366, 2 Intel Xeon E5620 Westmere 2.4GHz LGA 1366 80W Quad-Core Server Processor, NORCO RPC-4020 4U Rackmount Server Case, Crucial 8GB (2 x 4GB) 240-Pin DDR3 SDRAM ECC Registered.
Now what we need to figure out is what kind of drives with what kind of controller and what kind of raid would you recommend for this setup? Data integrity is the top priority, speed is secondary.
Thanks again for all your input.
. They would like to have 40 Tb storage available to them. And they would like to create backups using blue ray disks.here is what they already purchased: ASUS Z8NA-D6C Dual LGA 1366, 2 Intel Xeon E5620 Westmere 2.4GHz LGA 1366 80W Quad-Core Server Processor, NORCO RPC-4020 4U Rackmount Server Case, Crucial 8GB (2 x 4GB) 240-Pin DDR3 SDRAM ECC Registered.
Now what we need to figure out is what kind of drives with what kind of controller and what kind of raid would you recommend for this setup? Data integrity is the top priority, speed is secondary.
Thanks again for all your input.
If data integrity is the top priority as you say, then you really ought consider something like ZFS. NexentaStor is a good choice for a bulletproof O.S. (Solaris) incorporating ZFS. Failing that, unRAID looks like a good choice.
However, if Windows is entrenched, a combination of a decent hardware RAID card, enterprise-class hard drives and weekly array patrols should be good enough.
For the card, I'd look at brands such as LSI and Areca. For hard drives I'd say Hitachi Ultrastar and Western Digital RAID Edition GP.
A software alternative for Windows would be something like FlexRAID, which deals in files rather than devices. More information can be found in the forums.
However, if Windows is entrenched, a combination of a decent hardware RAID card, enterprise-class hard drives and weekly array patrols should be good enough.
For the card, I'd look at brands such as LSI and Areca. For hard drives I'd say Hitachi Ultrastar and Western Digital RAID Edition GP.
A software alternative for Windows would be something like FlexRAID, which deals in files rather than devices. More information can be found in the forums.
Thanks PartyBoy, I am going with windows, since the cataloging software is going to run on it. since i already have a case and i am limited to 20 drives max, what do you think is the raid i should go with in order to get me close to 40 tb. Stability and data integrity are very important, so let me know if you think this will not work. What card specifically would you recommend.
drescherjm
[H]F Junkie
- Joined
- Nov 19, 2008
- Messages
- 14,933
Do you currently have an LTO4 or LTO5 tape drive or archive?
drescherjm
[H]F Junkie
- Joined
- Nov 19, 2008
- Messages
- 14,933
That will be expensive, unreliable and time consuming. Remember that with any good backup plan you need at least 2 full backups. Think of all the time needed to burn several thousand BD-Rs..
Last edited:
No, we are looking to do it on blue ray disks, based on monthly additions
the backups would be for data that was added over the last month. not a FULL backup. I would like to rely on raid for data integrity.
drescherjm
[H]F Junkie
- Joined
- Nov 19, 2008
- Messages
- 14,933
To me that is a scary thought for this amount of data. Make sure you divide the disks into 2 arrays of raid 6 or use raid60.
drescherjm
[H]F Junkie
- Joined
- Nov 19, 2008
- Messages
- 14,933
Make sure they backup the added stuff at least 2 times to 2 different BD-Rs. Also remember that the disks will degrade over time so do not expect to be able to read them in 10 years. Although by that time the array would have been replaced anyways since consumer hard drives do not last much more than 5 years anyways.
BTW, I do have 40TB of raid myself here at work. For me this is in 6 servers bought over the last 7 years. The latest addition (done this month) was swapping out 9 500GB drives for 9 2TB drives to create a 14TB raid 6 + spare.
You should outsource this job to someone with accountability. Unless you're a very small business, it's not worth saving a few bucks while leaving yourself completely accountable because you decided to take advice from a public forum. If something goes wrong, it's your ass on the line.
I don't know what business time is worth, but backing up to hard drives seems a much better option. Backing up 2TB costs $100 for hard drives plus a small amount of labor time versus the labor time to change 40 or so blue ray disks plus a small amount for disks - maybe not so small amount.
I would keep most of the data off line. If you need regular access to old files, put them in E-sata external cases.
I don't think a large RAID is going to do much for you. You might copy the files you need to a small local RAID for processing and then back them up to hard drives when done.
---
Multiple backup copies are great ideas. If the data is valuable, rent a safe deposit box at a local bank (about $25/year).
I would keep most of the data off line. If you need regular access to old files, put them in E-sata external cases.
I don't think a large RAID is going to do much for you. You might copy the files you need to a small local RAID for processing and then back them up to hard drives when done.
---
Multiple backup copies are great ideas. If the data is valuable, rent a safe deposit box at a local bank (about $25/year).