Recommendations for 20TB RAID 6 system

Sgt Rock

Limp Gawd
Joined
Jul 1, 2004
Messages
190
Looking to build a NAS with about 20TB usable in RAID 6. I am looking to spend about 3k give or take. What I would like to get from this forum is hardware options as well as software. This will be a central media server for the my house for music/movies/pictures, etc. It will all be running on a 1 Gb backbone to all systems (about 7) in the premise.

Thanks for help.
 
What is your expected usage? Large media files? Databases? Virtual machines? Where are you going to store and run it? Acceptable noise level? How many machines are going to be accessing it simultaneously and in what ways? Are you going to run anything on the box, or is it purely a file share?

Give us some more info and we can point you in some directions.
 
With that much data, I'd consider dropping hardware RAID for ZFS. Check out _Gea's thread (http://hardforum.com/showthread.php?t=1573272) for a basic rundown of what it can offer, and there is a good idea of stable hardware to use too. You'll also save money not buying expensive RAID cards too and blow it on more hard drives. General hardware though:
- Norco
- Supermicro MB
- Xeon, low power
- Lots of ECC ram
- IBM M1015
- Hard drives
 
With that much data, I'd consider dropping hardware RAID for ZFS. Check out _Gea's thread (http://hardforum.com/showthread.php?t=1573272) for a basic rundown of what it can offer, and there is a good idea of stable hardware to use too. You'll also save money not buying expensive RAID cards too and blow it on more hard drives. General hardware though:
- Norco
- Supermicro MB
- Xeon, low power
- Lots of ECC ram
- IBM M1015
- Hard drives

Considering that 10x3TB (~21 TiB formatted) drives he will need will leave him with ~$1200, that really is his only route. If he drops to 2TB drives and builds one pool now and another pool later...costs become more reasonable.
 
With that much data, I'd consider dropping hardware RAID for ZFS

ZFS is a terrible choice for a media server. It cannot expand a RAIDZ vdev at all. If you start with, say, a 5-drive RAIDZ2 vdev, then if you want to expand later, you have little choice except expanding a zpool by another 5 or more drives at a time. And each vdev must use drives of the same size, or the extra space is wasted.

With Snapshot RAID, you can mix drives of different capacities, utilizing all the available space, and you can add additional drives of any size, 1 at a time.
 
SnapRAID looks like a good fit for my files, now as for the hardware with the 3TB drives already in the mix what type of hardware would you recommend? As this will be delivering music to the whole house SONOS system and sending full TrueHD, 1080p to a theater room and up to 6 other bedrooms (more like 2 concurrent at a time unless the in-laws are in town). Again thanks for the time and thought processes going in to your recommendations.
 
ZFS is a terrible choice for a media server. It cannot expand a RAIDZ vdev at all. If you start with, say, a 5-drive RAIDZ2 vdev, then if you want to expand later, you have little choice except expanding a zpool by another 5 or more drives at a time. And each vdev must use drives of the same size, or the extra space is wasted.

With Snapshot RAID, you can mix drives of different capacities, utilizing all the available space, and you can add additional drives of any size, 1 at a time.

so build another raidz(2/3) with whatever new disks you buy and export a new nfs/cifs mount point.

at some point you can replace the disks in the original with larger disks and resilver. or just mirror. usable space kind of sucks but w/e you can expand with as few as 2 disks.
 
SnapRAID looks like a good fit for my files, now as for the hardware with the 3TB drives already in the mix what type of hardware would you recommend? As this will be delivering music to the whole house SONOS system and sending full TrueHD, 1080p to a theater room and up to 6 other bedrooms (more like 2 concurrent at a time unless the in-laws are in town).

There is a lot of hardware that would be capable of doing that.

I suggest a Norco hot swap chassis like the 4224, which would give you plenty of room for expansion (24 HDD bays).

I like Supermicro server motherboards for servers, just about any SM MB should be fine for your server. You can also get one with built-in mini-SAS ports. But I prefer discrete HBA cards, so if you go that route, make sure your MB has enough PCIe slots for the number of HBAs you need. I like the IBM M1015 HBAs, which are 8-ports each and can often be found on ebay for less than $100. You could go with a SAS expander, but I have found that a few discrete HBAs are easier to deal with and more reliable than dealing with an expander.

For the HDDs, I've been buying Hitachi recently. It is a tough choice, because the Hitachi HDDs have always had a high return rate, but they still come with a 3 year warranty, which is better than most Seagate and WD models. I strongly suggest 5400rpm model since the 7200rpm use more power, generate more heat, and appear less reliable. You don't need 7200rpm for bulk media storage.

http://www.hardware.fr/articles/862-6/disques-durs.html
 
Last edited:
ZFS is a terrible choice for a media server. It cannot expand a RAIDZ vdev at all. If you start with, say, a 5-drive RAIDZ2 vdev, then if you want to expand later, you have little choice except expanding a zpool by another 5 or more drives at a time. And each vdev must use drives of the same size, or the extra space is wasted.

With Snapshot RAID, you can mix drives of different capacities, utilizing all the available space, and you can add additional drives of any size, 1 at a time.

ZFS is a fine solution for a media server. Many people on this board are using it for that purpose currently. You don't have to expand a VDEV. You can expand a pool by adding another VDEV to it. Drives make up VDEVs, VDEVs make up pools. Also you probably wouldn't use Z2 for 5 drives. You could but it'd be odd. Snapshot RAID does not provide real time protection. If you haven't synced and something happens you can lose data. It also doesn't preserve all file attributes, permissions for example. There is also no performance gain by pooling drives. As long as you are aware of its specific limitations it could be useful. But I don't know why you'd trade real time protection and performance gains if you didn't have to. ZFS also has significant talent coding and updating it. Including a huge contract from Lawrence Livermore National Labs for their 55 petabyte array for their new supercomputer. It is well established and tested.
 
ZFS is a fine solution for a media server. Many people on this board are using it for that purpose currently. You don't have to expand a VDEV. You can expand a pool by adding another VDEV to it. Drives make up VDEVs, VDEVs make up pools. Also you probably wouldn't use Z2 for 5 drives.

In which case you would only be able to expand by MORE than 5 drives at a time. Which is crazy. A media server should be able to easily expand by 1 drive at a time. ZFS cannot do that with RAIDZ. That is why ZFS is a terrible choice for a media server.
 
ZFS is a terrible choice for a media server. It cannot expand a RAIDZ vdev at all. If you start with, say, a 5-drive RAIDZ2 vdev, then if you want to expand later, you have little choice except expanding a zpool by another 5 or more drives at a time. And each vdev must use drives of the same size, or the extra space is wasted.

With Snapshot RAID, you can mix drives of different capacities, utilizing all the available space, and you can add additional drives of any size, 1 at a time.

I think "terrible" is a bit strong....

With the current release of can't add a disk to a 5 x raidz2 and make it a 6 x raidz2, but you can add a disk at a time (5 x raidz2 + 1(no raid)) or 2 at a time (5 x raidz2 + 2 (mirror)) or even more (5 x raidz2 + 3 x raidz)... you get the idea. When you do this they will still act as a single volume. You can also do single disk replacements of your array and realize the new size after all have been replaced (or hook them all up and do a zfs send).

ZFS does offers exceptional data integrity, a built in volume manager, snapshots (especially useful for backups), is open source, and a whole pile of other features. Personally, I will always hesitate to trust my data to propriety payware, even if it has a good track record.

It is "a" choice, and maybe an "exceptional" choice, depending on the OSes you comfortable with, strategy for future expansion, and comfort level with dependance on payware.
 
With the current release of can't add a disk to a 5 x raidz2 and make it a 6 x raidz2, but you can add a disk at a time (5 x raidz2 + 1(no raid)) or 2 at a time (5 x raidz2 + 2 (mirror)) or even more (5 x raidz2 + 3 x raidz)... you get the idea. When you do this they will still act as a single volume. You can also do single disk replacements of your array and realize the new size after all have been replaced (or hook them all up and do a zfs send).

Only a crazy person would mix a RAIDZ2 vdev with a single disk vdev in a zpool. It makes no sense to protect some of your media files with dual parity and other media files with no redundancy or just mirroring.

Your list of ZFS advantages is a joke, right?

You have my sympathy for making a bad choice and being stuck with it since it is much more difficult to migrate away from ZFS. But you really should not give others bad advice just because you are in denial or have a perverse desire to make others share your pain.

ZFS is a terrible choice for a media server.
 
Last edited:
I prefer to give people a fair explanation of advantages and the disadvantages and allowing them to make choices based on their situation. I do not scare them toward the decisions I made with inflammatory language. It will be the end-user's choice, and I will be happy if they are content with their informed decision. My comment is only to facilitate this decision making.
 
This may be not what Your looking for, but what the hell.

SuperMicro MB
Norco case
LSI HBA
HP Expander

Your choice of a Linux distro and use MDADM to manage the raid on a XFS filesystem.

Manage it all with Webmin or command line.

Like I said, may not be right for You, but works very well for me so far.

0fece862.jpg
 
ZFS is a terrible choice for a media server. It cannot expand a RAIDZ vdev at all. If you start with, say, a 5-drive RAIDZ2 vdev, then if you want to expand later, you have little choice except expanding a zpool by another 5 or more drives at a time. And each vdev must use drives of the same size, or the extra space is wasted.

With Snapshot RAID, you can mix drives of different capacities, utilizing all the available space, and you can add additional drives of any size, 1 at a time.

bad advice
 
In which case you would only be able to expand by MORE than 5 drives at a time. Which is crazy. A media server should be able to easily expand by 1 drive at a time. ZFS cannot do that with RAIDZ. That is why ZFS is a terrible choice for a media server.

In some situations, ZFS would not be a good fit. In this situation though, where the OP is starting off with 20TB, when he's putting down over $1k in just drives, I don't think expansion is a problem. In addition, it's going to be difficult to have a backup plan for that much data so yeah, you pretty much need the best data protection there is.
 
I dont know about anyone else but I tend to buy my drives in "bulk", basically when I see a good price. If you need an emergency overflow use a couple of USB drives.

A good point made by fields_g is to lean towards with what you are used to - or at least make that one of your criteria in the selection choice. Do you want to learn a whole new OS and RAID system or watch movies? :)

Obviously FlexRAID appears to be a good (paid) alternative to SnapRAID if you are looking at the snapshot approach. You probably need to look at the drive pooling aspect if you go down this route.
 
In which case you would only be able to expand by MORE than 5 drives at a time. Which is crazy. A media server should be able to easily expand by 1 drive at a time. ZFS cannot do that with RAIDZ. That is why ZFS is a terrible choice for a media server.
ZFS is for Enterprise use, with large servers with 100s of disks and lot of CPU and RAM. They are interested in adding several disks at once, not one disk at a time. Not a home user filesystem, even though many use ZFS for home usage.

I also think it is bad that ZFS can not yet increase zpool capacity with one disk at a time. "Block pointer rewrite" in ZFS will allow this, but that functionality is not done yet. It works on an idle zpool, but not on a live zpool. On an idle zpool, you can add one disk at a time. This block pointer rewrite function, is under very slow development (and has been for several years).

But I have learned to live with this limitation. If I add one disk at a time, I will after some time, have 20-25 disks at 300GB-2TB. I mean, those old disks have bad capacity and low performance. They only use wattage, loud sound and low performance. I rather have one new fast 2TB disk instead of 5 slow 300GB disks.

Therefore, I have fixed the number of disks. I will use 11 disks in raidz3 (allows three disks to fail - it is only ZFS that offers this safety) and when my 2TB disks are old and slow, I will switch one disk to a 10TB disk, repair the raid and repeat the procedure. Then I will have 10TB disks. Thus, I will upgrade capacity by switching one disk at a time. Then I dont have 25 old disks. Who wants that?
 
I think everyone is to busy arguing ZFS to see what the OP's needs are.

If it is just a home media server, you probably want to keep it simple and low power consumption.

While flexraid is an option. I personally dont care for it. no live protection and can take up to 48 hours to build the snapshot...

ZFS pools are an option. it requires a bit of learning for for someone new to it. It also has expansion issues. it also spins all drives in the pool to watch a single movie.. with 20+ drives, that can be a big power bill to watch 2 movies a day. needs lots of RAM. Geared for enterprise use with high availability and fast IO.

unRAID, while not overly popular here, is pretty ideal for this usage. on the fly expansion of mixed size drives and live parity protection. it also only spins the drives you need to watch a movie. so 1 drive at a time..there is a minor learning curve in first set up. after that, it runs by itself. It can run on cheap hardware and needs very little RAM. this will save in power costs and heat.

WHS2011 plus a drive pool addon like drivebender. cheap (picked up a copy for $29 this week), is similar to the unraid pros and cons. based on windows for those that are familiar with that. cons... uses mirroring instead of striping so it wastes drive space.(a lot of wasted space in larger pools)



for hardware. i think everyone can agree.
Case: a norco 4224 or similar, (other option is a full 5.25 bay pc tower with 5-in-3's)
MB: Supermicro (almost any X7, x8 or x9 series will do. stick with server class and save yourself headaches)(an x8sil or x9scm with an I3 would be ideal)
RAM: the correct amount for for your OS. (4-8GB for flexraid, unraid or whs, all you can get for ZFS)
RAID card: not needed get an HBA like the M1015
 
unRAID, while not overly popular here, is pretty ideal for this usage. on the fly expansion of mixed size drives and live parity protection. it also only spins the drives you need to watch a movie. so 1 drive at a time..there is a minor learning curve in first set up. after that, it runs by itself. It can run on cheap hardware and needs very little RAM. this will save in power costs and heat.

WHS2011 plus a drive pool addon like drivebender. cheap (picked up a copy for $29 this week), is similar to the unraid pros and cons. based on windows for those that are familiar with that. cons... uses mirroring instead of striping so it wastes drive space.(a lot of wasted space in larger pools)

SnapRAID is a better choice than either of those.

http://snapraid.sourceforge.net/compare.html

UnRAID problems:

- Does not support dual-parity (can only do single-parity)

- Requires data migration to/from reiserfs filesystem when you switch to/from UnRAID

- Does not keep checksum data on your files

- Requires you to run a stripped-down linux distro, whereas snapraid can run on top of whatever OS you like

- Is not free and open-source / GPL

WHS problems:

- Mirroring is a waste of space (as you wrote)

- Requires data migration to/from whatever you were using before

- Does not keep checksum data on your files

- Is not free and open-source
 
SnapRAID is a better choice than either of those.

http://snapraid.sourceforge.net/compare.html

UnRAID problems:

- Does not support dual-parity (can only do single-parity)

- Requires data migration to/from reiserfs filesystem when you switch to/from UnRAID

- Does not keep checksum data on your files

- Requires you to run a stripped-down linux distro, whereas snapraid can run on top of whatever OS you like

- Is not free and open-source / GPL

WHS problems:

- Mirroring is a waste of space (as you wrote)

- Requires data migration to/from whatever you were using before

- Does not keep checksum data on your files

- Is not free and open-source

The fact you list zero negatives to SnapRAID really leaves me wondering how deep your bias goes and how you are associated with the product.
 
I'd look at a Linux distro and md raid. Fairly simple to setup and highly scalable. You can increase raid arrays live, have hot spares etc. I'd start with 2TB drives, maybe 3TB drives. Best of all, it's free as well and very robust. I've done tests on it like pull the plug on a server in the middle of large data transfers, rebuilds, resize etc and it always comes back fine. One time I lost two drives in a raid 5 array as they both failed at once after being spinned down for 5 hours during a power outage. Thinking I lost all the data on the raid array my first step was plugging in my latest backup to make sure everything is there just to keep me reassured. I then put that drive aside and tried to see if I have any ways to recover this raid. I forget what I did, I think I literally just rebooted and manually reconstructed it including the failed drives which I had managed to get to spin back up. The data was all there! Lot of it was corrupted (the VMs mostly) but it was there! It was a stressful moment but I managed to rebuild the entire array using all new drives, without ever taking it off line (other than that first crash). If I had raid 6 I would have been better off too.
 
As a person using and recommending ZFS I can only say that JoeComp's advice to look at SnapRAID for a media-server is very good and SnapRAID is a good fit.

It is open-source, does not cost you a dime and supports single-drive-expansion and basic protection via on-demand-snapshots (only usable for data that does not change very much, read media files). therefore it is really currently the best fit for media storage.

on the other hand, I would strongly recommend ZFS (for example a simple 3-drive RAID-Z1) for anything important (documents, etc.)

cheers
 
The fact you list zero negatives to SnapRAID really leaves me wondering how deep your bias goes and how you are associated with the product.

I agree, although I hope this doesn't put someone off considering the product itself. I have absolutely no idea why someone is so fanatical over a computer product. Whilst I can't be 100% certain, I don't think this is the author. For example, his compare page on his site is very honest and I think shows FlexRAID to be a better choice in some scenarios like 3 disk failures (http://snapraid.sourceforge.net/compare.html).

I have used it successfully - seems to work fine for me (ony a 7 x 2TB disks so far including 2 parity)

If someone was to read the snapraid forums there are some negatives there - for example some reported issues with growing parity files when disks become full.
 
Btw, even in a snapshot RAID setup, a hardware RAID card can be a good thing. Something like an Areca means that you can spin down disks when they aren't being used which saves you some £'s
 
The fact you list zero negatives to SnapRAID really leaves me wondering how deep your bias goes and how you are associated with the product.

You also listed zero negatives for snapraid. By your criteria, it would seem you are biased and cannot be trusted.
 
on the other hand, I would strongly recommend ZFS (for example a simple 3-drive RAID-Z1) for anything important (documents, etc.)

I agree. ZFS (or mdadm / linux software RAID) is a good choice for non-media files, by which I mean files that are frequently changed or deleted.
 
Btw, even in a snapshot RAID setup, a hardware RAID card can be a good thing. Something like an Areca means that you can spin down disks when they aren't being used which saves you some £'s

What are you talking about? HDDs can be spun down without a hardware RAID card.
 
If someone was to read the snapraid forums there are some negatives there - for example some reported issues with growing parity files when disks become full.

I believe I know the posts you are referring to. SnapRAID does not work well if your data disks are 99.99% full, or if you have hundreds of thousands of small files on a drive and the drive is 99% full, since snapraid has a little overhead in the way it divides the data into blocks. You specify a block_size in the config file (default 256KiB), and the space taken up in parity by each data file is rounded up to an integer number of blocks. If the largest data drive is the same size as the parity drive(s), then your parity file could exceed the size of the parity drive.

For a media server, the "hundreds of thousands of small files on a drive" is not usually an issue. But if you do have your data drives filled up to more than 99.9% capacity (and you don't want to shuffle files among drives to get below 99%), and your parity drive is the same size as your largest data drive, then snapraid is probably not for you.
 
Last edited:
TBH i can't recommend raidz for the non-enthusiast or non hardcore person.

Firstly for its lack of "cheap" expandability.

1) you have to ideally expand by identical vdevs.
2) ideal performance you should have your final array complete at creation. Drives added later will have uneven striping as the disks fill.
3) In addition to the loss of space from your choice of parity (1 drive for Z1, 2 drives for Z2) you also loose space from reserved Copy on Write areas.

20TB of disks with RAIDZ2 will net you a maximum usable 14TB of space. (4TB loss from parity, 2TB loss from reserved copy on write areas.)
 
you dont lose that much space. there is some reserved space but not 10% of raw capacity i forget the exact number.
 
you dont lose that much space. there is some reserved space but not 10% of raw capacity i forget the exact number.

I have never heard of any needed reservation due to copy on write.
With my napp-it I do a 10% reservation to avoid more than 90% fill rate of a pool.
But thats only to avoid massive speed degration that ZFS may suffer like any other file system. If you like, you can delete this reservation and go up to 100% (slow down performance, heavy fragmentation , not a good idea but possible - with ZFS like with any other file system)

Such a reservation allows to go up to 100% from a user's view without speed degration, because real pool fill rate was the below this setable pool reservation.
Beside ZFS reservations you can set ZFS quotas to limit max available space of a user or a ZFS dataset. So this is not a ZFS weakness but a setable feature.
 
Last edited:
TBH i can't recommend raidz for the non-enthusiast or non hardcore person.

Firstly for its lack of "cheap" expandability.

1) you have to ideally expand by identical vdevs.
2) ideal performance you should have your final array complete at creation. Drives added later will have uneven striping as the disks fill.
3) In addition to the loss of space from your choice of parity (1 drive for Z1, 2 drives for Z2) you also loose space from reserved Copy on Write areas.

20TB of disks with RAIDZ2 will net you a maximum usable 14TB of space. (4TB loss from parity, 2TB loss from reserved copy on write areas.)
Reserved copy on write areas? I have never heard about it. Do you have more information on this? Or is it hear say?
 
I noticed the reservations while copying over my data to my new Z2 (10x2TB and 30gx2 zil stripe) array.

When fully writing to the array (from 1g E and 10g E interfaces) I was noticing that the disk full counter was giving some strange readings. Notably the 14TB total that I mentioned. After some research I noticed that link I posted in my last post and it made sense.

After I filled my array with my data the limit seems to have dynamically adjusted itself back to 16TB free. So I assume it is a dynamic lock.

OS is Solaris 11 (non express) zpool version 33
 
Static lag, I think your maths might be wrong there. Looks like you lose 1/64 of the 14TB usable space, so more like 250GB ?
 
Last edited:
You also listed zero negatives for snapraid. By your criteria, it would seem you are biased and cannot be trusted.

Seems I hit a nerve. That is fine. However, I didn't sing the praises of any product (as you did). I just agreed ZFS might be a good fit when starting with a large "pool/dev" as the OP indicated. I didn't say anything was a "bad fit". However, you went down the road of "all else sucks and thus use SnapRAID". That was the only reasonable way to read your post and why I called you out on it. If you can't recongize your own biases then your ability to provide advice is greatly compromised; hardest lesson I have learned on my engineering life.

More back to the OP, can "Sgt Rock" provide a bit more info on his target goals? Is it a "build it once and be done" or is it preferred to grow over time? Is bit-rot important or can original data be recovered from other sources? How much time does the OP have to devote to the server "per week"? How much time can the OP devote the server if shit goes wrong? The fact that he is using a Sonos system does give a bit of knowledge on to the OP's mindset (the $3k budget is something he seems to have and be willing to drop all at once).
 
Sure, I must say first though that the amount of knowledge that has been transpiring back and forth on this thread has been most educational to read and I thank all of you for it.

To address your question, I have currently about 12TB of media that I wish to transfer over, and as all media collections go that will grow. I have large files (40GB video files) and small files relatively (MP3's and JPEGS). Most of the space is the video section and that is what I am concerned about the most as the rooms in the house can each stream them from source to their corresponding screen. The SONOS is nice but as the bit rate for that to each section is negligible compared to the video stream.

I have time to maintain the storage system each week but it is not something I want to worry about and baby either. I have systems that I do rsync backups to now that rotate and use off site storage for the most critical.

If things happen to go sideways, other than being a very unhappy person, I would be able to restore a significant portion of the data.

Any other questions I may assist with so that I can take advantage of the brain power in this thread please let me know.

Regards




Seems I hit a nerve. That is fine. However, I didn't sing the praises of any product (as you did). I just agreed ZFS might be a good fit when starting with a large "pool/dev" as the OP indicated. I didn't say anything was a "bad fit". However, you went down the road of "all else sucks and thus use SnapRAID". That was the only reasonable way to read your post and why I called you out on it. If you can't recongize your own biases then your ability to provide advice is greatly compromised; hardest lesson I have learned on my engineering life.

More back to the OP, can "Sgt Rock" provide a bit more info on his target goals? Is it a "build it once and be done" or is it preferred to grow over time? Is bit-rot important or can original data be recovered from other sources? How much time does the OP have to devote to the server "per week"? How much time can the OP devote the server if shit goes wrong? The fact that he is using a Sonos system does give a bit of knowledge on to the OP's mindset (the $3k budget is something he seems to have and be willing to drop all at once).
 
Sgt Rock,

One very significant question you need to address is whether you have to have the ability to increase the size of your storage by one drive at a time. If you must have this then ZFS is not for you at this time, this is being worked on but isn't available currently. If you don't need this then ZFS may be for you. Given your budget and goals, and if you follow the common recommendations on this board, you'll likely end up with a 20-24 bay case. With 3TB drives you only need 8 or so drives to get the 20TBs you wish to start with. That will leave you with 12 or more empty bays, or plenty to expand by more than one drive in the future if you need to. As long as you are willing to do multiple drives simultaneously. Deciding this will narrow your choices and help you determine which path you will take.
 
Back
Top