Home Server Filesystem

[Kelvarr]

So I know that I have asked this before, but I'll shoot off again.

I have a home Server that is now completed. 2xL5639's, 24GB ECC RAM, 9x1.5TB HDD, 2x32GB SSD.

I plan on making a home server, and I am pretty sure I have settled on Linux, either Fedora or CentOS. For the 9x1.5TB storage, I am not quite sure how I am going to fully break it up, but I know that at least 6TB of it will be for media storage.

My question is what kind of filesystem? Originally, I was going to use MDADM RAID6. Then I started reading a bit more on the forums and a couple other places about ZFS. Recently, I read a nice article (to me) about ZFS vs BTRFS. I have to say, BTRFS has me intrigued.

One of the big things I don't like (that I've read) about ZFS is adding drives to the pool. If I understand correctly, I could make a 9x1.5TB vdev and add that to the pool. But if I ever wanted to expand that storage, I would either have to make a completely separate pool, or add another 9x1.5TB vdev, correct? What would be the best way to set up 9x1.5TB RAIDZ2?

BTRFS (according to the article), seemed to handle adding drives much better, although the biggest downside is that it seems to be very experimental/unstable. What are your thoughts?

Here is a breakdown of what I want to achieve:

Main Server (Linux)
-Media Storage
-Document/Other PC storage
-Small "Miscellaneous" storage
--Linux Sandbox VM
--Windows 7 VM
--Test VM (Test NAS4FREE, FREENAS, maybe Unraid, etc), associated with Media Storage

Originally, as I said, my plan was to go with MDADM RAID6 for the Storage, and probably some kind of RAID1 for the OS. I have recently started to worry about the bitrot, and keeping my data in-tact. I know it is only media, but it would SUCK to have to re-rip everything. Hence, I should probably take advantage of the ECC in my server and run something that has protection against that, ala ZFS or the recently discovered (to me) BTRFS.

One thing that "worries" me about ZFS is the number of drives in the vdev, as I would probably only be adding 1-2 (hopefully) drives at a time to expand storage. I probably won't have enough money to purchase 4 drives or more all at the same time, hence the incremental upgrading.
The other thing is that ZFS has all drives spinning all the time. I know Unraid only powers up the drives it needs, and I will spend the money if I need to, but I would rather not spend that much. I have already dumped too much money into the server, and another amount on the OS when free alternatives exist might push my wallet over the edge.

Let me know if I can answer any other questions. Thanks in advance!

 

[drescherjm]

BTRFS (according to the article), seemed to handle adding drives much better, although the biggest downside is that it seems to be very experimental/unstable. What are your thoughts?

The biggest problems (from a person who has on the order of 30TB of data on BTRFS filesystems between home and work) is that the filesystem can hang for long periods of time while (several minutes) mounting when it does not like what is in the space cache and also when deleting a large subvolume. That is on top of me not trusting the raid5/6 yet so I have to put btrfs on top of mdadm raid 6.

 

[iroc409]

For ZFS, you wouldn't have to add another 9-drive vdev, you just need to add another vdev. I think that it's pretty easy to work around ZFS' disk additions. In your case with so many drives it may not be the best, but for many people starting with smaller pools, you can just use mirrors and add drives in pairs. Mirrors aguably have better performance in many situations, anyway.

You can also generally expand ZFS pools by replacing all the disks one at a time with bigger disks, and then expanding to the new disk size. Not sure how well that works outside of mirror sets, though.

 

[danswartz]

Yes, you can trivially add two disks at a time as new mirrored vdevs.

 

[Kelvarr]

The biggest problems (from a person who has on the order of 30TB of data on BTRFS filesystems between home and work) is that the filesystem can hang for long periods of time while (several minutes) mounting when it does not like what is in the space cache and also when deleting a large subvolume. That is on top of me not trusting the raid5/6 yet so I have to put btrfs on top of mdadm raid 6.

Oh cool...I wasn't sure if you could do that. My friend was just asking about putting ZFS on top of mdadm raid 6...do you know if it can be done with that? btrfs on top of raid 6 might be interesting.

For ZFS, you wouldn't have to add another 9-drive vdev, you just need to add another vdev. I think that it's pretty easy to work around ZFS' disk additions. In your case with so many drives it may not be the best, but for many people starting with smaller pools, you can just use mirrors and add drives in pairs. Mirrors aguably have better performance in many situations, anyway.

You can also generally expand ZFS pools by replacing all the disks one at a time with bigger disks, and then expanding to the new disk size. Not sure how well that works outside of mirror sets, though.

I thought the general thought was that it is best to add vdev's of identical size to the pool? I'm also not sure I want to lose that much disk space by adding them in mirrors.

Could one add 3 vdev's of 3x1.5TB drives in RAIDZ? I know I would lose 1/3 of my disks, but would that be better/worse than a RAIDZ2?

 

[Kelvarr]

Yes, you can trivially add two disks at a time as new mirrored vdevs.

So if I did that right from the start, I would have 4 vdev's of 2x1.5TB drives, and one spare/single drive. However, I would only have 6TB of usable space from the vdev's, correct?

 

[iroc409]

Yep, that's about right. Generally speaking, for many use cases the mirrors will provide better overall performance. If you're worried about adding disks in the future and want to use RAIDz1/2/3, you can add RAIDz vdevs in the future as well. It doesn't have to be mirrors, and they don't have to be equal.

That being said I've read some opinions that it's also better to have your disk numbers pretty stable and would be better off replacing existing disks for capacity expansion than adding vdevs. For example, instead of adding another vdev, you'd just replace your current 1.5TB drives with 3TB drives down the road.

The reason for this is that ZFS generally basically balances data across the storage. My server runs 2 mirrors with two 2TB drives and two 1TB drives. The 1TB drives are almost full, because ZFS distributes data evenly. I guess if you start adding vdevs down the road, you lose some of this balancing.

However, I'm not really all that sure it would make much of a difference, so YMMV.

 

[Aesma]

Many uses cases mean databases, VMs etc.

For data storage I feel mirrors are inappropriate, both a waste of space and too limited protection. If two drives in the same mirror quit, the whole pool is dead.

You might be better served by a data pool (RAIDZ2 or 3) and a VM pool, this one with mirrors, maybe on SSD if space needs vs costs allow.

 

[BLKMGK]

Funny, I'm nearly in the exact same boat as you are and was participating in that ARS thread trying to figure out what to do as well. The biggest difference is that I've GOT an unRAID system running now and running really well too. The downside being that waiting for disks to spin up is a hassle and transfer speeds aren't exactly stellar. Oh and I have no protection from the threat of "bitrot" which I'm not sure I've ever seen.

Your concerns also seem to mirror mine. Expanding my system is done slowly, generally when I run out of space I buy more and I try to buy what looks like good deals. ZFS doesn't seem to easily allow this. Yes, you can add Vdevs to a pool (think I've got that right) but those Vdevs themselves must be RAID or mirrors - each of which lose data to whatever parity scheme they each use. That's a TON of data storage lost to parity if you expand slowly! A dual failure within any of those small RAIDs does what exactly? unRAID loses one drive to parity but doesn't protect against a dual failure - except that a dual failure also doesn't lose ALL of the data in that RAID. The head spins pondering it all!

Btrfs catches my eye as it has your's. It sounds like it's easier to expand but it also sounds more difficult to manage that say Nap-It or FreeNAS. I'm about to tinker with some drives and a spare machine, figuring out the migration path for some 30TB worth of data if I schoose to do so will be no picnic either let me tell you but at least with unRAID I can clear data from drives individually and even pul them to mount elsewhere to copy. <sigh>