Please evaluate my NAS solution for home use

G

GioF_71

n00b
Joined
Mar 4, 2010
Messages
39
- I also posted this on serverfault, but it is offtopic because of home use -

Hello,
I am currently running my home server using Debian Squeeze, on a Intel Pentium E6700 based PC (Intel DQ45CB Board), and 6GB of regular (non-ECC) RAM.
There is a total of 15 drives connected, mostly green.

I decided to dedicate one a relatively small drive to rsnapshot; before this action, when rsnapshot was running, I was experiencing lots of frame drops and interruptions on my HTPC, which obviously accesses the NAS files through samba. No further issues after this action.

The backup of important data is mostly addressed and automatic, and it is not part of the question.

The other drives host media files, and of course the workload is mostly sequential. Currently, I have no raid volumes on this system. I use aufs to join the filesystem and have a samba share that show files contained on all the individual drives.

I own another system, an old AMD64 x2 3800+, socket 939, Asus A8N32-Sli board, 2 GIGs of RAM, some cheap controllers to allow more drives.
This system runs OpenMediaVault (debian based), and hosts a 4x2TB software RAID5.
I used xfs, tuning the su and sw parameters accordingly with my configuration. The performance is excellent for my workload.
This system has other drives, used as single disks. I use aufs to join them and show all files in a single smb share. It works just fine. Obviously the performance is lower when I access files on the single drives.

I am trying to modify the main system (the Intel based one) to use raid 5.
I know about baarf, read a lot of scary stories about very long recovery times of raid 5/6, but I really cannot spend an equivalent time trying to recover data from a failed hard drive when this happens. This has happened twice, but I have been lucky enough to be able to recover 99% of the data. Only, it took a lot of time.

My idea is to create multiple raid-5 arrays (4 drives) on the Intel based system, and to keep up using aufs to join those filesystems. I have no doubt it works as I essentially already used this approach on the other system. I also would like to have enough free space on all the array, at least to be able to tolerate a degraded array. I mean: when a disk failure happens, and an array changes its state to degraded, instead of immediately replacing it with a new drive and start the (dangerous) recovery, I would start to copy (or move) files to the other available arrays. At the end, I would simply replace the failed drive and rebuild a new array with the good drives and a new one, and eventually (if needed) re-balance.

I understand that leaving the array in a degraded state is equivalent to having a raid-0 array, and that this solution is obviously not enterprise proof, but take into account that we are talking about a home server, and that the costs must stay low (along with the noise). Also, the data on the raid volumes is not that important, some data loss is tolerable (of course I would like to avoid or minimize the chance of data loss).

I usually checksum new files before putting them on the server. I like ZFS features, but I would prefer to stick to linux. I'm satisfied with the checksum system (wrote some handy scripts), and had no issues since I started with it; also, I would prefer to avoid server-grade components, because of the higher cost.

Now, the real question. As I am open to alternatives, I am asking you to propose alternatives. Thank you, and sorry for the long post.
 
You could take a look at SnapRAID and/or ZFS-on-Linux.

SnapRAID is well suited to protecting home media NAS units, as the data isn't constantly changing, and different drives (ie capacity and model) don't pose a problem!

ZFS-on-Linux is pretty stable now, so that's another option. You could even combine the two if you wanted and get online checksumming thrown in.
 
Hello Billy_nnn, thank you for the quick response.
I have already considered snapraid, an although it is a very interesting piece of software, seems to me it is missing a feature that I really need: drive pooling.

On ZFS drive pooling (afaik) writes are striped through vdevs, indeed creating a dependency between the two (or more) raidz1 volumes, and definitely creating a weekness. For similar reasons I am not considering LVM as a solution in my scenario. Please note that I am not saying that ZFS Pooling or LVM are flawed, they simply are not good for me.

Some other information: I am not considering Greyhole (also a clever piece of software) because it is tightly linked to samba, and I want to be able to access my pool through ftp(s) coherently.
 
Oh sorry unless I create a snapraid volume with all my drives... I will investigate asap. Thanks again
 
I forgot to say that the extra performance (sequential read/write) given by the raid-5 4-disk arrays is desirable, so using snapraid would add another level of redundancy and waste space.

What I am looking for is a drive pooling solution, that allows also arrays as pool elements.

Thanks again
 
Copying all the data off the degraded pool will be almost as bad as a rebuild. Its requires the same parity calculation and reading data from all drives to copy the data to another location. If you are worried about the rebuild you should be just as worried about trying to move all the data off. In either scenario you are stressing a degraded array that could result in another failed disk and complete array loss.
 
Thanks Ripley, you remind me an aspect that I was not taking into account... what's your suggestion for my build? Stick to RAID 1+0 for example?
 
I'd suggest looking at RAID6 or ZFS Z2. And possibly consider keeping a cold spare drive around. That'll help minimize the risk. Also might want to consider an actual backup, maybe something like Crashplan. RAID is great but it isn't a backup.
 
Yes... it looks like at the end Raid6 (or Raid Z2) is the only viable solution.
I know it's not a backup, but as I wrote in the original post, the data that really need backups, already have backups (multiple and automated, phew, after a lot of work!)
 
I have a question about raid6. As I would like to continue using XFS, does this filesystem grow correctly when I add a new disk? When I create the file system, I specifiy the su and sw parameters. What happens when I add a new disk? Do I need to specify new su and sw parameters, or the filesystem re-arranges itself correctly?
I'm asking because I would like to start slow, say with 4 disks, then grow as I fill the raid from other drives' data and then add those to the array.
Thanks again.
 
GioF_71 said:
Hello Billy_nnn, thank you for the quick response.
I have already considered snapraid, an although it is a very interesting piece of software, seems to me it is missing a feature that I really need: drive pooling.

On ZFS drive pooling (afaik) writes are striped through vdevs, indeed creating a dependency between the two (or more) raidz1 volumes, and definitely creating a weekness. For similar reasons I am not considering LVM as a solution in my scenario. Please note that I am not saying that ZFS Pooling or LVM are flawed, they simply are not good for me.

Some other information: I am not considering Greyhole (also a clever piece of software) because it is tightly linked to samba, and I want to be able to access my pool through ftp(s) coherently.
Click to expand...

Have you considered Flexraid, which is pretty much snapraid with drive pooling using a GUI interface?
 
As I understood it, the OP is already using aufs as his pooling solution.
On Linux, mhddfs is another (similar) option.
You can operate such pooling mechanisms entirely separately to raid protection schemes such as SnapRAID or FlexRAID.

For instance if you had 8 drives, you could create a standard filesystem on each (any filesystem really - you could even mix them if you really wanted). Then create the SnapRAID/FlexRAID configuration, using the largest drive(s) as the parity drive(s). That takes care of the protection.
eg, use drives 1-6 for data and 7+8 for parity (just make sure that 7+8 are the largest drives)

Then configure and mount the pool, using drives 1-6, with aufs or mhddfs.
The parity drives would not form part of the pool - they aren't needed to present any data to the outside world.


However the OP then said he also wants striping, for performance - so his only real options on Linux (without any hardware solutions) are the linux md driver with his preferred XFS on top, or ZFS.

ZFS won't meet the expansion plans (adding a drive at a time as they become free), so that really leaves just linux md and XFS as the only viable solution given the OP's preferences and desires!
It's not the way I'd do it, but you pays your money and takes your choice! :)
 
Billy_nnn said:
As I understood it, the OP is already using aufs as his pooling solution.
On Linux, mhddfs is another (similar) option.
You can operate such pooling mechanisms entirely separately to raid protection schemes such as SnapRAID or FlexRAID.

For instance if you had 8 drives, you could create a standard filesystem on each (any filesystem really - you could even mix them if you really wanted). Then create the SnapRAID/FlexRAID configuration, using the largest drive(s) as the parity drive(s). That takes care of the protection.
eg, use drives 1-6 for data and 7+8 for parity (just make sure that 7+8 are the largest drives)

Then configure and mount the pool, using drives 1-6, with aufs or mhddfs.
The parity drives would not form part of the pool - they aren't needed to present any data to the outside world.


However the OP then said he also wants striping, for performance - so his only real options on Linux (without any hardware solutions) are the linux md driver with his preferred XFS on top, or ZFS.

ZFS won't meet the expansion plans (adding a drive at a time as they become free), so that really leaves just linux md and XFS as the only viable solution given the OP's preferences and desires!
It's not the way I'd do it, but you pays your money and takes your choice! :)
Click to expand...

Thanks Billy_nnn.

I also tried mhddfs once, by, despite the fact that it is easier than aufs (less options, and it does exactly what I was needing), it is much slower in write operations. That happened on an old atom 330, but with the same hardware aufs was much faster. In the reality, it was not slow on the i/o, but it was excessively taxing the cpu. Looks like there are some (simple) operations that are repeated over and over while writing to the mhddfs target. I also contacted the author but received no reply from him. Anyway, aufs is just fine.

I really don't like much the snapraid/flexraid approach, especially the fact that I have to choose the larger disk for parity (today's larger may not be tomorrow's larger). Will I have to deal with file copy/move when I decide to replace a parity drive with a larger one?

Looks like I am going to plan to switch to a raid-6 system.

Anyway, you said you would do something different, I'm interested, would you like to tell me?
Thanks
 
Personally I'd either use ZFS and accept the expansion limitations, or else use SnapRAID!

That said, the Linux md driver looks a lot better today than it was a few years ago - using it will mean aufs isn't needed.

Are all the 15 drives connected to the Intel machine the same size?
If you use multiple RAID6 devices though, (say 3x 5drive raid6) you'll be losing a lot of space to parity!

One of the upsides of SnapRAID/FlexRAID, is that when you suffer a drive failure (or failures), then if for some reason you cannot recover from the surviving drives and the parity drive(s), then you'll only lose the contents of the failed drive(s), not the whole array!
On a large number of spindles (large in home terms), another advantage might be the ability to only have the disks spinning that you need at the time, not all 15!

The downside is that you write to a single drive, not a stripe, so performance isn't as high - though if you upload over a network, you are typically limited to c80-100MB/s anyway!
 
Not sure what you mean by a con being that "you have to choose the larger disk for parity(today's larger may not be tomorrow's larger)"? You want to choose a smaller disk for parity?

If you decide to replace the parity drive with a larger one, all you would have to do is copy the parity data from the old parity drive to the new parity drive and then designate the new larger parity drive as your parity drive(this is the case with Flexraid anyway). Not sure how much simpler it could be.

Don't have much experience with standard RAID5/RAID6 solutions, and how it deals with user wanting to add larger disks,etc.. so I can't really comment on that vs how to replace parity disk with Flexraid

I do agree though, that performance wise, FlexRAID's not the fastest as its not striped. Works for my purposes though as I use it just as a NAS serving movies around my home network.
 
ghostdunks said:
Not sure what you mean by a con being that "you have to choose the larger disk for parity(today's larger may not be tomorrow's larger)"? You want to choose a smaller disk for parity?
Click to expand...

Of course not. I worry about the fact that, for instance, to be ready for next 3TB drives, it would be better to choose a 3TB drive for parity, even if I currently have only 2tb drives.

If you decide to replace the parity drive with a larger one, all you would have to do is copy the parity data from the old parity drive to the new parity drive and then designate the new larger parity drive as your parity drive(this is the case with Flexraid anyway). Not sure how much simpler it could be.
Click to expand...

Surely simple, I agree, but also surely time-consuming!

Don't have much experience with standard RAID5/RAID6 solutions, and how it deals with user wanting to add larger disks,etc.. so I can't really comment on that vs how to replace parity disk with Flexraid

I do agree though, that performance wise, FlexRAID's not the fastest as its not striped. Works for my purposes though as I use it just as a NAS serving movies around my home network.
Click to expand...

Thanks for your help!
 
Billy_nnn said:
Personally I'd either use ZFS and accept the expansion limitations, or else use SnapRAID!

That said, the Linux md driver looks a lot better today than it was a few years ago - using it will mean aufs isn't needed.

Are all the 15 drives connected to the Intel machine the same size?
If you use multiple RAID6 devices though, (say 3x 5drive raid6) you'll be losing a lot of space to parity!

One of the upsides of SnapRAID/FlexRAID, is that when you suffer a drive failure (or failures), then if for some reason you cannot recover from the surviving drives and the parity drive(s), then you'll only lose the contents of the failed drive(s), not the whole array!
On a large number of spindles (large in home terms), another advantage might be the ability to only have the disks spinning that you need at the time, not all 15!

The downside is that you write to a single drive, not a stripe, so performance isn't as high - though if you upload over a network, you are typically limited to c80-100MB/s anyway!
Click to expand...

Thanks.
If I'd choose RAID-6, my "blocks" would be composed of 8 disks, to keep loss of space due to parity at an acceptable 25%.

About the performance: as I know that a SINGLE home server will never be enough, unless I use very expensive & loud SAS expanders, I already have a secondary NAS. So sometimes I transfer data from a NAS to the other.
Cheap 10Gbit will come eventually, and I'd like to be ready!

About ZFS: looks like that ZFS on FreeBSD/FreeNAS is not as fast as mdadm. More, I like linux flexibility and I don't know whether I can find something like aufs on FreeBSD.
If you have different informations and comparative benchmarks on typical today's consumer setups, please tell me! By "consumer" I don't mean to use crappy PCI controllers, still I would prefer not to spend many hundreds for a controller.

Thanks again.
 
GioF_71 said:
Thanks.
If I'd choose RAID-6, my "blocks" would be composed of 8 disks, to keep loss of space due to parity at an acceptable 25%.

About the performance: as I know that a SINGLE home server will never be enough, unless I use very expensive & loud SAS expanders, I already have a secondary NAS. So sometimes I transfer data from a NAS to the other.
Cheap 10Gbit will come eventually, and I'd like to be ready!
Click to expand...

By the time 10Gb ethernet is down to cheapo "consumer" price levels, your current NAS will likely be history ;)

Your configuration of two md raid 6 arrays, presented as one by aufs, certainly looks workable. Personally, for a config like this, I'd use ZFS on Linux and run 2x RAIDZ2s but you have to go with what you feel comfortable with.




GioF_71 said:
About ZFS: looks like that ZFS on FreeBSD/FreeNAS is not as fast as mdadm. More, I like linux flexibility and I don't know whether I can find something like aufs on FreeBSD.
If you have different informations and comparative benchmarks on typical today's consumer setups, please tell me! By "consumer" I don't mean to use crappy PCI controllers, still I would prefer not to spend many hundreds for a controller.
Click to expand...

Personally I'm not really all that interested in performance much beyond that of a gigabit link at the moment - it's all I use really, so the eg. 5-600MB/s internal transfer rates are only of academic interest TBH, at least for now! :)
I have no figures for mdadm raid6 with XFS vs RAIDZ2 - although I'd be surprised if they weren't broadly comparable TBH - if you are configuring for absolute performance though, then 8 drive RAIDZ2s are probably not the ideal setup (try 6 or 10)
One advantage ZFS might have is that under rebuild, it only has to rebuild the live data, whereas AIUI, md would rebuild the whole disk, regardless of how much of it is just empty space!
 
Billy_nnn said:
By the time 10Gb ethernet is down to cheapo "consumer" price levels, your current NAS will likely be history ;)
Click to expand...

you're probably right....
But, on your opinion, when will a single 10Gb/s card reach a price tag of roughly 100$? This price would still be interesting for a direct connection between two NASes.

Billy_nnn said:
Personally I'm not really all that interested in performance much beyond that of a gigabit link at the moment - it's all I use really, so the eg. 5-600MB/s internal transfer rates are only of academic interest TBH, at least for now! :)
Click to expand...

AND again, you're most probably right.

I have no figures for mdadm raid6 with XFS vs RAIDZ2 - although I'd be surprised if they weren't broadly comparable TBH - if you are configuring for absolute performance though, then 8 drive RAIDZ2s are probably not the ideal setup (try 6 or 10)
One advantage ZFS might have is that under rebuild, it only has to rebuild the live data, whereas AIUI, md would rebuild the whole disk, regardless of how much of it is just empty space!
Click to expand...

Yes, and also, ZFS vdev creation is faster than mdadm's. I only tried once to create a mirror, it didn't take long. Do you confirm that ZFS vdev creation is faster also while creating a RAIDz1/2?

Thanks
 
Pool creation takes just a few seconds (this creates the first vdev too)

Subsequently adding a vdev takes just a few seconds too.
Be aware though that if you start with a single vdev pool, and add a second vdev to that pool when the first vdev is getting full, then there will be a load imbalance in the pool - as more data gets written, the first vdev will become full and only the second vdev will have any space. It's supported to do this and it works fine, but performance might be down a bit! (you can manually do a bit of load balancing though by just copying some files which existed before the second vdev addition, and then deleting the original)
Unless you particularly want/need all your storage in a single pool though, you might be better off just creating another pool (and perhaps using something like aufs if you want a "pooled" view of your storage)

As is so often the case, there can be several ways to do things, each, usually, with it's own set of pros and cons :)
 
GioF_71 said:
But, on your opinion, when will a single 10Gb/s card reach a price tag of roughly 100$? This price would still be interesting for a direct connection between two NASes.
Click to expand...

TBH I don't know, but I'd guess that's still a few years away yet! There's really no pressing need to get this kind of speed into the home, so I don't think there's much of a push yet!

Even so, personally I wouldn't consider $100 for NIC to be quite down to consumer level....perhaps $20-30 would be nearer the mark!
You may also need a switch.
 
Billy_nnn said:
TBH I don't know, but I'd guess that's still a few years away yet! There's really no pressing need to get this kind of speed into the home, so I don't think there's much of a push yet!

Even so, personally I wouldn't consider $100 for NIC to be quite down to consumer level....perhaps $20-30 would be nearer the mark!
Click to expand...

Now that would be nice :)

Billy_nnn said:
You may also need a switch.
Click to expand...

Not if I am fine with a NAS to NAS 10Gb connection (sort of the old 'crossover' cable connection).

No need to change the full network. As you pointed out, 1 Gb/s speed is more than adequate now.
 
GioF_71 said:
Of course not. I worry about the fact that, for instance, to be ready for next 3TB drives, it would be better to choose a 3TB drive for parity, even if I currently have only 2tb drives.
Click to expand...

Again, you have flummoxed me! :) how do you choose a 3TB drive for parity, if you only have physically have 2TB drives?
 
ghostdunks said:
Again, you have flummoxed me! :) how do you choose a 3TB drive for parity, if you only have physically have 2TB drives?
Click to expand...

I wouldn't choose 3TB when I only have 2TB drives, but what I were planning to throw in some 3TB drives in say a few months? I would have to change the 2TB parity drive with a 3TB one and throw the 2TB (ex-parity drive) to the pool.
Isn't it a bit clunky?
Perhaps it is only that I don't finish to accept this approach :)
 
Fair enough, but wouldn't you have even more of a limitation with a linux md raid5 array made from 2TB drives, if you were "planning to throw in some 3TB drives in say a few months"?
 
Billy_nnn said:
Fair enough, but wouldn't you have even more of a limitation with a linux md raid5 array made from 2TB drives, if you were "planning to throw in some 3TB drives in say a few months"?
Click to expand...

yes... it reflects the fact that I am still trying to figure it out. If money weren't a factor I would go RAID6 with 8 3TB drives. I'm still in the process of making a decision!
 
GioF_71 said:
I wouldn't choose 3TB when I only have 2TB drives, but what I were planning to throw in some 3TB drives in say a few months? I would have to change the 2TB parity drive with a 3TB one and throw the 2TB (ex-parity drive) to the pool.
Isn't it a bit clunky?
Perhaps it is only that I don't finish to accept this approach :)
Click to expand...

Echoing Billy's thoughts on this as well...if you were "planning to throw in some 3TB drives in say a few months", I'm not sure any other RAID solution on the market will make it any easier on you, if you wanted to swap the 2TB parity disk with a larger one in the future. Based on my limited knowledge of how RAID5/RAID6 would work, or something similar, you would need to rebuild the entire RAID array, rather than just copy just the parity disk.

Its all good that you don't wish to accept "this approach", but the question to ask then would be, what is the other approach that are willing to accept and how does that compare to this? Because I'm not seeing any other approaches that are simpler if you wanted to add bigger drives to your array later.

I don't have much experience with ZFS, other than creating some test ZFS stuff for fun and evaluating against RAID5, etc.. but I'm not aware of a simpler solution from that either.
 
ghostdunks said:
Echoing Billy's thoughts on this as well...if you were "planning to throw in some 3TB drives in say a few months", I'm not sure any other RAID solution on the market will make it any easier on you, if you wanted to swap the 2TB parity disk with a larger one in the future. Based on my limited knowledge of how RAID5/RAID6 would work, or something similar, you would need to rebuild the entire RAID array, rather than just copy just the parity disk.

Its all good that you don't wish to accept "this approach", but the question to ask then would be, what is the other approach that are willing to accept and how does that compare to this? Because I'm not seeing any other approaches that are simpler if you wanted to add bigger drives to your array later.

I don't have much experience with ZFS, other than creating some test ZFS stuff for fun and evaluating against RAID5, etc.. but I'm not aware of a simpler solution from that either.
Click to expand...

ghostdunks, I probably didn't point out clearly enough why I arrived to the decision NOT to adopt flexraid or snapraid. I try to write it down, also to reorganize my ideas:

1) I don't think I will adopt snapshot/flexraid much, mostly because I would be limited to single disk performance. Sequential read/write speed is a plus as I usually create checksums of large files, just to be able to verify data integrity when needed
2) I'd prefer to stick to a standard approach, and only use debian repositories packages

So I was thinking to:

1) buy 2x2TB drives and add them to the existing 4x2TB raid5 and reshape it to RAID6. Then fill it just enough to free 2x2TB (buying another drive if needed) from the main server ad throw them on the raid6, to reach my 8 drive RAID6 target.
2) Fill this raid6 volume, and go on to migrate all my drives to a raid6 configuration.

Does this sound good/viable?

Thanks for all advices.
 
Ah ok, well, if that's the reasoning why you didn't want to go with snapshot RAID-based solutions, that makes more sense. It just didn't make sense when you were mentioning that you were looking to upgrade the parity disk to bigger disk in the future, and how much hassle it was going to create if you went with the Snapraid/Flexraid approach.

If you're after pure speed, then yes, snapshot RAID probably isn't what you're looking for, as like you said, you're limited to single disk performance.
 
You must log in or register to reply here.
Back
Top