So storage people...22 disks

[Kelvarr]

How to best create an array from 22 disks? Reliability is the goal, performance is secondary. I am leaning toward RAID 50. Either 3x7(disk RAID5 SET) + 1 Spare or 4x5(disk RAID5 set) + 2 spares. Discuss.

 

[drescherjm]

Raid 60.

I would probably do a 3 x7 raid 6 with 1 spare. I mean if you can not use zfs. If you can use zfs then 2 X 10 disk raidz3s in your pool.

 

[Kelvarr]

ZFS is a possibility. Is there a big difference between RAID60 and Raidz3? Also, why can't you do 2x11 for Z3? Or are you accounting for spares?

 

[bexamous]

I see no reason to ever use RAID5.

I feel like 8 drive RAID6 vdevs mesh well with controllers and cases, and then go for multiples of that to make bigger arrays, no hotspares. But with 22 drives I guess I'd go with 7 drives RAID6 vdev and have a spare.

 

[drescherjm]

Is there a big difference between RAID60 and Raidz3

The added protection zfs gives you with end to end data integrity.

Also with triple parity 3 drives can die from each half without any data loss.

Or are you accounting for spares?

I am accounting for spares although you can go without them and use 11 drives for each vdev.

 

[_Gea]

Reliable: ZFS is best
Pool layout: 2 x Z2 vdev à 10 disks + spare disks or 2 x 11 disks Z3 without spare

 

[JoeComp]

It seems irresponsible to give any recommendations without more information. What is the purpose of the array? What size(s) of HDDs are you talking about? What OS are you expecting to use? Do you have a backup strategy yet?

 

[drescherjm]

It seems irresponsible to give any recommendations without more information

Although I gave advice I did think about that. I mean if this is an HTPC I would have a totally different answer for 22 disks. For an HTPC I would recommend snapraid (17 data + 5 parity or 18 data + 4 parity) with whatever filesystem the OP is most comfortable with.

 

[SirMaster]

Reliable: ZFS is best
Pool layout: 2 x Z2 vdev à 10 disks + spare disks or 2 x 11 disks Z3 without spare

I always heard RAIDZ3 was plain better than RAIDZ2+spare if performance is not a factor.

When you lose a drive why temporarily drop a level of redundancy when you could have included the drive in the array and started at 1 higher level of redundancy from the get-go?

The only reason I see using spares is for getting the array out of degraded mode ASAP for performance reasons. If you are OK with degraded perf for a few days while you get a replacement then it should be the more reliable option.

 

[omniscence]

Don't do 10 disk Z3. Lots of wasted space.

Although I gave advice I did think about that. I mean if this is an HTPC I would have a totally different answer for 22 disks. For an HTPC I would recommend snapraid (17 data + 5 parity or 18 data + 4 parity) with whatever filesystem the OP is most comfortable with.

Five parity disks? I assume that the chance of your PSU, a fire, a flood, or an asteroid killing all your drives is higher than even 3 disks failing at the same time.

 

[omniscence]

<delete>

 

[Kelvarr]

It seems irresponsible to give any recommendations without more information.

Fair enough. A reasonable response.

What is the purpose of the array?

It is a backup of the backup box. (Yes, 2 other boxes configured similarly)

What size(s) of HDDs are you talking about?

2TB

What OS are you expecting to use?

Ubuntu Linux

Do you have a backup strategy yet?

In discussion, but unlikely given its purpose.

 

[Kelvarr]

Although I gave advice I did think about that. I mean if this is an HTPC I would have a totally different answer for 22 disks. For an HTPC I would recommend snapraid (17 data + 5 parity or 18 data + 4 parity) with whatever filesystem the OP is most comfortable with.

Interesting. I have a completely separate home server that I am setting up that has 9x1.5TB disks. Primary usage will be Media Storage/Home Network backup. Some VM sandboxes will be there too.

Would you recommend snapraid for that? I have been tossing around ZFS, BTRFS, or plain old MDADM RAID6

 

[Concentric]

Definitely not RAID 50, especially with 22 disks.
You mentioned that this is for a backup of a backup, so your priority is to maximise your storage with reasonable redundancy and sequential performance. I would probably use ZFS and do one RAID-Z3 with several hot spares. If using another RAID technology and triple parity is not an option then perhaps RAID 60 with spares.

 

[tsrtg]

Interesting. I have a completely separate home server that I am setting up that has 9x1.5TB disks. Primary usage will be Media Storage/Home Network backup. Some VM sandboxes will be there too.

Would you recommend snapraid for that? I have been tossing around ZFS, BTRFS, or plain old MDADM RAID6

As far as I understand snapraid, it will not be able to recover all data from a failed drive if any file on any drive in the array changes after the last sync. Which means the moment there is a change in your VM image, you lose the protection no matter how many parity drives you have. So snapraid is only useful for completely static data which never changes.

I'd be inclined to use ZFS for both rigs. For the 22 disk array, use 2xRAIDZ3 because 11 disks is a perfect size for Z3 (2^3=8 plus 3 parity). For the home server, I would add 1 more disk and use RAIDZ2 (2^3=8 plus 2 parity).

 

[_Gea]

I always heard RAIDZ3 was plain better than RAIDZ2+spare if performance is not a factor.

Depends on the (future) pool layout. A single Z3 vdev is always better than a single Z2 vdev + a hotspare. If you build a pool from more than one vdev you can share a hotspare between them to reduce number of disks.

 

[omniscence]

As far as I understand snapraid, it will not be able to recover all data from a failed drive if any file on any drive in the array changes after the last sync. Which means the moment there is a change in your VM image, you lose the protection no matter how many parity drives you have. So snapraid is only useful for completely static data which never changes.

You are basically right. For that reason I combine ZFS with SnapRaid and sync based on ZFS snapshots.That way you can always recover to the state of the last snapshot on the lost disk.
This is not suited as VM storage as you state, but RAIDZx on conventional harddrives is generally not the best for multiple simultaneously running VMs anyway. A dedicated SSD pool would be the best.

 

[_Gea]

As far as I understand snapraid, it will not be able to recover all data from a failed drive if any file on any drive in the array changes after the last sync. Which means the moment there is a change in your VM image, you lose the protection no matter how many parity drives you have. So snapraid is only useful for completely static data which never changes.

This is correct. Another point is that snapraid & co are limited to a single disk performance while realtime raid like ZFS scale sequential performance over number of data disks and iops with number of vdevs/raids.

 

[drescherjm]

So snapraid is only useful for completely static data which never changes.

I believe parity still helps with changes. I believe it can ignore the changed data (treat the changes as a removed disk) and use the parity to recover a second bad block. However this increases the need for extra parity disks.

 

[JoeComp]

As far as I understand snapraid, it will not be able to recover all data from a failed drive if any file on any drive in the array changes after the last sync. Which means the moment there is a change in your VM image, you lose the protection no matter how many parity drives you have. So snapraid is only useful for completely static data which never changes.

No, that is incorrect.

If files are deleted or changed on one or more drives, and then a different drive fails (with no changes since the last sync on the failed drive), then you will be able to recover all your data as long as you have sufficient parity. The surviving drives with changes or deletions can be lumped together with the failed drive(s) in your analysis. If you have one failed drive and two surviving drives with changes or deletions, then if you have 3-parity or greater you can recover all your data. If you only have 1-parity or 2-parity, there will be data loss of an amount with upper bound equal to the total amount of data changed or deleted on the surviving drives.

Of course, you cannot recover any files from the failed drive that changed since the last sync.

By the way, omniscience, it is generally a good idea to throw in an extra parity drive or two with snapshot RAID, if you can. The reason being, as I explained above, having modified or deleted files on a surviving drive is similar to having that drive as a failure for recovery purposes. I am myself diligent in running a sync whenever I add content to my SnapRAID volumes (which I do usually once a week or less), so it is not a big issue for me. But for a person who is not so diligent, or has one or more drives with frequent changes in the SnapRAID array, it is a good idea to have extra parity drives. The SnapRAID FAQ recommends 1 parity drive for every 7 data drives (once you get above 1 parity drive), and I think that should be considered a minimum unless you are diligent with running syncs.

You are basically right

No, he and you are incorrect.

This is correct.

No, it is not correct.

I believe parity still helps with changes. I believe it can ignore the changed data (treat the changes as a removed disk) and use the parity to recover a second bad block. However this increases the need for extra parity disks.

Bingo.

 

[JoeComp]

It is a backup of the backup box. (Yes, 2 other boxes configured similarly)

2TB

Ubuntu Linux

I think ZFS on linux is your best bet, with second best being SnapRAID + mhddfs. Third choice would be XFS on mdadm RAID 60.

With either ZFS or SnapRAID, you would have checksum protection for your data, and you can do regular scrubs to check for any errors. XFS / mdadm can also do scrubs, but since it does not have data checksums, you are relying on dual parity to find errors.

ZFS would be my first choice because it combines realtime RAID and pooling.

If you are okay with Snapshot RAID (which, as a backup box, it should be fine, since you just schedule automatic syncs after each backup completes), then SnapRAID would work, but it has the added complication that if you need pooling, you would need to use an additional tool such as mhddfs. SnapRAID does have the advantage that you can use different sized HDDs in the array if you like, and you can do up to 6-parity (although I think 3- or 4-parity should be sufficient with 19 or 18 data drives).

 

[omniscence]

No, he and you are incorrect.

Well, my comment was directed at the inability to restore the changes to VM images. Rereading his comment I think that was probably not what he meant.

I see that adding more drives could help with restoring in case of altered data. But in my eyes that is still bit unreliable unless you are aware of what data is on what disk when you change something. I use aufs to pool the filesystems together and at a certain point you can lose track of the physical location of the data. To alleviate that I run syncs only on snapshots and run with two parity drives only. That way I can only lose data added since the last sync. Seems like the best compromise to me.

 

[JoeComp]

Interesting. I have a completely separate home server that I am setting up that has 9x1.5TB disks. Primary usage will be Media Storage/Home Network backup. Some VM sandboxes will be there too.

Would you recommend snapraid for that? I have been tossing around ZFS, BTRFS, or plain old MDADM RAID6

I would recommend you split the VM sandboxes off. For example, you could have an mdadm RAID 1 of 2 drives for the VMs. Then the other 7 drives could be used with SnapRAID (I recommend 2-parity drives).

Alternatively, if all the VM sandboxes will fit on one drive, then you could have a single SnapRAID volume using all 9 drives, but you would want at least 3-parity, since 1 parity drive would be needed to protect against the frequent changes in the VM files. However, that puts the recent changes to your VM files at risk if the VM drive fails, so I would not recommend this as my first choice.

 

[bexamous]

I can't imagine using snapraid at work... even mdadm at this point for anything but a mirror... any amount of storage ZFS, even ZoL. If this is backup of other systems even more reason to... for cheapo 'replication' you can run zfs send/receive and send incremental snapshots ever 5-10 minutes with so little overhead.

At home I use snapraid because I don't have unlimited funds... snapraid lets you have efficient use of your disks, I have a mix of 2-4TB drives in snapraid array and any time I need more space I buy one more disk. At least for static content.

More frequently modified stuff, eg 'My Documents' type data, 2-4 disk zpool.

I can't stand VMs on anything but SSDs at this point, but I also can't really justify spending tons of mirrors and stuff... so VMs go on individual SSD zpools but I use zfs send/recieve to send incremental snapshots to the previsouly mentioned 2-4 disk zpool. Worst case some SSD wtih VMs on it fails I have a snapshot that is at most 10 minutes old or so on a spinning disk.

 

[omniscence]

Alternatively, if all the VM sandboxes will fit on one drive, then you could have a single SnapRAID volume using all 9 drives, but you would want at least 3-parity, since 1 parity drive would be needed to protect against the frequent changes in the VM files. However, the puts the recent changes to your VM files at risk if the VM drive fails, so I would not recommend this as my first choice.

Would it even make sense to add the VM images to the SnapRaid volume? The frequently altered data cannot be restored anyway. And unless you shutdown the VMs during the syncs you may end up with inconsistent filesystems. Additionally each sync would have to read a lot of data from all disks because of the VM images. The additional parity disk could be used as a mirror for a dedicated VM storage disk instead.

I can't stand VMs on anything but SSDs at this point, but I also can't really justify spending tons of mirrors and stuff... so VMs go on individual SSD zpools but I use zfs send/recieve to send incremental snapshots to the previsouly mentioned 2-4 disk zpool. Worst case some SSD wtih VMs on it fails I have a snapshot that is at most 10 minutes old or so on a spinning disk.

Completely agreed, that is what I do as well. Although in a production environment you may not be able to live with a VM state that is 10 minutes old. Database servers for example. And a block-level snapshop of a VM image without a saved VM state is like you pulled the plug on that VM after a restore.

 

[JoeComp]

Would it even make sense to add the VM images to the SnapRaid volume? The frequently altered data cannot be restored anyway. And unless you shutdown the VMs during the syncs you may end up with inconsistent filesystems. Additionally each sync would have to read a lot of data from all disks because of the VM images. The additional parity disk could be used as a mirror for a dedicated VM storage disk.

As I said, I recommend splitting off the VMs in their own mirror.

The only reason I mentioned the other possibility is that some people seem to want to have everything on one large pool with mhddfs or whatever, and they cannot be convinced to split off frequently changed data.

As for VM files changing during syncs, I think SnapRAID will notice if a file changes during a sync and either error out or not include the changes in the sync. So no "inconsistent filesystems", but certainly difficulties. Of course, this is a hassle that can be avoided by splitting of the VMs into their own mirror, as I recommend.

 

[drescherjm]

When file changes during a snapraid sync or scrub it will list the file that changed and continue on the rest of the files it needs to process. I think it also warns you about changing files during the sync and asks you to sync again.

 

[tsrtg]

If files are deleted or changed on one or more drives, and then a different drive fails (with no changes since the last sync on the failed drive), then you will be able to recover all your data as long as you have sufficient parity. The surviving drives with changes or deletions can be lumped together with the failed drive(s) in your analysis.

Thank you (and drescherjm) for this explanation, this was completely not obvious for me after studying snapraid documentation on their web site. Hell, even now when I read the snapraid FAQ it seems confusing:

In the worst case, any file deleted or modified in not broken disks may prevent to recover the same amount of data in the broken disk. For example, if you deleted 10GB of data, you may not be able to recover 10GB of data from the broken disk. The exact amount of data lost depends on how much the deleted and broken data overlaps in the parity.

To reduce this problem you can use two parity drives. This improves a lot the chances to recover the data.

"Reduce this problem" and "improves the chances" is not the same as "With two parity drives you will be able to recover the data if files on only one drive were modified". They really need to rephrase this.

 

[JoeComp]

"Reduce this problem" and "improves the chances" is not the same as "With two parity drives you will be able to recover the data if files on only one drive were modified". They really need to rephrase this.

The SnapRAID developer is not a native English speaker, so a lot of the documentation has awkward phrasing. But it is difficult to state clearly and simply, since technically, "improves the chances" is correct in general. I mean that, in general, you could have modified or deleted files on multiple data drives, so however many parity drives you have, it may not be enough for certainty of 100% recovery, in general.

I guess the best way to improve the documentation would be to write some specific examples. But I think the developer much prefers to write code than English documentation. (not that I have a big problem with that...those who consider it more important could always pitch in and help with the documentation)