zfs raidz vs mirror 2-4 disk configuration

chrcoluk

[H]ard|Gawd
Joined
Jul 7, 2008
Messages
1,247
Hi guys, to me storage isnt cheap so I am trying to find the most affordable way to do this but also taking into consideration risks.

Not buying a NAS, using my 2600x rig as the storage unit.

Peformance is not important other than maybe resilver speed. The load on the system will be writes when transferring files to it from PC, and reads if I am playing back those files (video clips). So 5400rpm drives fine no problem. Priority is bitrot mitigation and redundancy.

Also potentially taking into account backups on top of redundancy which of course makes costs even more expensive (paying for 4 gig to have 1 gig of data).

So in my PC I have a bunch of spindles, one ar empty or almost empty so not available to remove and put into the project.

In the ryzen machine the only reasonably sized drive is a 3tb RED (CMR). Looking at new priced WD red plus drives, 3tb is a silly buy only 10% cheaper than 4tb, 6tb cost is on par with 4tb, above that it gets silly again and also way too expensive for me.

The current case is poorly designed for taking drives in and out, its hideous, it also only has max capacity for 6 drives, I will probably want to get a fractal define r5 for it, 8 easy accessible bays.

So looking at options. Lower capital investment is important here, also remembering I dont need 10s of tb's of space, the vast majoirty of dicussion seems to be around the needs of mega huge 8+ drive setups with massive storage.
My actual immediate needs is redundancy for my video clips, mostly from games I am playing but also I have downloaded from others, Current needs are about 700gig, obviously I want more then that so it isnt filled immediatly. I also want to move my macrium backups on there as I have had bitrot issues with my macrium backups in the past, so thats another approx 500gig.
I am undecided if my video clips on there will be copies, or if they will be deleted from the PC meaning whilst I would have redundancy/bitrot mitigations, they would still be the only copies of the files. So another potential option is I start doing scheduled macrium backups of my video files, keeping them on my PC with thos ebackups going on the storage. But either way the expected disk usage will be the same.

If I move the videos of my PC as well as the backups it potentially opens up the option of me removing a couple of drives from my PC and then expanding the pool.

I am ok with 50% redundancy cost, so was looking at between either of the following 3 options initially.

2 x mirrored VDEV's 4 drives 50% space. Fastest resilver, potentially 2 drives can fail, or only 1 drive, if the 2nd failure is in same pool. Expansion can be done 2 drives at time.
4 drive RAIDZ2 50% space. Slower resilver, any 2 drives can fail. Expansion would probably require another 4 drives for 2 x RAIDZ2 in pool.
Single mirrored VDEV 50% space, lowest initial cost, one out of 2 drives can fail, expansions in 2 drives at time by adding mirrored VDEV's.

I have not really managed to find out how much "faster" mirror resilvering is vs RaidZ2, its faster but by what %, to me if its only e.g. 10% faster, then RAIDZ2 makes more sense for safety, in this case comparing 4 drive mirror setup to 4 drive RaidZ2 setup, I know would be the advantage that a resilver in mirror only needs to do one drive's wirth of data, but the speed comparison I want is on a gig to gig basic, so basically the throughput of the resilver, has this ever been tested by anyone? Longer resilver adds risk, but having any 2 disks been able to fail is lower risk. I dont have concerns for performance in normal operating mode as single disk performance is enough for me.

The plan I had was to buy either 1 or 3 4tb drives, take out the spare 3tb thats in the amd system, clone a 4tb drive to the 3tb drive, then physically swap the drives in PC, add the 4tb drive from PC which would make either a 2 or 4 pool setup depending if I buy 1 or 3 drives. A 4tb pool would be enough for my immediate needs and probably for at least a year, a 8tb pool (4 drives), I expect would last me for the forseeable future, at least 3 years minimum.

Even with this approach my absolute minimum spend will be about £100 for 1 drive to get it going, £300 for three, plus if I do it £110 for the new case, storage costs can ramp up very fast. For giggles I checked the price of a 8 bay NAS and pretended I purchased 8 6tb drives, enough to buy an rtx 3090 and rtx 3080 together o_O

For the time limited readers, essentially the main thrust of the post is a question of is a slower resilver time of raidz2 in a 4 drive setup worth it for the better redundancy vs 2 mirrored vdevs.
 
Last edited:

Zedicus

Gawd
Joined
Nov 2, 2010
Messages
937
well based on your needs the other thing to consider would be expandability. the mirrored drives in a VDEV, will allow you to add another VDEV to the pool with only adding 2 drives at a time. the 4 drive RAIDZ2 would require you to add 4 drives as a VDEV and then add that too the pool.

as for the resilver times, in RAIDZ configs they get exponentially longer as the drives get bigger. right now, comparing 4tb drives in mirror vs RAIDz2, resilver has minimal difference. in 4 years when you have added drives or done a drive swap to larger drives, you resilver times will be significantly longer. I recently moved from 4tb RAIDZ2 arrays, to 8tb mirrors, and while i did not have a ton of space increase initially, expandability in the mirror config was worth it.

also, don't forget ECC ram if ZFS is your use case. an ASRock desktop board with a Ryzen PRO CPU will work with ECC NON-REG RAM. Even an older opteron rig is usually way more than enough performance for ZFS at home, and it is easy and cheap to get tons of ECC REG ram on an older opteron rig off thiefbay. My current ZFS server is actually an ancient opteron rig with perc H330 cards and a 10gb NIC. silly cheap for the performance.
 
Top