Which ZFS layout would you use for the following, and why?

J

Johnyblaze

Limp Gawd
Joined
Nov 9, 2003
Messages
300
Hardware
24 drive bay case
4TB drives, all same make/model/etc.

Possible ZFS configurations, arranged by number of disks
20 disks
raidz2 = 1 zpool, 2 vdevs of 10 disks, 2 can fail per vdev, 4 can fail in total
32TB per vdev, 64TB total usable, 20% of total drives can fail

22 disks
raidz3 = 1 zpool, 2 vdevs of 11 disks, 3 can fail per vdev, 6 can fail in total
32TB per vdev, 64TB total usable, 27.27% of the drives can fail

24 disks
raidz2 = 1 zpool, 3 vdevs of 8 disks, 2 can fail per vdev, 6 can fail in total
24TB per vdev, 72TB total usable, 25% of total drives can fail
 
Last edited:
I let the type of data, useage needs, redundancy and other environmental requirements dictate how I organize available resources into zpools. But then again, who am I? :D

"RAID Greed" can be real for some, not so much for others. But no one ever got fired for using mirrors. Jus sayin.


Give us some more details about your data and wishes, because you seem more than capable of doing the math on your own.




protip: It's sound practice to mix up makes and even models of disks in an array (while keeping sizes consistent in a vdev). The more disks in an array, the greater the statistical need not to get slammed by a bad manufacturing run that affects ALL your disks, instead of just some of them.
 
Last edited:
paret0 said:
I let the type of data, useage needs, redundancy and other environmental requirements dictate how I organize available resources into zpools. But then again, who am I? :D

"RAID Greed" can be real for some, not so much for others. But no one ever got fired for using mirrors. Jus sayin.


Give us some more details about your data and wishes, because you seem more than capable of doing the math on your own.




protip: It's sound practice to mix up makes and even models of disks in an array (while keeping sizes consistent in a vdev). The more disks in an array, the greater the statistical need not to get slammed by a bad manufacturing run that affects ALL your disks, instead of just some of them.
Click to expand...

Good point on the different disks, thanks.

danswartz said:
+1 on the mirrors. or at the least, more vdevs for better random performance.
Click to expand...

What type of config would you recommend with smaller vdevs?

omniscence said:
What about the application and workload?
Click to expand...

Home use, 2-3 machines at once, media, personal files, ESXi homelab datastore. Non production, basically. Also, the server will have 64GB ram so I figured I could get by without ZIL and L2ARC.
 
Last edited:
For home use I would not build such monstrous pools, difficult to handle if you have change something in your config.
Just build two spearate 11 disk RAIDZ3 or 2 two 10 disk RAIDZ2 pools. Your media files will not require the IOPS to justify multiple vdevs, it would also be a waste of disks to put these on mirrors.
I would put the VM datastores on mirrored SSDs instead for high IOPS.

ZIL and RAM have not much to do with each other. ZIL is for higher sync write performance.
 
omniscence said:
For home use I would not build such monstrous pools, difficult to handle if you have change something in your config.
Just build two spearate 11 disk RAIDZ3 or 2 two 10 disk RAIDZ2 pools. Your media files will not require the IOPS to justify multiple vdevs, it would also be a waste of disks to put these on mirrors.
I would put the VM datastores on mirrored SSDs instead for high IOPS.

ZIL and RAM have not much to do with each other. ZIL is for higher sync write performance.
Click to expand...

Thanks everyone for the replies. With that said, I've narrowed it down to the following:

20 disks
raid z2 = 2 zpools, 2 vdevs of 10 disks, 2 can fail per vdev, 4 can fail in total
32TB per zpool, 64TB total usable, 20% of total drives can fail

22 disks
raid z3 = 2 zpools, 2 vdevs of 11 disks, 3 can fail per vdev, 6 can fail in total
32TB per zpool, 64TB total usable, 27.27% of total drives can fail

24 disks
raid z2 = 2 zpools, 4 vdevs of 6 disks, 2 can fail per vdev, 8 can fail in total
32TB per zpool, 64TB total usable, 33.33% of total drives can fail

Thoughts?
 
I have 24 3tb setup with 2 pools: 2x10raid2z and 3raidz (used to have stripe mirror with 4 hdd). Although the large pool is fast, its not very safe (already had 2 drives died from the same vdev at the same time, but everything was okay and still works). If I could do it again, I would use 2 pools with 12raid2z. Gives me maximum space and good redundancy.
 
Just so the OP understands his math of % total drives CAN fail is an optimistic case. You could get unlucky and lose the whole pool with less drives failing if the the wrong ones fail.
 
Also meant to add with his math mirrored vdevs offer the best "% total drives can fail" of 50% :). As long as the right 50% fail.
 
If this is a HTPC you probably are better with individual disks and snapraid. Possibly with 4 or 5 parity disks instead.

If this is not an HTPC and is used in a business I would do the 22 or 24 disk setup and make more than 1 full backup to an LTO6 autochanger.. Or a second option for backup would be to create 1 or 2 more zfs boxes with the same amount of storage or greater and use send and receive to push your snapshots to the second server. I would have the backup server(s) power down when not doing the send and receive.
 
Last edited:
If you plan on using this as a datastore for virtual machines as well, you'll want a separate pool for it. With this many disks, and considering the use case, you'll probably want to spin down most of those drives a lot of the time.

When I built my AIO I thought "I'll just build one pool, assign an SSD for L2ARC and be done with it"... It worked.. Just not very well. I could never spin down the drives and I got the performance of a single, slow drive.

I did the math, and calculated how much I would save on power by being able to spin down the drives. In my case, by spinning down 5 drives 16 hours per day, I could save enough money to pay for a couple of 2.5" 1TB WD RED's that I'm now using for my virtual machines. If it was today, I would probably just get SSD's.

TLDR; in your case with 24 drives, you'll save a lot of money by being able to spin those suckers down, and that will not happen if you use them for virtual machines.
 
I always thought in a 24 bay case that 4 6-disk RAIDZ2 vdevs would be best.

You have 6 backplanes I think (6 rows 4 columns), so you can set it up so you could lose 2 backplanes at once and the array would still be up and running.

Also 6-disk RAIDZ2 has a lower overhead than 8-disk RAIDZ2. About 1TB per vdev lower.

I would expect 4 6-disk RAIDZ2 vdevs to have about 64TB - 1.4TB overhead = 62.6TB usable.
I would expect 3 8-disk RAIDZ2 vdevs to have about 72TB - 4.6TB overhead = 67.4TB usable.

It's just a difference of about 4.8TB.
 
spazoid said:
If you plan on using this as a datastore for virtual machines as well, you'll want a separate pool for it. With this many disks, and considering the use case, you'll probably want to spin down most of those drives a lot of the time.

When I built my AIO I thought "I'll just build one pool, assign an SSD for L2ARC and be done with it"... It worked.. Just not very well. I could never spin down the drives and I got the performance of a single, slow drive.

I did the math, and calculated how much I would save on power by being able to spin down the drives. In my case, by spinning down 5 drives 16 hours per day, I could save enough money to pay for a couple of 2.5" 1TB WD RED's that I'm now using for my virtual machines. If it was today, I would probably just get SSD's.

TLDR; in your case with 24 drives, you'll save a lot of money by being able to spin those suckers down, and that will not happen if you use them for virtual machines.
Click to expand...

Thanks for the tips! I didn't even realize the drives would not spin down with the ESXi datastore.

SirMaster said:
I always thought in a 24 bay case that 4 6-disk RAIDZ2 vdevs would be best.

You have 6 backplanes I think (6 rows 4 columns), so you can set it up so you could lose 2 backplanes at once and the array would still be up and running.

Also 6-disk RAIDZ2 has a lower overhead than 8-disk RAIDZ2. About 1TB per vdev lower.

I would expect 4 6-disk RAIDZ2 vdevs to have about 64TB - 1.4TB overhead = 62.6TB usable.
I would expect 3 8-disk RAIDZ2 vdevs to have about 72TB - 4.6TB overhead = 67.4TB usable.

It's just a difference of about 4.8TB.
Click to expand...

I think I'll settle on the following:

2x 10 disk z2 zpools
1x 4 disk zpool mirror dedicated for ESXi datastore

Thanks for the help everyone!
 
For the ESXi datastore pool, consider doing two mirrors with two disks each, it'll give you twice the IOps and space.
 
You can't do 1 mirror with 4 disks, it'll always be 2 vdevs with 2 disks each (unless you're doing 3-way mirrors of course)
 
And they load-balance unlike mdraid (at least last I checked it didn't load balance reads...)
 
I haven't read much about it, but I do recall reading that low spindle count vdevs have terribad write performance?
 
For raidz* yes. Actually they have terrible read performance too :) Well, random read/write sucks. Sequential is okay.
 
You must log in or register to reply here.
Back
Top