ZFS vdev of vdevs

[weav03]

Hi All,

I'm new to zfs, and just exploring for right now, so please forgive me if the terminology is slightly off.

I have the following hard drives:

6x 4TB
8x 2TB
6x 1.5TB

What I want to do, logically speaking is break everything into 4tb arrays and drives, IE

6x lone 4tb drives
4x 4tb raid 0 (consisting of 2 x 2TB each)
2x 4.5tb raid 0 (consisting of 3x 1.5TB each)

which gives me, essentially 12 x 4tb devices. Then I want to run raid-6 over these 12 and end up with 40TB usable and the ability to handle at least 2 drive failures. I know I should consider 3 parity bits for this many drives, but its a home media server/lab, I think it should be okay so long as I have a backup for the important files. Anyways. I cannot figure out a way to do this with zfs. The best I've managed is to have 3 raidz2 vdevs based on the size of the drive IE

1 raidz2 of 6x 4TB
1 raidz2 of 8x 2TB
1 raidz2 of 6x 1.5TB

and put them all in the same zpool. But this only gives me 33TB of usable space. Is there a way to get the setup I am describing in zfs? I am hoping to avoid have to go back to mdadm on linux as it is a pain to setup and manage manually (I'm running napp-it on OpenIndiana for the zfs)

Thanks for any advice!

-weav03

 

[JoeComp]

What is a "home media server/lab"? It is the "/lab" part that I don't get.

For a home media server, you'd probably be better off with SnapRAID or FlexRAID managing your parity data. You could devote two (or three) of your 4TB drives to parity, and then you could use the full capacity of the remaining drives.

 

[weav03]

What is a "home media server/lab"? It is the "/lab" part that I don't get.

For a home media server, you'd probably be better off with SnapRAID or FlexRAID managing your parity data. You could devote two (or three) of your 4TB drives to parity, and then you could use the full capacity of the remaining drives.

Hi Joe,

The "/lab" is because it is actually running on top of EsxI with 10G fiber cards and 10G-BaseT cards. I experimenting with some old switches, and a new one. I want to spawn new virtual machines, try out pfsense, try and use openvpn. I also have a raspberry pi I want to play with. Basically the home lab is just a way for me to experiment with "new" technologies (new to me anyways). I've read about SnapRAID, but it sells itself as a backup solution, not an availability raid solution. That has me a bit worried for my use cases. I have also heard of FlexRAID, but haven't look too hard into that. Does it have an easy to use front end?

Thanks!

 

[JoeComp]

I've read about SnapRAID, but it sells itself as a backup solution, not an availability raid solution.

Why are you concerned about occasionally having some of your media temporarily unavailable from a home media server?

 

[weav03]

Why are you concerned about occasionally having some of your media temporarily unavailable from a home media server?

I'm not so much concerned about whether the media is available, but everything else. If I'm running pfsense and it can't find its datastore, then I've lost my internet gateway. If I'm running openvpn and it lost its datastore, then I can't vpn to the network. I don't think the issue is with the media part of this.

 

[JoeComp]

I'm not so much concerned about whether the media is available, but everything else. If I'm running pfsense and it can't find its datastore, then I've lost my internet gateway. If I'm running openvpn and it lost its datastore, then I can't vpn to the network. I don't think the issue is with the media part of this.

You should separate your media from the rest of your data. I would use SSD(s) for the OS and other critical data. But if SSDs are not an option, you could put two of your 1.5TB drives in a mirror for the data that requires high-availability. Then you use SnapRAID for the media drives.

 

[weav03]

You should separate your media from the rest of your data. I would use SSD(s) for the OS and other critical data. But if SSDs are not an option, you could put two of your 1.5TB drives in a mirror for the data that requires high-availability. Then you use SnapRAID for the media drives.

Hmm, I do have some ssd's but they are all really small (30-70 GB). I was wanting to set them up as a read cache/log for the zfs. They are quite old. I will keep this idea in mind, but I am really thinking I want more than just media service here, though I certainly understand where you are coming from. I am liking zfs because it should be able to handle whatever I want to throw at it for my lab.

 

[JoeComp]

will keep this idea in mind, but I am really thinking I want more than just media service here, though I certainly understand where you are coming from.

I'm not so certain you understood the configuration I explained, but that's fine. Good luck!

 

[_Gea]

Building a pool like

6x lone 4tb drives
4x 4tb raid 0 (consisting of 2 x 2TB each)
2x 4.5tb raid 0 (consisting of 3x 1.5TB each)

is theoretically possible
- with file-devices that you span over several pools from smaller disks (very bad use)

other option
- partition each disk to 500/680 MB or 1TB partitions and build vdevs from these partitions (may work but I would not use)

On problems you must be very carefull or you may loose the whole pool although some Linux Raid go this partition way (I suppose Synology do it this way). You need to carefully build your vdevs from the partitions (may need 3 vdevs as well) to keep the "any two disks may fail" rule intact.

Other option is to build a non ZFS pool from snapraid. (ex single disks as basic pools) As you already know this is a backup solution that do not offer realtime protection and is limited to single disk speed.

Other option is to mix the two.
A snapraid solution for static data and a realtime raid where you need performance and realtime protection.


Or just build the pool from three vdevs and replace the smaller disk later to increase capacity (I would do)
1 raidz2 of 6x 4TB
1 raidz2 of 8x 2TB
1 raidz2 of 14x 1.5TB

 

[Aesma]

With ZFS you can use files as hard drives so what you want is possible, I've read (from _Gea for example) that it's not recommended but I'm not sure why, as I have not searched such solution extensively, at some point I prefer things to stay relatively simple (KISS).

napp-it won't offer this as it's a pretty exotic config, you just have to enter the commands manually.

 

[Concentric]

Hi All,
...
What I want to do, logically speaking is break everything into 4tb arrays and drives, IE

6x lone 4tb drives
4x 4tb raid 0 (consisting of 2 x 2TB each)
2x 4.5tb raid 0 (consisting of 3x 1.5TB each)

which gives me, essentially 12 x 4tb devices. Then I want to run raid-6 over these 12 and end up with 40TB usable and the ability to handle at least 2 drive failures. ...

I can see what you're trying to do there, but this is not the way to do things (at least with ZFS).

Forget trying to combine your smaller drives to make 4TB units. The pool doesn't have to consist of vdevs of the same size.
Ideally you wouldn't use a variety of different drives of different sizes and characteristics, but it's not a major problem.
I would just do as you said here:

The best I've managed is to have 3 raidz2 vdevs based on the size of the drive IE

1 raidz2 of 6x 4TB
1 raidz2 of 8x 2TB
1 raidz2 of 6x 1.5TB

and put them all in the same zpool.

But this only gives me 33TB of usable space.

First world problems! ONLY 33TB!
You could increase the capacity you get by changing down to RAID-Z, but then you run more of a risk of downtime from a drive failure. Perhaps that's OK for a home setup? The only space you would lose to redundancy would be one of each type of drive (4 + 2 + 1.5 = 7.5TB)

 

[danswartz]

Just don't have vdevs too too far apart in size, or when a small one gets full-ish, performance will get sucky...

 

[raiderj]

I think the time and effort you spend on building some hybrid setup would be better spend to selling those smaller drives and building a proper array from the start.

 

[danswartz]

Agreed.

 

[brutalizer]

If you are going to mix disks with different sizes for a media server, then I think it is better that you try snapraid/flexraid/unraid/etc.

Mixing disks with different sizes in ZFS is not really recommended. ZFS also has heavy data protection (according to researchers in computer science), but media files dont need heavy protection. If a bit is blue instaed of green - it does not matter. So therefore Snapraid/etc might be a better choice.

Of course, you can run snapraid ontop of ZFS disks.

 

[TGK]

OP: If you insist on doing what you proposed, you can accomplish it by utilizing mdadm to create software RAIDed /dev/mdX devices for the smaller drives. You can then add those /dev/mdX devices along with the 4TB drives to a ZFS raidz2. Due to the large # of drives, make sure you reference the disks with mdadm via the /dev/disk/by-id mechanism. Do the same for the physical 4TB disks you add to ZFS.

If you have it out for mdadm, you can use LVM to create stripe sets also, but it is even more complex.

This is a crappy idea for at least two reasons:
1) You're introducing a layer of abstraction between ZFS and the raw disks. This is generally frowned upon because it raises the chances that ZFS will not be able to detect disk issues/errors.
2) This is a more complex setup, and will have a much higher chance of going into a degraded state due to the RAID-0 drive arrays (they have a 2x higher chance of failing). When it happens, you'll have to remove both of the drives supporting the RAID-0 and replace with either a new RAID-0 array or a new 4TB HD. During all of this, if you mix something up you could accidentally hose your array.

Be absolutely sure you have a real backup method besides just a RAIDZ2 (which is not a backup).

 

[TGK]

Another thing you could do is you could create combinations of RAIDz arrays out of the different sized disk arrays, then get a unified view of them in the filesystem by utilizing mhddfs.

Ex:
tank1: Raidz2 of the 4TB drives
tank2: Raidz2 of the 2TB drives
tank3: Raidz of the 1.5TB drives

Mhddfs all 3 tanks together into a /pool.

That way you can independently manage the different sets of disks in their own arrays.