[Q] RAID-Z migration from unRAID

[ironicbadger]

I'm looking to make the move very soon to ZFS. I've checked out BTRFS but I don't think it's stable enough yet to be considered.

I intend to run ZFS within Arch Linux whilst it is acting as the host for a Xen setup with VGA passthrough for my Windows gaming VM. This setup works well currently with unRAID as a VM but I'm always after a new project / performance boost.

The machine has 32GB ram and an i5 3470 processor. Plenty powerful enough for a home setup in my opinion - I'm not running a data centre from home so unless 32gb ram is REALLY not enough I think I should be fine.

My unRAID setup comprises of the following drives giving me ~19TB of usable space protected by parity in the case of a single drive failure:

• 5x 3tb
• 1x 1.5tb
• 2x 1tb
• 2x 500gb
• 1x 320gb

My questions surrounding the migration are as follows.

1. Drive redundancy (RaidZ vs Z2 etc) - The single drive failure is enough resilience for me with unRAID because each drive is individually accessible when something breaks. Is this wise on raidZ?
2. As I migrate across the 18TB data I have stored do it bit by bit? ie remove one drive from unraid and give it ZFS, copy the data across then repeat?
3. How will ZFS cope with mismatched drive sizes?
4. Does it require all drives to spin at all times when accessing data? I like this feature on unRAID as it's primarily a media collection and there's no need to spin up 11 drives to access the files on 1.

 

[Phog]

Just FYI, your unraid box will have better read speed than a ZFS box with those drives. Obviously the write speed will be better on ZFS (because unRAID write is slow). Boosting ZFS with SSD's will help, and will probably get the read speeds on par with unRAID.

I think everyone concurs RaidZ2 is the way to go. On RaidZ, if you lose two drives, you lose it all. unRAID is unique in the way it handles parity, meaning you could lose all but one disk but still recover some data.

It won't cope with mismatched drives as seamlessly as unRAID.

Tinkering is fun, but unRAID (imo) is unmatched for an easy, cheap, low-power, and fast media streaming server.

Granted, if you're planning on hosting some VMs over ZFS, that's another story. I'm sure plenty of people will RAH RAH RAH ZFS, good for them, it has some serious data protections built into it, but it also adds a layer of complexity and expense if you're looking for solid performance.

 

[JL6speed]

If you have a lot of drives, I would go with RAIDZ#. However, I think in unRAID, you are able to mismatch drive sizes and in RAIDZ# you can't (since data is stripped).

 

[ironicbadger]

how do the pools of drives work? noone has ever explained it in simple luddite terms (what i require)

could i make a "pool" 3TB in size and then add up to 3tb of drives to each pool?

 

[Phog]

If you dumb it down, it's still RAID, it's just you can add a bunch of raid sets to a giant pool.

 

[danswartz]

I think you are going to have problems down the road. ZFS pools do not do well with mixtures of disparate size like you have. You can do it, but you will likely regret it.

 

[JL6speed]

Oh forgot to add, unRAID u can expand your array and drive size doesn't matter. In ZFS, you are pretty much set going in and drives need to be same size. Expanding will be an issue.

 

[drescherjm]

I would sell all the drives that are less than 3TB and buy 1 or 2 more 3TB drives.

 

[Nex7]

5x 3tb
1x 1.5tb
2x 1tb
2x 500gb
1x 320gb

Yikes. There's not really much you can do here, in ZFS. ZFS zpools are made up of 'virtual devices', or vdevs. The vdevs are comprised of disks, and where you specify a RAID type. The zpool stripes (RAID0) across the vdevs. As a general rule of thumb, you should not have vdevs with different parity (RAID) types in the same pool, nor should you have different sized disks in a vdev, nor vdevs of different size in a pool. All of that is actually doable, but it will reduce performance and cause weird performance issues as you grow.

So with that in mind, you basically have insufficient drives to build any sane pool. I don't really recommend raidz2 vdevs less than 6 disks ever, so you have insufficient drives of the same type to make even one of those. Probably the biggest pool you could presently build would be a pool with 2 vdevs, each comprised of 2 of your 3 TB drives in mirror, so you'd have a 4-drive pool made up of 2 mirror pairs, thus equivalent to RAID10. And having an initial capacity of ~50% of the total, or 6 TB.

I'm with drescherjm - sell'm all but the 3T's, get a few more 3T's, and then build either a pool with 1 x 8-disk raidz2 vdev or 4 x 2-disk mirror vdevs. For the record, the 4-vdev mirror pool will be 4x 'faster' from a random IOPS perspective than the 1-vdev raidz2 pool.. but in most home environments this is not actually noticeable.

As for the comment about read/write speeds - generally speaking if you hand the ZFS box 8+ GB of RAM it can utilize for ARC, in a home use-case you'll find it tends to get a >98% ARC hit rate, not only significantly reducing the number of reads that even hit disk, but also massively improving latency and throughput (though often that's then nixed by home network, as those tend to be 1 Gbe). Without either a dedicated GOOD, power-safe SSD for log or the conscious decision to disable sync (ZIL) mechanics, and in both cases also only if you have sufficient RAM to hold your outstanding writes w/o incurring write throttling mechanics, you'll find write speeds to be less when compared to other RAID arrays/filesystems. I don't bother to have an SSD, I left the ZIL on, I only have 8 GB of RAM, and I myself have just a 4-disk 2TB mirror pool, and I've never had a problem with speed at the home (though of course, this would be way too small to survive in any sort of business environment).

 

[HobartTas]

Greetings

Your situation is a bit complicated and there are several things to consider and I don't think in your case migration will be possible given you have existing drives which I gather are mostly full of data, anyway I will explain the constraints you have to work with and then you can decide yourself how you want to approach this problem.

Drive redundancy (RaidZ vs Z2 etc) - The single drive failure is enough resilience for me with unRAID because each drive is individually accessible when something breaks. Is this wise on raidZ?

This is a judgement call, one drive failure is a reasonable possibility, two in a home environment at once is less likely, one parity drive Raid-Z is probably sufficient. You will most likely find that people that run two parity Raid-Z2 setups probably don't/can't afford to backup their data and sort of "harden" their array "availability wise" in the hope of not losing their data, usually people run Raid-Z for 4-6 drives and Raid-Z2 for 10-12 Drives and Raid-Z3 for say 16-20 drives.

Kindly remember that the IOPS of the array usually amounts to that of a single drive in the array so given say 10 drives a home user would run a 10 drive Raid-Z2 array to minimise the wasted space (2 parity) drives whereas a business would run two 5 drive Raid-Z2 to maximise IOPS at the expense of more parity drive usage and hence less available space.

If you run a Raid-Z and have URE's and also lose a drive you will have broken stripes everywhere there is a URE bad block but unlike other filesystems like NTFS then ZFS will tell you which affected files are damaged and need to be replaced.

could i make a "pool" 3TB in size and then add up to 3tb of drives to each pool?

Pools are made up of any combination of

(a) single drives
(b) mirrored drives (including 3 way mirrors, actually N way mirrors if you want)
(c) Raid-Z? arrays

You can have any number of pools you like with any combination of drives as described above.

The constraints in your case are

(1) Any drives added to a pool cannot be removed once added, your only option in that case is to take all the information off the pool and then destroy the pool which releases all the drives.

(2) Since data is distributed to all drives in the pool It is not a good idea to mix the above types e.g.

(a) say you have a 5 drive Raid Z-1 and you add a single drive then in that case if you lose the single drive then you will lose data as there is no parity or mirror protection for it.

(b) you have a 6 drive Raid-Z2 and you add a 5 Drive Raid-Z then two hard drive failures in the Raid-Z2 array will be no problem but two drive failures in the Raid-Z will lose data in that array, hence the weakest link in the pool is the lowest protection level.

(c) however, you probably could mix Raid-Z1 and mirrored drives together successfully if all you were concerned about was losing a single drive as both types will cope adequately with a single drive failure.

(3) the MAJOR CONSTRAINT in your situation (and everyone else's for that matter) is that you cannot extend or shrink a Raid-Z? array, whatever you create stays until you destroy it. e.g. in my case I created a 10 drive Raid-Z2 array in an older Q6600 PC in a tower case as I wanted to minimise parity drives and maximise storage space, once I fill it up then as I am building a Socket 2011 system as my Socket 1366 is a bit underpowered for BF4 I will retire it as my current gaming PC and the X58 will become another NAS and I will buy another 10 drives for an new Raid-Z2 array to put in there.

So the answer to your question is yes you can add a 3TB drive to your pool, the question is How will you add it? as a single drive which is not recommended as it has no parity and will weaken any redundant volumes, you could add two of them as mirrors or you could add X drives as a new Raid-Z? array. The choice is up to you.

[*]How will ZFS cope with mismatched drive sizes?

• 5x 3tb
• 1x 1.5tb
• 2x 1tb
• 2x 500gb
• 1x 320gb

Assuming the drives are all empty (which they obviously are not) you could do

1. 320 GB partition on all drives and do a Raid-Z? array on those 11 drives
2. 180 GB partition on 10 drives and Raid-Z? array and add that to the pool.
3. 500 GB partition on 8 drives and Raid-Z? array and add that to the pool.
4. 500 GB partition on 6 drives and Raid-Z? array and add that to the pool.
5 1500 GB partition on 5 drives and Raid-Z? array and add that to the pool.

It's messy but it would probably work, a drive failure would require you to replace with a similar sized drive and a larger one would have wasted space if its bigger.

It's desirable to have the number of data drives in the array to be a power of 2 so I think it's best if you just buy say 10 new 2TB or 3TB drives and make a new Raid-Z2 array and just transfer the existing data across and retain your old drives as a backup. I presume all the Sata ports you have can cope with those size drives.

As I migrate across the 18TB data I have stored do it bit by bit? ie remove one drive from unraid and give it ZFS, copy the data across then repeat?

If you buy new drives as I suggest you could temporarily attach the drives and copy the data across or transfer them over the network, otherwise I can't see how you are going to manage this unless you are just going to have a pool of JBOD's starting by erasing your existing parity drive and starting your pool with that and then transferring one drive at a time then erasing that and adding that to the pool as a new single drive and repeating this procedure ten times.

Does it require all drives to spin at all times when accessing data?

Hard to say as ZFS does cope with green drives so I presume the only way to do this is have the drives spin down automatically due to APM, given that most manufacturers more or less put this on all consumer drives (and you can't disable this) as I believe they don't want people to use consumer drives in hardware raid arrays (as they want to sell them more expensive enterprise drives) then this is fairly easy to achieve. There is always the possibility however, that ZFS may poll drives on a regular basis and keep them awake so it would be best if you can find out elsewhere from other people how often their green and other APM drives spin down (or not).

Conclusion

I think I have answered all your questions and you should have enough information to decide how you want to proceed. ZFS is definately a superior filesystem and raid setup compared to anything else but it comes with its own advantages and disadvantages. One major advantage that recently presents itself is with the latest CryptoLocker ransomware floating around is the fact that you can snapshot your pool on a regular basis as snapshots have relatively little overhead (unless you start changing or deleting files) and your snapshotted data will be immune from the ransomware as snapshots are read only. As the ransomware encrypts the PC and any other nework shares it has access to I have read that even people who are dutifully doing regular backups are getting both their PC and the external hard drive(s) they are currently backing up to are being encrypted with this ransomware.

Cheers

 

[danswartz]

" I would sell all the drives that are less than 3TB and buy 1 or 2 more 3TB drives. "

This!

 

[Aesma]

Agree. Also RAID is not backup. I have a 19 drives RAIDZ3 and still full backups.

 

[JoeComp]

I'm guessing the vast majority of your Bytes are media files -- movies and/or music. If that is so, and you only have a relatively small amount of non-media files (media files = write once, then only read; non-media files = frequently modified, deleted, etc.), then I suggest segregating your data.

SnapRAID is an excellent choice for handling your drives with media files. It supports snapshot RAID with different size drives, and the current released version handles up to dual parity. Version 5.0, which is at the RC stage, supports up to 6 parity drives (RAID-S6). Like UnRAID, it will only spin up the drive that has the media file you are playing. And each drive has its own filesystem, so if you lose more drives than you have parity, you still can access the data on the remaining drives. Unlike UnRAID, SnapRAID allows you to use any filesystem you want on your data drives (even mix and match). You could use ZFS on each drive if you like. Unlike ZFS RAIDZx, SnapRAID allows you to expand your array one drive at a time, of any drive size, or to migrate an array to a different number of parity drives. There is even an Archlinux AUR package of SnapRAID.

Importantly, you will have an easier time migrating your data to SnapRAID than to a ZFS RAIDZx array. The reason is that you can migrate 1 drive at a time with SnapRAID. In fact, you would not necessarily have to migrate your data at all since I believe UnRAID writes a mountable filesystem to each drive. But you would still probably want to segregate out your non-media files. And it would probably be better to switch to another filesystem for your media drives, even if you are not required to (you could go ZFS, ext4, XFS, btrfs, etc.)

For your non-media files, you could create a simple 2 drive mirror with ZFS (or mdadm). Maybe your two 1TB drives could serve that role if 1TB is enough for your non-media files. Or add in a 2-drive mirror of your 500GB drives to the pool for 1.5TB for your non-media files.

 

[bexamous]

Yeah I run Arch w/single ZFS pool for VMs and non-media stuff, and then everything else is snapraid array w/btrfs on each drive, then mhddfs to pool them together to share on network.

 

[ironicbadger]

wow, so much info...

my feeling overall is that i will stick with unraid as it mixes all the features i want in a simple package. it just seems a lot of work for not a huge gain after reading all that.

 

[Nex7]

Go for it. Just -- keep backups. Which would be my advice if you went ZFS, too.

 

[ironicbadger]

so, i've dutifully sold all my mismatched drive sizes and replaced them with a pair of 3TB drives. I now have 7 drives, all 3tb.

I think then, that I will going for a 6 drive single parity setup based on the above advice. What will my total useable array space be with this setup?

I've read plenty about BTRFS and how it's not yet stable in 'raid' form, how true actually is this?

The data stored is non-critical and backed up elsewhere. About 95% is media files, the rest is my photo library and ripped music collection which I'd prefer not to loose from the main server should a drive (or two but this is unlikely at home concurrently) die. I've been bitten by a lack of backups before, so don't worry I don't see RAID as backup - I see it as a convenient risk reduction tool.

Some more ZFS questions:

- are the drives themselves individually readable (as they are in unRAID) by plugging them into to any Linux OS on their own?

- the SSD required for 'logging', how much does this improve performance bearing in mind i'm not a business - this is just at home for my family's usage (although they don't tolerate slowness, hence the move from unRAID).

- what sort of read / write speeds am i likely to achieve? unRAID sees 25-30mb/s write, and if i'm lucky 40-60mb/s read over samba / nfs.

- i'll be running ZFS on Arch, what pitfalls are there? How much RAM is minimum, ideally I don't want to give it more than 8gb but if 16gb = much better performance then I'd do that.

- why should i use ZFS with it's complications over mdadm with an LVM layer on top?

- finally, what are the rebuild times like vs standard mdadm?

 

[Aesma]

If you use RAIDZ then data is stripped so you get nothing from a single drive. On the other hand you get great sequential performance (hundreds of MB/s). I wouldn't be comfortable with less than RAIDZ2 with 3TB drives.

8GB of RAM is fine (that's what I use with a 19 drives RAIDZ3).

For your usage you don't need a logging device.

Can't answer for ZoL as I'm using OpenIndiana.

 

[brutalizer]

ZFS drives in a raidzX are not individually readable, if you plug them into a Linux OS. Snapraid drives are. For your use case (lot of media) I suggest Snapraid.

 

[ironicbadger]

I discount snapraid as I want realtime parity. Its just simpler and I know its done.

I have used snapraid in the past but found it unreliable if files were open or something. Duno, it just seems clunky as a concept.

 

[brutalizer]

For media files you dont need realtime checksumming. Why do you want this for media files?

 

[ironicbadger]

i can update over 750gb in one day if i go on a binge, granted that doesn't happen every week but i turn over at least 1tb a month. so the files aren't that static.

besides, i have the hardware to cope with ZFS / any other solution. I want to use it and I'm doing this to learn so I can use the knowledge in interviews...

I just don't think snapRAID is what i'm after. I do appreciate your advice but I want to learn ZFS / any viable alternative.