Large, Dynamically Expanable Storage Array

[bluemonkeyfish]

I have been scouring the forums looking for several solutions to my problem.

i am currently in the process of redesigning my storage system (Freenas, ZFS Raid 5+1 10x750GB SATA HDs.)

I have two 16bay SAS Chassis that im looking to install sas expanders in using an areca HBA or Raid Card. My only issue is im looking for the following.

1. for every 16 drives, use 2 or 3 for parity (in one array) EG: 13 x CAPACITY + 13 x Capacity = Array

D D D D D D D D D D D D D P P P

D D D D D D D D D D D D D P P P

2. Dynamically Expandable Like X-Raid2, so when i get a new chassis and or new drives i can either replace existing drives with larger ones or add new ones to increase my storage. I am trying to store all DVDs, HDDVDs and Bluerays. i also want to be able to store pictures, music and recorded TV shows over the network from any of my 11 pc's latops, or media extenders.

the question i cant find an answer to is there dynamically expandable ZFS, and can i add more parity drives as i add more disks (Eg: 128 in one array)

I also was looking for something that will provide better security such as NTFS but it's second as my data's integrity is most important and security is second.

Thanks!

 

[sub.mesa]

ZFS can't expand an existing 4-disk RAID-Z to a 6-disk RAID-Z, for example.

What you can do with ZFS, is add another array called a vdev, to your existing pool. So if you have a 4-disk RAID-Z, add another 4-disk RAID-Z, and then later another 4-disk RAID-Z. Now you have 12 disks total of which 3 parity drives; essentially three RAID5 in RAID0. This is the way to expand when using ZFS.

RAID-Z: groups of 3 to 6 (when using 4K sector disks: 3, 5 or 9)
RAID-Z2: groups of 6 to 12 (when using 4K sector disks: 6 or 10)

RAID-Z = RAID5, single parity
RAID-Z2 = RAID6, double parity

You can do the same with mirroring, start with a single mirror of 2 disks, then add mirrors as you go. You can even degrade a mirror to a single disk, to temporarily 'steal' a disk from that mirror, and give it back later.

Another advantage here is that your next vdev you add, can be of different capacity. You can start with 1TB disks, later add a vdev of 2TB disks, and in the future use 3TB disks; all in one pool, all sharing the same free space.

Also, when expanding you don't degrade the redundancy level, like you would with traditional RAID5 expansion: adding disks but not adding redundancy means you lower the effective redundancy. With ZFS you can maintain the redundancy level: when you add disks you also add parity disks.

Cheers.

 

[OldSchool]

Hardware RAID 6

Can add one drive at a time if you want to

 

[odditory]

Also, when expanding you don't degrade the redundancy level, like you would with traditional RAID5 expansion: adding disks but not adding redundancy means you lower the effective redundancy. With ZFS you can maintain the redundancy level: when you add disks you also add parity disks.Cheers.

No offense but I think that's a little too generous in terms of putting a positive spin on what's essentially a huge shortcoming in ZFS - the inability to OCE. The simple answer I think the OP was looking for is no you cannot take say a 10 drive raidz2 and add a single disk at a time whenever capacity is almost full. ZFS is essentially chaining arrays together and presenting as one volume. And anyone in their right mind running hardware raid with > 8 drives is running RAID6 these days anyway, so "redundancy" as an argument for ZFS's approach to pool is pretty thin since striping arrays is a performance multiplier, not a redundancy multiplier.

Hardware RAID: Expand a disk at a time
ZFS: Expand an array at a time

If you don't want to incur the penalty of a parity drive or drives every time you expand a ZFS pool, all you can do is backup and reformat. Hardly ideal when you've got say 16TB in use and transfer speed in and out of your ZFS box is bottlenecked at 100MB/s GigE. Even if you have a duplicate copy of the data it still takes days to restore dozens of TB over GigE.

 

[sub.mesa]

I didn't say traditional OCE would not be nice for ZFS; it would! And one day we might have OCE; expanding a RAID-Z to more disks, or shrinking it instead, or transforming a mirror to RAID-Z or RAID-Z to RAID-Z2 or degraded RAID-Z2 to normal RAID-Z; etc; yes it would be nice!

But that doesn't change that the current implemented way of ZFS expansion has some advantages over traditional RAIDs:

• You can expand with larger/different capacity disks (you can't with Hardware OCE)
• You can change the redundancy level as you expand (not that sexy but also possibly a benefit)
• It only takes a few seconds to actually use the free space (Hardware OCE takes ages)
• It's completely safe (hardware OCE would need very good error recovery from a BER issue during OCE for example; quite possibly alot of simple RAIDs fail here.
• You can scale as you expand; with hardware RAID a 4-disk RAID5 might seem a good idea, but once you surpass 8 disks RAID5 isn't that sexy anymore. A 2x 4-disk RAID5 is better, and 4x 4-disk RAID5 would continue to provide decent protection; so you can expand indefinitely since you're not lowering the effective redundancy level.
• It also enhances random I/O performance; where traditional RAID5 would involve all disks on a partial write ("random write") or non-aligned end I/O; ZFS now has two (or more) arrays which can do a random write in one I/O stroke; involving no sync reads like on hardware RAID5.

That said, i should also list the disadvantages of the current ZFS expansion:

• Expanding with a single disk will give you RAID0; not what you want.
• Expanding with two disks would give you 50% redundancy (mirror/raidz).
• Expanding with more than two disks still give you less usable storage capacity since you're adding at least one disk as parity/redundancy each time you expand.
• Expanding does not enhance performance on already written files on the pool. You can move them around to force them to be striped across all vdevs; but still a limitation.
• You cannot shrink a raidz, or migrate between between redundancy levels; except for single disks and mirrors, which can be degraded to stripes or upgraded to mirrors.

So while i agree it's certainly a limitation and thus negative aspect of ZFS, it also provides advantages not seen on traditional solutions available. That's not uncommon for ZFS; it also has different performance needs like alot of RAM memory and SSD for L2ARC/SLOG to be really fast; unlike traditional RAIDs where you don't need alot of memory but need TLER-capable disks instead, and also a BBU if you want to enable writeback; and you would want to for RAID5/6 write performance not to be utter poor.

Each solution has its own merits; the tricky part is figuring out which advantages/disadvantages carry the most weight in your specific situation. So write them down, compare, and think objectively and out-of-the-box to what solution would best suit your needs given your money constraints.

Cheers!