RAID controller recommendations for Debian Wheezy

[scruffters]

Hello all,

I have been reading this board for years, but this is my first post

I’m trying to spec a Linux powered storage rack based on Supermicro’s 826TQ chassis (12x SATA , built in port expander, so usable with most 4i cards). It will be used for video edit storage at home and will hooked up to a Netgear XS708 10GBE switch, mostly serving half a dozen video streams between 120-200 Mbps.

One of the issues I’m running into is provisioning the storage on Debian Wheezy, so I wondered if you folks would kindly lend me some opinions/advice?

Requirements:
The storage software I am using is packaged for Debian, so that is one restraint.
24TB usable space required
RAID6 should provide enough redundancy - I have a backup in place.
NAS drives (RED/HGST NAS) - enterprise HDDs are going to be out of my budget.

One of the issues I keep hitting is finding decent support for RAID controllers on Debian. While many have drivers in kernel, there are a lack of natively packaged tools. Indeed, while I’m happy to run CLI, it is nice to have something like 3ware’s old-school 3DM2 web GUI (which is .deb packaged).

Options considered so far:

ZFS on Linux
PRO: Big love for ZFS on Linux, I already use 0.6.3 on a much smaller rig and it works wonderfully
CON: concerned about creating such a big pool over a single vdev. Z2 would probably take a hell of a long time to resilver if/when I need to.

3ware 9750-4i
PRO: I’ve not seen much else boasting anything similar FW to StreamFusion+. 3DM2 GUI & native driver support.
CON: OLD + limited drive support in HCL. Unsure about buying 12 3TB+ NAS drives without a solid reccomendation of what works well.

HP Smart Array P222/P420
PRO: work well, HPSA native driver, VOD profile and Web ACU
CON: Not really recommended on non-HP hardware, LED indicators etc

So any ideas chaps/chap-esses? I’m open to suggestions

Thank you.

 

[dandragonrage]

ZFSonLinux has worked perfectly for me. In fact it has been significantly better than any other implementation of ZFS I have tried. I've had random and MAJOR performance issues with various versions of OpenSolaris-based distros (e.g. OpenIndiana) as well as updates that corrupt the system (mainly with NexentaStor). I didn't give FreeBSD's implementation much of a chance, I admit, but I did not like it when I did.

Your 24GB is actually not that big a constraint for ZFS (nor for just about any recent hardware RAID card) - you could do that with 4 6TB data drives (+2 more for dual-parity) or 8 3TB data drives (+2 more for dual-parity).

I've had several brands of hardware RAID cards throw drives - including RAID-safe (e.g. no TLER issue) drives - out of arrays for no reason and absolutely refuse to rebuild or reuse the drive. I've also had this issue with an IOMega Storcenter where one drive failed and it refused to recover the array even though it had enough good disks to do so (and I believe that machine uses Linux md software RAID). I've had Adaptec, LSI, 3Ware cards all repeatedly throw drives for no reason. I consider this to be completely unacceptable.

ZFS is literally the only RAID solution I have never, ever had these issues with. Personally, I don't trust anything other than ZFS these days. You still need backups to be safe, but in my experience, you will need to actually use them (for recovery) far less with ZFS. I have yet to need them.

 

[scruffters]

Hello dandragonrage, thankyou for the detailed reply.

I'm pleased to hear that you have also had good experiences with ZFS on Linux. My story with 3ware and LSI has been much the same as yours - drives getting dropped for seemingly no reason. That said, in those cases I had not adhered exactly to the HCL, although had used NAS style drives.

Part of my concern about ZFS is how to best design a large pool using 12 x SATA drives. The 10 x 3TB Z2 approach seems like a good option, as you mention (8 + 2)... especially as Z2 max drives is supposed to be 10 and I would probably use R6 with a hardware board anyways...

Do you have any experience with resilvering an array of this size? For example - expected time to complete when 80% full? I had considered using an i3 4130 to keep it low power, but might reconsider a Xeon if that is going to hamper the rebuild process....

Other than that I don't believe there would be much benefit using a mirror for video streaming workloads, which tend to be less about small file IOPS. According to HP's Smart Array documentation video streaming is mostly random, so you don't even really want to read ahead, but you do want to post writes - which might be a good case for a ZIL cache.

I suppose having 2 x hot swap caddies free gives you options to add caching as required, although I am really stumped at this point as to how that will work out. I expect there will be a certain amount of experimentation required here.

 

[dandragonrage]

Unfortunately I don't have experience with that; the largest ZFS pool I've used is 6 drives. Unfortunately, while I did almost get some of our stuff at my work onto ZFS (which would have meant 24+ drives), my plan was ultimately rejected and we went for some cheaper off-the-shelf solution that we have not been very happy with.

I don't think it would be any worse to resilver a 10-drive RAID-Z2 than it would be to do any sort of rebuild with a hardware RAID card.

There are several people on the forum who have dealt with ZFS pools much larger than mine - you can search for their threads, or maybe they will reply here at some point. From what I've read, these people are mostly happy with their ZFS setup and have had no major problems (nothing worse than with a hardware RAID card) when needing to replace/resilver.

The downside of ZFS is that it does require some knowledge to do certain tasks, but hey, it also took knowledge to repair the non-existant problems that RAID cards were convinced existed.

Hell, I've even had hardware RAID cards detect some non-existent problem, try to auto-correct it, and create real corruption problems on the whole array. (Edit to note: this is not impossible with ZFS, though I haven't heard of it. ECC RAM is recommended for this purpose and should cut the chance of this down by 1-2 orders of magnitude, but I would still use ZFS w/o ECC)

Even Areca cards, which have been well-liked on this forum, are known for throwing drives for no good reason. I have never used an Areca card, though.


On another note, BTRFS, the Linux filesystem that is supposed to be comparable to ZFS, is absolute crap. I've installed Linux dozens of times over the past few years and I've used various filesystems for boot and root (primarily EXT3, EXT4, XFS, and BTRFS). Recently installed with BTRFS root. I set up KVM virtualization and it froze my entire machine. Rebooted and my entire system was corrupt and while it was likely repairable, it turned out to at least be difficult to do so. I reinstalled with XFS - a filesystem not known for high relability / data integrity - and while KVM froze several times, it never caused a single issue. So do NOT be tempted to trust BTRFS to be anywhere near as good as ZFS, at least not as of kernel 3.17.2.

 

[SirMaster]

ZFS on Linux
CON: concerned about creating such a big pool over a single vdev. Z2 would probably take a hell of a long time to resilver if/when I need to.

I have a RAIDZ2 array made up of 12 4TB WD Red drives and I'm using ZoL 0.6.3 on Wheezy. It's set up as 2 6-disk RAIDZ2 vdevs. I get 700MB/s read and 500MB/s writes. It has 18TB of data on it currently and takes 14 hours to scrub and 8 hours to resilver a disk. I don't consider this a "long" time at all. Also WD Red drives are some of the slowest, so it would be even faster with faster disks.

 

[JoeComp]

I set up KVM virtualization and it froze my entire machine. Rebooted and my entire system was corrupt and while it was likely repairable, it turned out to at least be difficult to do so. I reinstalled with XFS - a filesystem not known for high relability / data integrity - and while KVM froze several times, it never caused a single issue. So do NOT be tempted to trust BTRFS to be anywhere near as good as ZFS, at least not as of kernel 3.17.2.

Have you tried that workload on ZFS on linux? I wonder why you switched the root fs from btrfs to XFS instead of to Zol.

 

[dandragonrage]

Have you tried that workload on ZFS on linux? I wonder why you switched the root fs from btrfs to XFS instead of to Zol.

I don't trust ZoL as root yet and I use git packages for ZoL which I haven't ever had corruption issues with or anything, but even though I use DKMS I still seem to always have to remove and reinstall the package with new kernel versions. Just a risk I don't feel like taking, even though nothing important is on my root partition. I don't upgrade kernels all that often, maybe once every couple months or so. But XFS is doing the job okay for root with nothing important. My VMs themselves are stored on ZFS, but I haven't tried a ZFS root partition.

It really surprised me to learn how much BTRFS still sucks. I really expected better from it. We use it at work and it makes me worried. We have gotten root corruption with it once at work as well, but I was able to easily recover it.

(I use Arch, for the record, and note, I am only 'decent' at Linux, so maybe someone else would have the DKMS stuff working better than I currently do.)

 

[cantalup]

using ZoL on root is bad decision, as already mention, Zol is compiled with DKMS

I am using btrfs for home , and using XFS for root on centos 7. this is perfect without error.

I am very concern when btrfs or Zol is using for root. this will be a mess during recovery.


for Zol, I have 12X2T raidz2 drive (mix drives).
I did resilverring when replaced a bad drive. total time was ~6 hours.
the system is running centos 7 with sas2 backplane.

love sas2 expander than sas1 expander(give me troubles mostly)

do NOT forget to disable APM on seagate desktop drive (barracuda aka desktop) if you are using it,
and use wdidle3 tool under linux to disable or make longer timeout on WD green drive (I prefer REAL WD blue drive, the max capacity is 1T as I know)
I did it via script that got called during booting time.

 

[scruffters]

I don't think it would be any worse to resilver a 10-drive RAID-Z2 than it would be to do any sort of rebuild with a hardware RAID card.

Past a certain point you have to just jump in and order the gear - I’m getting near to that I guess

Another option could be to use a mirror, which I believe will resilver a bit faster.

Then again the problem with mirroring is the utilisation of space, although it is easy to add pairs of drives later - right? …so if you have to initially spend on more expensive bigger HDDs that could also work out. Not sure on the general guidance for creating 10-12 drive mirrors.

Hell, I've even had hardware RAID cards detect some non-existent problem, try to auto-correct it, and create real corruption problems on the whole array. (Edit to note: this is not impossible with ZFS, though I haven't heard of it. ECC RAM is recommended for this purpose and should cut the chance of this down by 1-2 orders of magnitude, but I would still use ZFS w/o ECC)

I've never had any trouble with ZFS on Linux with small pools (3-5 drives), so familiar to some extent. I suppose my major concern is about scaling out 10 x drives, and the downsides associated with rebuilding a large vdev.

On another note, BTRFS, the Linux filesystem that is supposed to be comparable to ZFS, is absolute crap. I've installed Linux dozens of times over the past few years and I've used various filesystems for boot and root (primarily EXT3, EXT4, XFS, and BTRFS). Recently installed with BTRFS root. I set up KVM virtualization and it froze my entire machine. Rebooted and my entire system was corrupt and while it was likely repairable, it turned out to at least be difficult to do so. I reinstalled with XFS - a filesystem not known for high relability / data integrity - and while KVM froze several times, it never caused a single issue. So do NOT be tempted to trust BTRFS to be anywhere near as good as ZFS, at least not as of kernel 3.17.2.

Thanks for the heads up, I’ve always used XFS for large hardware arrays - BTRFS is by all accounts a work in progress..

I have a RAIDZ2 array made up of 12 4TB WD Red drives and I'm using ZoL 0.6.3 on Wheezy. It's set up as 2 6-disk RAIDZ2 vdevs. I get 700MB/s read and 500MB/s writes. It has 18TB of data on it currently and takes 14 hours to scrub and 8 hours to resilver a disk. I don't consider this a "long" time at all. Also WD Red drives are some of the slowest, so it would be even faster with faster disks.

Thanks for the info.

Part of the worry is that I need one large pool, rather than 2 vdevs as you have - so I suppose the rebuild time could be as long as 16 hours…

Out of interest what sort of impact did the resilver have on IO?

In the use case of video streaming, 16 hours of not being able to use the storage properly could be a bit of a pain. One of the pros of a hardware solution is that the impact of rebuilding can usually be minimised using various settings.

ZFS does seem like the best route though, its just a question if it will really perform how I need given the workload and requirements of the application. That is the real ball buster!

 

[dandragonrage]

Then again the problem with mirroring is the utilisation of space, although it is easy to add pairs of drives later - right? …so if you have to initially spend on more expensive bigger HDDs that could also work out. Not sure on the general guidance for creating 10-12 drive mirrors.

It's very easy to add pairs of drives and new RAID-Z* groups. The two things you can't easily do (that many hardware RAID cards can do) are expand an existing parity-based RAID (RAID-Z*) and convert RAID types in-place. You'd have to back your stuff up, destroy and recreate the array to make those changes. But you shouldn't really need to make those changes, anyway.

I use RAID-10 equivalent, groups of mirrors. RAID-10 equivalent. Good performance.

 

[scruffters]

It's very easy to add pairs of drives and new RAID-Z* groups.

The idea of adding another storage pool makes sense.

The part that I find abstract is adding more drives to increase the pool capacity, without having to start over

When using a striped-mirror vdev (R10 equivalent) my impression was that you can dynamically extend storage pool by simply adding additional pairs of drives. This would allow me to 1) populate free bays later 2) avoid having to create additional share points located on separate storage pools (which is messy)

The two things you can't easily do (that many hardware RAID cards can do) are expand an existing parity-based RAID (RAID-Z*) and convert RAID types in-place. You'd have to back your stuff up, destroy and recreate the array to make those changes. But you shouldn't really need to make those changes, anyway.

To tell the truth I've never been to bothered about doing online changes of this sort. It always seems like a bit of a gamble.

I use RAID-10 equivalent, groups of mirrors. RAID-10 equivalent. Good performance.

By "mirror" I really meant R10 - so what you said

 

[dandragonrage]

The idea of adding another storage pool makes sense.

The part that I find abstract is adding more drives to increase the pool capacity, without having to start over

When using a striped-mirror vdev (R10 equivalent) my impression was that you can dynamically extend storage pool by simply adding additional pairs of drives. This would allow me to 1) populate free bays later 2) avoid having to create additional share points located on separate storage pools (which is messy)

Yes, you can. Each ZFS pool (volume) can consist of a number of vdevs, and vdevs can consist of individual drives, mirrored drives (2 or more), or a parity/RAID-Z* configuration. You can add more vdevs, be they individual, mirrors, or parity. But you cannot expand a parity group. You actually can add more mirrors to a vdev, though it obviously won't increase your total space. But it would increase your overall reliability.

Data is striped across all vdevs (which is why multiple mirrored vdevs are a RAID-10 equivalent rather than, say, a JBOD of mirrors).

Data is NOT redistributed when you add vdevs, so if you add them looking for more performance, you'd need to move the files you care about off the array and back onto it (or create a second copy on the array and then delete the first) if you want maximum performance. For most use cases, though, this won't matter and performance will always be good enough. Your data reliability won't really be affected either since the striping across vdevs does not add additional redundancy - the redundancy issue should be solved by the vdevs themselves (e.g. using mirrors or parity). Actually, the striping reduces redundancy, but your vdevs should be set up such that the redundancy they add cancels out the negative reliability the striping adds (and then some).

Another note in regards to RAID-10 and the ZFS equivalent: You can safely lose 1 to Vnum (number of vdevs) of drives depending on which drives fail. Sometimes the second drive failure causes data loss even if you have tons of drives in your array - sometimes you can lose half your total drives safely before a failure affects your data. It's not the safest RAID level out there. It is still pretty safe, though. You could always use 3-way mirrors, but obviously that is VERY inefficient. RAID-6/Z2 and RAID-Z3 are more reliable as far as drive failures themselves go, but are lower performance, especially with random writes. But parity-based RAID levels have other drawbacks such as being harder to recover outside of the environment in which they were created, and also you can't just pull a drive out and use it separately like you can with mirrors. None of this is unique to ZFS. But I've split mirrors by removing a drive from each mirrored vdev, creating a new array, copy my data from the old degraded array over, deleted the old array and then added the old drives to the new array. (Sorry for my mixing of generic and ZFS-specific terminology there.)

 

[scruffters]

Thanks again for the help dandragonrage.

I think I am starting to understand the concept of expansion a little better.

In short, there is no way to readily increase the 'usable space' available to a vdev (without starting over). However, one can add additional groups of drives at any time, which will improve performance/redundancy.

Most of all, it seems that the best bet will be a 10 drive Z2 over 3TB or 4TB drives. This will provide the required capacity of 24TB (or more) in a single vdev. On the negative, this will take some time to resilver - although probably no worse than using a hardware controller.

 

[dandragonrage]

Top level is pool (volume). You can expand it by adding vdevs (individual drives, mirrors, or drives w/ parity) which is almost always sufficient. You cannot modify parity-based vdevs though you can add mirrors to individual drives or existing mirrors. You cannot convert a 3-drive RAID-Z to a 4-drive RAID-Z. You cannot convert a 5-drive RAID-Z to RAID-Z2. etc. Each vdev is striped so multiple RAID-Z vdevs will be a RAID-50 equivalent, multiple RAID-Z2 vdevs a RAID-60 equivalent, etc. You can also use combinations of different types of vdevs if you want. You can have a pool that is part RAID-10, part RAID-60 if you want.

ZFS is thus stricter than most hardware RAID cards, but still sufficient for almost any use case. It is also less strict in some ways than RAID cards, though also not in ways that are all that important.

Only time I've needed to recreate the entire pool (by pulling one drive out of each mirror as described in the last post) is when I created the pool with bad options that can't be changed after creation, such as ashift and normalization.

 

[scruffters]

OK - If you'll indulge my curiosity a little further...

 

[scruffters]

Top level is pool (volume). You can expand it by adding vdevs (individual drives, mirrors, or drives w/ parity) which is almost always sufficient.

So one could theoretically have a single Z2 pool of 6x 3TB (12TB usable).

Later it would be possible to add another 6 drive group, which will give update giving a single 24TB volume?

...I imagine my interpretation is incorrect here.

You cannot modify parity-based vdevs though you can add mirrors to individual drives or existing mirrors. You cannot convert a 3-drive RAID-Z to a 4-drive RAID-Z. You cannot convert a 5-drive RAID-Z to RAID-Z2. etc.

I read that folks criticise ZFS for this, but it doesn't really seem that useful to be able to change parity type anyway.

 

[dandragonrage]

So one could theoretically have a single Z2 pool of 6x 3TB (12TB usable).

Later it would be possible to add another 6 drive group, which will give update giving a single 24TB volume?

You are correct.

 

[scruffters]

That is really quite something.

Previously we would have had to use LVM or the features integrated on the hardware controller (then risk expanding the filesystem to use the additional space).

Most of all its nice to be able to split the purchase of the drives - which would otherwise be a fair chunk all in one go.