Does this setup work with flexraid or snapraid?

[Q-the-STORM]

I'm looking into flexraid and snapraid atm...

current setup:
I got 12 drives atm, 1x1TB, 5x2TB, 4x3TB, 2x4TB all in one HDD tower.
they are connected to three 5x1 port multipliers which are connected via usb 3.0 to my windows 7 PC...
Basically I can access 5 individual drives through 1 USB connection. If I buy a USB3.0 Hub, I could connect all individual drives through one USB3.0 Port.
I can turn individual drives on and off.

current usage:
I usually need 1 drive per day for 5-16 hours, so I only turn on the drives I need, the rest stays powered off. I turn everything off when I go to sleep, I don't want to have them running 24/7 and I don't want to turn every drive on every day if I don't need them. Due to this, I sometimes access some drives only once in 2-3 weeks. I write to them rarely (once in 2-4 weeks), so I mostly just read.
I get speeds of about 50-100MB/sek, depending on the drive.


now to what I am thinking of.. I obviously don't need real-time, so it's gonna be snapshot...
What I basically want, is to use my drives exactly like I use them now, only that I have 1-2 parity drives in case of HDD failure or bad sectors etc, and to verify my drives once in a while to make sure all data is still intact.



1. Still want to use USB3.0, I've read about single USB drives working (with performance loss) but nothing about multiple drives through one connection (which I think *should* act like multiple usb drives on a USB hub) eSATA would be the alternative (I can use eSATA on my port multipliers), but it would not work with my current PC, so I really want to get it working with USB. Would this work with flexraid and snapraid?

2. I still don't want to have all drives powered on. Not even talking about spinning, just powered on. I still want to power them on individually if I need them. From what I've read, this shouldn't be a problem, since there's no striping and I don't want pooling, I still want to see each individual drive. I haven't found anything about snapshots though, I assume if I make a new snapshot or if I want to verify the drives, I have to have all drives powered on. Or can I snapshot/verify individual drives?

3. If a drive fails, do I have to power on all drives on a rebuild even if it's just some bit errors due to bad sectors? (I assume yes, that's basically how raid works ) and if it's just some bit errors, would I still need to rebuild the whole drive or can it fix the single affected file quickly?

4. I read this on the snapraid homepage regarding fixing of silent errors:

Flexraid uses checksums to validate files, but such checksums are not verified when data is read to update the parity. This means that any silent error present will propagate into the parity, making impossible to fix it later, even if it can be still detected comparing the file checksum.
You can get in a state where the "Validate" operation reports errors, but the "Verify" one reports no problem in the parity, making impossible to fix the errors.

can someone elaborate on that a bit? I would like to check for bit errors and fix them, before making a snapshot... it already happened a couple of times that videofiles got corrupted, which resulted in glitches when playing them... having a way to fix those errors is the second most important feature I want (complete HDD failure being the first), but that segment is a bit unclear. In which situation does flexraid not see/fix errors? How does that differ from snapraid?
What exactly would I have to do to insure that there are no errors since the last snapshot on flexraid and snapraid?

 

[vzjrz]

1. Multiple drives should work but ofcourse you'll be limited to the throughput of the usb connections.

2. You can have whatever drives you want on or off but when creating the parity or recovering from the parity you'll need all drives on.

3. You'll need to power on all the drives. In snapraid you can just 'fix' a single file you don't have to rebuild the entire drive.

4. The way I'm reading this, when you call for flexraid to create a new snapshot of your data flexraid will just go ahead and and make a new snapshot of you data without checking if your data is free of errors. Here's a thread on the flexraid forums with users talking about the problem: http://forum.flexraid.com/index.php/topic,3233.0.html Their solution seems to be to run a "validate" on the entire data first to ensures it is error free. Snapraid on the other hand runs this validation as part of its "sync" on only the data that is needed to create a new snapshot. Read this part here: http://snapraid.sourceforge.net/faq.html (What are and how to fix "Data errors" in "sync"?) So in flexraid if you just create a new snapshot with bad data the parity if updated and then if you later "validate" your data you'll know there was an error but you wont be able to fix it since the parity has already been updated with the bad data. Snapraid on the other hand validates the data before making the new parity and if it finds an error stops the whole operation giving the user opportunity to fix it (either with snapraid itself or with an outside program). Note that making a new snapshot in snapraid does not guarantee that ALL your data is error free, only that the data needed to make the new parity is error free. Snapraid has its own command, "scrub", to validate all the data for errors.

 

[Q-the-STORM]

looks good so far then..
I'll go the snapraid route then...

one question, since I can't find anything solid about it on the snapraid site:
do the drive letters have to stay consistent?
Or do the drive letter always have to stay the same for each drive?
Usually the drive letters don't change, but it has happened from time to time...
I'd rather not use volume mount points.

also can I move files on a drive without having to recompute parity? how about moving it to another drive (still on the array)?

 

[bexamous]

It is recommended to use volume mount points. Snapraid has some file that is like:
disk d1 D:\
disk d2 E:\
disk d3 F:\
parity G:\
q-parity H:\

If disks change letters snapraid will not make sense of it and either you'll get some failure right away, or a bajillion data errors. Snapraid seems to do some simple checks before giving you a bajillion data errors. Eg if it sees D:\ is an empty drive it'll immediately go "D: is empty, are you sure this drive is plugged in?"

In any case, if drive letters change you can simply update this file to match the disks current letters, but it's easier to just use mount points.

Also moving files is enough to recompute parity. Moving a file will look like you deleted some file and then added some new file.

 

[JoeComp]

If disks change letters snapraid will not make sense of it and either you'll get some failure right away, or a bajillion data errors. Snapraid seems to do some simple checks before giving you a bajillion data errors. Eg if it sees D:\ is an empty drive it'll immediately go "D: is empty, are you sure this drive is plugged in?"

In any case, if drive letters change you can simply update this file to match the disks current letters, but it's easier to just use mount points.

Also, SnapRAID checks the disk UUID in linux to see if it has changed. I'm not sure if there is an equivalent ID in Windows. But the point is that SnapRAID will warn you if the data drives have changed ID, at least in linux.

Since SnapRAID does not write to data drives (unless you explicitly configure a content file on a data drive, or do a `snapraid fix` operation), it is not practical for SnapRAID to determine which data drive is which (perhaps it could go by UUID, but I am not sure how robust that would be). You could always do it manually, by creating a file or folder on each data drive specifying the drive label from the SnapRAID config file. Then if the drive letter changes, you could edit the SnapRAID config file to point each label to the new drive letter.

But I think using volume mount points would be better. That is what I would do if I were running SnapRAID in Windows.