• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Data integrity vs. uptime

TeeJayHoward

TeeJayHoward

Limpness Supreme
2FA
Joined
Feb 8, 2005
Messages
13,339
Previous setup:
3x3TB in zpool "pool"
2x3TB in zpool "backup"
1x3TB unused (I was about to upgrade "backup")

I noticed that some of my downloads to "pool" were being corrupted. Running a scrub returned several files which were permanently damaged. I restored from my "backup" pool and thought nothing of it. The next day, the same thing happened. Read and Checksum errors out the wazoo. Downloads had a 50/50 chance of being corrupted by the time they finished. All the errors came from one drive. I figure that "pool" is gone. I destroy it and recreate it as the following:

Current setup:
4x3TB in raidz "pool"
2x3TB in zpool "backup"

Hopefully, setting it up as a raidz will prevent me from needing to repopulate "pool" from backup again if the drive fails. I can just rip out the bad drive and replace it. Unfortunately, it also shorts me a drive for my backup solution, and I'm out of ports on my motherboard. I currently have about 4TB of data. At my current growth rate and configuration, I will exceed the capacity of "backup" in a few months. And yes, I am investigating the source of the corruption. I have replaced the cable going to that one drive, and replaced the quiet fan with a higher CFM one in hopes that it's not a bad drive. If the issues continue, I will replace the disk.

Data integrity is very important to me - it would take me years to recover some of these files. At the same time, I don't want to waste another day without my primary array because a drive failed. And then there's the whole running-out-of-space issue. So what should I do? Do I purchase a standalone NAS, move my backups over to it, and add another disk or two? Do I build a duplicate of my existing NAS and move my backups over there? Do I buy a controller and add another disk to my already hot (and cramped!) box? Or is there a way for me to keep my existing data and somehow ensure both integrity and uptime without purchasing hardware?

edit: Wonderful. Looks like I've got a bad drive.
Code: 
root@solaris:~# zpool status -x
  pool: pool
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
        repaired.
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        pool        DEGRADED     0     0     0
          raidz1-0  DEGRADED     0     0     0
            c5t0d0  ONLINE       0     0     0
            c5t2d0  ONLINE       0     0     0
            c5t3d0  FAULTED      1    59     0  too many errors
            c5t4d0  ONLINE       0     0     0

errors: No known data errors
 
Last edited:
Since you mentioned that it would take you years to replace some of your data, since it isn't clear in your original post if the two pools are in different machines.....My first suggestion will be to move your backup to a completely separate machine/location and NOT have it reside in the same machine (if it is). Disk failures are just one calamity that can befall a system... A bad power supply can fry each and every drive in a single system. A bad HBA can scramble the data on every drive. Best to keep them completely separate.
 
I have a 1TB drive that has a 'backup' zpool on it. It always has the two most recent backups. I hot-plug it, do a rename of backup/foo to backup/foo.old, then do a zfs send/recv from my pool (the stuff I care about - some is windows 7 backup stuff I don't care about) to backup/foo. When do, I then scrub the backup pool, export it, and remove it... Forgot, obviously I need to delete backup/foo.old before doing the rename :)
 
danswartz said:
I have a 1TB drive that has a 'backup' zpool on it. It always has the two most recent backups. I hot-plug it, do a rename of backup/foo to backup/foo.old, then do a zfs send/recv from my pool (the stuff I care about - some is windows 7 backup stuff I don't care about) to backup/foo. When do, I then scrub the backup pool, export it, and remove it... Forgot, obviously I need to delete backup/foo.old before doing the rename :)
Click to expand...
I had a setup very similar to this back when I only had 3TB of data. It's simply not practical to do with more than 6TB.

I would like to put my backup pool on a completely separate box. It would allow for expansion... But it's not cheap. I'm not entirely convinced that it's worth the thousand bucks or so it would cost me to get another 12TB of storage. Even if I re-use my 2x3TB backup drives, I'm looking at over $700. I'm hoping to find a cheaper method of doing things, but if it comes down to it, I may have to pick up a standalone NAS or build a little Atom box.
 
Why not an external JBOD chassis which you plugin, "zfs send" to, and then plugout the chassis?
 
brutalizer said:
Why not an external JBOD chassis which you plugin, "zfs send" to, and then plugout the chassis?
Click to expand...
Can you recommend one? Specifically, I'd be looking for one which supports at least 3 drives over 2TB in size. I'd prefer one that's a standalone NAS, because I don't have USB3 on my box, and 20MB/s is a several day long backup.
 
Look for the SE3016 on ebay. I scored one for $150 (plus another $60 shipping). 16 hot-plug 3.5 inch bays. The specs *say* it supports 3TB drives... There is a thread here about that. Specifically, it takes an 8088 cable, so you'd need an adapter for that to connect to the HBA, which will generally provide 8087 port(s). I got an adapter that screws into one of the backplane slots (nothing to plug into the mobo) that has an external 8088 port and an internal 8087 port. Then I got a short 8087/8087 patch cable.
 
TeeJayHoward said:
Data integrity is very important to me - it would take me years to recover some of these files. At the same time, I don't want to waste another day without my primary array because a drive failed. And then there's the whole running-out-of-space issue. So what should I do?
Click to expand...

Later you say you only want to spend $700. But you never say what the data is worth. (I have lots of datra that would take years to recreate. But most of it is worthless.)

I would use external drives, USB3 or SATA. Drop the right controller card into the computer. Buy enough 2TB hard drives to hold the data plus one or 2 to do incremental backups.

I have a Thermaltake device that lets me plug in bare drives and Vantec cases. Both work well but solve different problems. Botha are cheap.
What I suggest is certainly cheaper than recreating your data. But you data may not be worth recreating.
 
GeorgeHR said:
Later you say you only want to spend $700. But you never say what the data is worth. (I have lots of datra that would take years to recreate. But most of it is worthless.)

I would use external drives, USB3 or SATA. Drop the right controller card into the computer. Buy enough 2TB hard drives to hold the data plus one or 2 to do incremental backups.

I have a Thermaltake device that lets me plug in bare drives and Vantec cases. Both work well but solve different problems. Botha are cheap.
What I suggest is certainly cheaper than recreating your data. But you data may not be worth recreating.
Click to expand...
My data is pretty much worthless. Most of it has not been accessed in years. Much of it will not be accessed for years. Despite all that, I still want to keep it around just in case. I didn't even think about those BlacX docks. One of those and a USB 3.0 card would work great, assuming that they're Solaris-compatible.
 
I would be leery of a usb HD with solaris - too many issues people have had with the usb controller flaking out and resulting in corrupted data. ESATA is much safer...
 
I use eSata docks (I got two) connected to my windows machine and use microsoft synctoy to back up each of my OI shares from individually to individual hdds once a week. Since sync toy can track the changes after the initial syncing it goes very quickly. I disconnect the drives from the docks afterwards and store them, just incase an a power surge or spike gets in and destroys the PSU and drives. Just to give you an idea, I have a Raid Z2 (6x2TB) and back it up with 4x2TB drives. Very easy and cheap to implement. I get an added benefit when I go and double my capacity to (Raid Z2) 6x4TB drives because the old 6x2TB drives can cover the additional space.
 
I keep all my primary data on a big ZFS RaidZ2 pool, and I keep backup's of that on a box running windows using software mirroring on 2 arrays of 2 drives each.
 
brutalizer said:
Are you backing up ZFS data to Windows NTFS? So your backup is unsafe NTFS?
Click to expand...

Unsafe is relative, some people would consider that having data only on the ZFS filesystem (primary and backups) is unsafe since recovering from ZFS filesystems with problems is nearly impossible.
 
how would data backed up on NTFS be unsafe..

keeping all your data one 1 single location is unsafe, having multiple locations of data is saf"er" no matter what the underlying OS is..

i have 3 storage boxes all running windows as opensolaris refused to play nice with my domain and i have not had any problems in 3 years.. knock on wood.
 
MrGuvernment said:
how would data backed up on NTFS be unsafe..

keeping all your data one 1 single location is unsafe, having multiple locations of data is saf"er" no matter what the underlying OS is..

i have 3 storage boxes all running windows as opensolaris refused to play nice with my domain and i have not had any problems in 3 years.. knock on wood.
Click to expand...

brutalizer sees anything that is not ZFS/ECC based as substandard and so unstable as to have bitrot breed like a bacteria.
 
mwroobel said:
brutalizer sees anything that is not ZFS/ECC based as substandard and so unstable as to have bitrot breed like a bacteria.
Click to expand...
Almost true. After reading research papers on data corruption, I (and many others) believe that you should use a storage solution which is built to protect against all forms against data corruption.

There are research papers that shows NTFS is unsafe. There are research papers that show that ZFS is safe.

Whom do you trust, researchers or common hear say?
 
or usage over years with no corruption yourself :)

i wont deny, ZFS is a great filesystem, but when things go wrong.. it isnt for the faint of heart...
 
MrGuvernment said:
or usage over years with no corruption yourself :)
Click to expand...
How do you know that your data is not corrupted?

Something like this:
CERN did a study where they wrote a specified bit pattern to 3.000 hardware raid racks. Something like "010101010101...01" over the whole raid. After three weeks of constant writing, everything was fine and CERN did not saw any error reports. Then, CERN checked the raid to see if the bit pattern still was "010101...01" - and guess what? In 500 circumstances, the bit pattern differed from "01010101...01". CERN got silent corruption. Here are all the details:
http://storagemojo.com/2007/09/19/cerns-data-corruption-research/


i wont deny, ZFS is a great filesystem, but when things go wrong.. it isnt for the faint of heart...
Click to expand...
When things go wrong - that is why you have a backup, which you do, right?

(Just kidding, I myself have no backup but I feel much safer with ZFS than other solutions, after reading all the research)
 
You must log in or register to reply here.
Back
Top