• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Backup Drives - Checksums?

S

s0rce

Limp Gawd
Joined
Jan 17, 2011
Messages
495
My boss wants a bunch of data backed up to an external hard drive, about 700GB. We have most of the data also on an external raid array on a server in the office so this is strictly archived backup for a project that has been finished.

I was wondering if there is a good way to store some sort of parity information on the drive or at least storing file checksums so you can know if something is wrong with a file. I

I thought about par2 but it seems that this will require archiving all the data into rar/zip to then make the par2, is there a better way?

I figure I could just use md5deep to write a list of all the file checksums so that if a file is broken then the correct version could be restored from either source.

We are using OS X and the drive is formatted withHFS+.

Any ideas would be great.

Thanks!
 
Last edited:
http://tenscomplement.com/

ZFS for OS X

Format external drive as ZFS, set copies = 2 (will require 1400GB of space).

Will give you two copies of everything in there, with built in checksum (run a scrub to force a check) - so you are protected from a bad sector, just not a failed drive. Or just leave copies = 1 and you will still know if you have a checksum failure.
 
ChrisBenn said:
http://tenscomplement.com/

ZFS for OS X

Format external drive as ZFS, set copies = 2 (will require 1400GB of space).

Will give you two copies of everything in there, with built in checksum (run a scrub to force a check) - so you are protected from a bad sector, just not a failed drive. Or just leave copies = 1 and you will still know if you have a checksum failure.
Click to expand...

That's what I would do, well I would just use ZFS on Solaris like I do at home. However, my boss would be too confused and not know how to open the files with something like this as I'm sure he will try to access the files from the backup instead of the server.

I need something invisible to the casual user but can be used to recover from problems should they occur.
 
If you install the tenscomplement stuff on his box it will "just work" :)

But otherwise you just have to generate PAR files for everything - they don't have to be in a RAR format - you can have side by side par files - so for FILENAME.EXT you will have a FILENAME.PAR

http://gp.home.xs4all.nl/Site/MacPAR_deLuxe.html

seems to be a GUI with automates the command line portion.

Only caveat is that if he modifies the files on the backup the par will have to be regenerated - and you will have to explicitly run the program to recover the data/verify files.
 
I tried the other day with QuickPAR, and indeed it will take any files, it just needs several files. The problem is that's it's very intensive work to do.
 
I've looked at par2, the main issue with it's speed is the code hasn't been optimized to use any new cpu features in the last 10 years.

It needs mmx/sse and threading support added.

I'm working on a tapebackup solution, and want to use par2 to handle any errors on the tape, but lots of programming I need to do, as currently, it takes par2 on a good cpu, 15days to run.
 
Hmm, issue is both are windows only.

I did see phpar2 though, and even it's code is old. But that is what gave me the idea to update the routines into more modern cpu instructions for good speedups.
 
If you can make a faster PAR2 I will be very happy :) I thought you were using windows, for UNIX like systems including OSX there is par2+tbb http://chuchusoft.com/par2_tbb/ which seems to do basically what you're talking about. Looks like they're also working on a version that uses nVidia CUDA to assist.
 
Oh, that looks more promising, solves the threading issue atleast, from first look.

I'll have to dig into it when I have time, (that is atleast a month away)
 
You must log in or register to reply here.
Back
Top