Backup with checksum

csnv

Limp Gawd
Joined
Nov 11, 2009
Messages
205
Hi everyone,
I'm having hard time finding software (open source is even better) that would do below:
0. create checksum snapshot for new items under "SRC" folder
1. checksum on "SRC" folder against snapshot. If fail, checksum same item in "DEST" folder, if pass copy over.
2. mirror SRC to DEST
3. checksum on DEST, if anything fail, recopy from SRC

basically, per schedule, it would mirror the SRC and at the same time checksum all files in both SRC and DEST. My goal would be having a set of data on a separate machine with corruptions/failures minimized.
 
Last edited:

drescherjm

[H]F Junkie
Joined
Nov 19, 2008
Messages
14,740
I assume you are talking about some version of Microsoft Windows and are not on linux (or others) or using zfs.

Your request sounds like what zfs send and receive is.
 

drescherjm

[H]F Junkie
Joined
Nov 19, 2008
Messages
14,740
There are utilities like FreeFileSync and Unison that may do what you want on windows.
 

csnv

Limp Gawd
Joined
Nov 11, 2009
Messages
205
There are utilities like FreeFileSync and Unison that may do what you want on windows.
Looks like FreeFileSync using Time and Size to determine if a file is changed. Many utilities do that, but i'd like to have a tool that uses checksum instead.. to make sure the data not corrupted for any reason.

looking at unison now...
 

csnv

Limp Gawd
Joined
Nov 11, 2009
Messages
205
There are utilities like FreeFileSync and Unison that may do what you want on windows.
Unison "simply looks at each file's inode number and modtime; if neither of these have changed, then it concludes that the file has not been changed." It doesn't do checksum either.
 

iroc409

[H]ard|Gawd
Joined
Jun 17, 2006
Messages
1,320
rsync will do what you are asking for, and I there is a Windows version or two floating around somewhere. I believe it uses checksums when actively moving files, but to do the comparison you have to invoke the checksum. In "normal" mode it just uses timestamps.

If you're willing to do it manually, you can do the same thing with TeraCopy, which is native to Windows. I guess what you'd do is do a manual copy with verify (checksum based), skip all similar files and let it perform the post-copy checksum verification.

ETA: Sorry, I don't think either of these will check against a snapshot. You'll need something else for that.
 

grim4593

Limp Gawd
Joined
Nov 30, 2014
Messages
260
Looks like FreeFileSync using Time and Size to determine if a file is changed. Many utilities do that, but i'd like to have a tool that uses checksum instead.. to make sure the data not corrupted for any reason.

looking at unison now...
In FreeFileSync if you select the blue gear next to "Compare" it will give the option to identify by file content (e.g. checksum). Note, it will take much longer to compare since it has to hash every file.
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
3,933
If you can put your data on ZFS (filer, smb/nfs-share, local filesystem, iscsi lun) all your data and metadata has full realtime checksum protection with data versioning via readonly snaps. You can keep a ZFS filesystem in sync with another filesystem on same or different server with ZFS replication. This gives full checksum (and encryption when wanted) protection on data transfer and on the backup system.

This is far superiour to any checksum, copy, sync tool on top of a non-checksum filesystem. It even keeps open files in sync and works even on a high load Petabyte system with a sync delay down to a minute.
 
Last edited:
Top