Rsync with Version Control - Backup

[Asgorath]

I have a file server that houses our company files. Mostly word and pdf docs. All the files are nested under a single folder root.

How can I 'echo' these files to another server in a fast efficient manner (millions of files) while at the same time giving myself some sort of version control?

I was looking at using Deltacopy for transferring the files. But it would be great if I could get many point in time snapshots of those files. Shadow copy is great, but it is limited to 64 snapshots. I've tried Symantec Continuous protection server, but it bogs down under the load of keeping track of that many files. I have an online backup vendor that does delta copies on all the files and folders and it is wonderful. How do they do that?

Thanks,

Greg

 

[Lazn_Work]

If you are on Server 2003 R2 or newer, and have enough free space use DFSR and Shadow Copies. You really need more than 64 copies of each file?

edit:
I am doing this with 13 servers all synched and two of them have shadow copies enabled for quick and dirty file version restores. (not a real backup, but a restore solution)

http://msdn.microsoft.com/en-us/library/bb540025(VS.85).aspx
http://technet.microsoft.com/en-us/library/cc785914.aspx

 

[Asgorath]

If you are on Server 2003 R2 or newer, and have enough free space use DFSR and Shadow Copies. You really need more than 64 copies of each file?

edit:
I am doing this with 13 servers all synched and two of them have shadow copies enabled for quick and dirty file version restores. (not a real backup, but a restore solution)

http://msdn.microsoft.com/en-us/library/bb540025(VS.85).aspx
http://technet.microsoft.com/en-us/library/cc785914.aspx

It's not that I need 64 copies, but I want the copies similar to:

Rotating 4 copies per day
7 daily copies
16 weekly copies
Monthly Copies for 2 years

which means
4 + 7 + 16 + 24

but in order to do that I need to retire the shadow copies as appropriate, which I don't know how to do.

 

[Lazn_Work]

It's not that I need 64 copies, but I want the copies similar to:

Rotating 4 copies per day
7 daily copies
16 weekly copies
Monthly Copies for 2 years

which means
4 + 7 + 16 + 24

but in order to do that I need to retire the shadow copies as appropriate, which I don't know how to do.

I do a separate backup, and just use shadow copies for convenience as I don't want all my data to exclusively reside on live servers, one lightning strike and that will all be gone.. So I backup to tape daily for my archiving purposes and put a monthly in to permanent storage..

 

[Steve01S4]

This can be done easily using rsync with hard links. That's exactly how my current backup system works. Using bash scripts, you could throw it together in an hour, easy. I'm a *nix guy, so I don't really know if there's a way to script it on Windows without Cygwin or some equivalent. Look up rsync with the --link-dest option, and you should find some ideas. If you feel like using Cygwin I'd be happy to help you get the script figured out.

 

[Asgorath]

This can be done easily using rsync with hard links. That's exactly how my current backup system works. Using bash scripts, you could throw it together in an hour, easy. I'm a *nix guy, so I don't really know if there's a way to script it on Windows without Cygwin or some equivalent. Look up rsync with the --link-dest option, and you should find some ideas. If you feel like using Cygwin I'd be happy to help you get the script figured out.

I think this will be the best set up. Would you help me with that script?

That'd be great.

Thanks.

 

[Boomslang]

Another option from the Linux world is rsnapshot. It doesn't move bits of files that haven't changed, and doesn't keep identical files on disk - rather just inserts a hardlink.

rsync for windows: http://www.itefix.no/i2/node/10650
rsnapshot http://www.rsnapshot.org/

It's probably identical to what the poster above me suggested, but instead of having to script things out you can just follow the existing config file. It uses rsync.

 

[irrision]

Or of course Solaris ZFS supports fully automated snapshots doing exactly what you are asking for: http://www.sun.com/software/solaris/howtoguides/zfs_snapshots.pdf

Just got this setup on a home fileserver with 5 x750GB Sata and its unbelievably easy to implement and pull restores from using the time slider in gnome. You can also pull copy of write clones of snapshots if you want to get a writable version of a snapshot.

 

[Steve01S4]

I had forgotten about rsnapshot. It does the same thing my scripts do, only differing in a few details. If you're not too familiar with scripting on Linux, it's probably best to look into rsnapshot first to see if it fits your needs, since it's well documented.