Incremental File Versioning

[GreenLED]

It's been a while since I have posted. I think becoming a father soon has something to do with it . Anyhow, I am having a very tough time solving a simple problem and I thought I would ask in this forum.

My goal is to have an automated backup for all of my clients who I work for that automatically creates file versions on the fly. Meaning - when a file is saved or on a regular basis (plus if the file has changed) the system automatically saves differential data so you can look at previous versions of that file. Here's the problem, all of the applications that I've looked at do not do this. AutoVer is a fantastic piece of software and it works flawlessly, however, it does not accomplish what I want. It makes a copy EVERY time you save. I want this behavious (creating a new version every time a file is saved), BUT I don't want the EXACT same bits saved AGAIN, that's redudant and very space draining. Imagine if you saved your 10 G final cut project every ten minutes!!!!

So, I guess what I am looking for is some recommendations on what program might do this for me. So, to review.

The program should ...

1. Be close to free or low priced.

2. Monitor and Sync files between folders or drives -- w/e

3. Store ONLY differential data (thus saving space)

4. Run on Windows - because every one of my clients uses Windows, Mac would be a plus, but not required

. . .

Here's a product I just found that seems to do this, but it's like $60 - ouch. Still looking. Any help is apreciated.

https://www.syncovery.com/

 

[zrav]

CCollect uses rsync and symlink magic to do differential backups while making them appear as complete point-in-time backups. But I'm afraid it won't work in Windows although there is a rsync port for it.

For straight syncing without versioning/point-in-time recovery look at vanilla rsync or SyncToy.

If you need better you'll have to pay (you talk about clients and $60 is already "ouch"?). Tbh it sounds like a use case for a NAS with rolling snapshots...

 

[danswartz]

If your NAS runs an opensolaris derivative of ZFS (say omnios?) and you share a dataset to windows users, if you do periodic snapshots on that dataset (say via a cron job), windows users can use the 'previous versions of this file' functionality automatically...

 

[_Gea]

If you want a painless solution for Windows clients and do not want duplicate files or a file-copy after modifications you should look at Copy On Write filesystems with snapshot capability that offers a Windows Previous Version integration.

Your best options are mainly a ZFS/Solaris based SMB fileserver like OmniOS or a Windows SMB Server with ReFS Filesystem. From these two, ZFS is free, faster, very robust and offers more options.

 

[faugusztin]

How about Sparkleshare ? It backups to a git repo.

 

[GreenLED]

I want to take the time to thank all of you for putting in the effort to answer my question. I am considering all of your comments. I knew this was a community I could count on!

Let me give some more background on the situation.

Firstly, thank you for the suggestions to use ZFS. In fact, if you look on this forum for "Orbit" you'll find a post about a file server that I am in the process of building. I fully intend to use ZFS as the SOLE antidote for this problem, but in the meantime I need a stop-gap measure that can tide my clients over until I get all of that system polished and ready for prime time.

I had high hops for "GoodSync", but it failed. It creates previous versions of files just like all of the other crappy previous versions implementations out there -- a simple copy of the same file - stupidly inefficient !!! By the way, AutoVer works MUCH faster than GoodSync on side by side testing, so if you need to use a syncing tool that's freeware, I would recommend that piece of software. It really does the job well and for $0.

I'm starting to realize how badly ZFS like technology is needed in this industry. I'm still looking for a system that will ONLY write CHANGED bits, not re-write old ones. This is NOT a problem for small doc files which no one cares about. I'm dealing with video editors that might have four or five versions of a video. I don't want a 3 GB file rewritten to disk five times! Not only is that inefficent, but it burns through bandwidth - which is a limited resource.

I'm now shifting my focus to the VSS service in windows. If you think I'm headed in the wrong direction, stop me, but this is my next point of interest at the moment.

Oh, and as far as why $60 is so "ouch" for my clients ... I charge approximately $15 per client for a cloud backup type service (there's a lot more to the service than just this, but not really the scope of this post). So, if I purchased a $60 product for each client, that means I would have to wait five months before I get ANY profit - that's a terrible business model. Anyhow, that was my thinking behind that.