Best way to make a program "backup friendly"

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
One major issue with lot of programs that keep files open is how to backup it's data properly without causing corruption.

I want to implement a "backup" folder to my application, then the backup job would get data from that folder, and not the live data.

However, there's still the possibility that the backup runs while my program is in the middle of doing it's own backup. What is the best way to make this a safe operation?

I'm thinking, if I do the backup to a temp folder, then do a move to the backup folder, will that be safe? My app would be using system commands like cp and mv.

In the corporate world most apps that do this will backup maybe once a day at a certain interval, then the backup job is set so it does not conflict, but I want to make my app not rely on a setup like that, if possible.
 
I'm also interested in hearing others feedback on this thread.

But a few questions I have include:
- What technologies is your application using?
- What kind of files would be backed up?
- What are the file size(s) of the backup?
 
It's not really using anything special, just fstream. The files are just, well, files. There are various containers, each has an index file and data file. There's also a master index of all the containers. The file size will grow over time and probably not get smaller as data will always be added to it, it's rare that it will be deleted. So it might start off at a few 100MB then in a couple years it could maybe reach a few GB and so on. Like anything else, I guess. It will be housing data for a game server, so the more players, the more items, the more mobiles and so on, the bigger this db will be.

So a copy will take longer over time, but I expect a move to always take the same amount of time, assuming it's always on the same disk, which it will be.
 
I'd recommend using some sort of semaphore. In this case your primary application could create a temp file (ex: .backup_running) before it begins generating the backup folder and delete the temp file when complete. The backup job could then wait until the temp file is deleted before beginning.
 
Rather than implementing a half baked and obnoxious backup system yourself, just avoid the issue and use an existing backup sstem.
 
I'd recommend using some sort of semaphore. In this case your primary application could create a temp file (ex: .backup_running) before it begins generating the backup folder and delete the temp file when complete. The backup job could then wait until the temp file is deleted before beginning.

I'm actually doing that now for the current system. I was thinking of maybe doing the same thing. That makes it so I need to code something extra in the backup script though, so I'm trying to simplify this, but if the move idea does not work, then I might just stick to that.


Rather than implementing a half baked and obnoxious backup system yourself, just avoid the issue and use an existing backup sstem.

I'm not coding a backup system. This is just an app that produces data which needs to be backed up regularly, and it often has open file access to said data. I use rsync for backups. I just don't want my backup job to conflict with the program's access to the data.
 
One major issue with lot of programs that keep files open is how to backup it's data properly without causing corruption.
There shouldn't be any corruption. One program has write access, the other has read access.

And use your operating system's temp folder for temporary files.


Are you talking about backing up the whole hard drive or just making copies of save data?
 
Do you need to handle raw files? If you use a DB + transactions, you can use existing backup tools to make sure that you always end up in a consistent state. Another, surprisingly common, way to handle the problem is to require daily downtime for backups.
 
Back
Top