Yikes, glad I did a backup audit

R

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
Decided to audit all my system backups to make sure I'm not missing anything. Well it turns out I was NOT backing up any of my code. The only code that was being backed up properly was my UO server's development + world database. It was a good backup but it had been setup specificity for that one app. Everything else in my dev/prod folders was not being touched. :eek: I retired my UO server so I archived all that stuff and stopped running that job.

The only way the code was being backed up is when I do a removable disk backup, those backups were catching those files. I tend to do those backups no more than once or twice a month as they are strictly for in case of raid failure etc. so not sufficient if I accidentally delete a whole folder while in the middle of working on code or something.

So yeah, just finished writing a shell script for that. Glad I caught this before anything bad happened.

Let this serve as a PSA to everyone, check your backups, and make sure you're actually backing up the stuff you want to backup! :p
 
Thanks for the reminder. Been a bit behind on my own backups as well
 
Oh the amount of customers who never test there backup environment. As you have experienced, testing and making sure your backup is actually good is at least as important as taking the backup itself.

A backup is not a backup if you cannot restore it, then it's just a high blood pressure trigger.
 
SantaSCSI said:
Oh the amount of customers who never test there backup environment. As you have experienced, testing and making sure your backup is actually good is at least as important as taking the backup itself.

A backup is not a backup if you cannot restore it, then it's just a high blood pressure trigger.
Click to expand...
Unfortunately it's not always possible. If you have very limited resources and do not have extra equipment to test the backups with, what can you do?
 
bigdogchris said:
Unfortunately it's not always possible. If you have very limited resources and do not have extra equipment to test the backups with, what can you do?
Click to expand...
Get the equipment.

An untested backup isn't a backup. It's just that simple.
 
bigdogchris said:
Unfortunately it's not always possible. If you have very limited resources and do not have extra equipment to test the backups with, what can you do?
Click to expand...

Yeah I find this stuff is always 10 times harder in a corporate environment. At home like in my case, you just go get what you need, no questions asked. At work, it's all about budget and saving money etc and if you have an IT manager that is a douchebag then you're pretty much not allowed to touch anything ever and definitely wont approve to buy new stuff unless they think they need it.

For backups I guess it's as simple as just trying to restore your own home directory somewhere to make sure the process works. That's another thing I'm not a fan of, backup software that is very convoluted to use and does not let you restore files to an alternate location. We used Backupexec when I was doing server support and it was a pretty decent program. We used Tivioli before that, and wow you need a PHD to run that. LOL. We always cringed at the thought of ever having to do a restore because nobody knew how to do it. Pretty sad lol. Eventually the backups just plain stopped working... was nice when we upgraded to the new system.
 
Red Squirrel said:
Yeah I find this stuff is always 10 times harder in a corporate environment. At home like in my case, you just go get what you need, no questions asked. At work, it's all about budget and saving money etc and if you have an IT manager that is a douchebag then you're pretty much not allowed to touch anything ever and definitely wont approve to buy new stuff unless they think they need it.

For backups I guess it's as simple as just trying to restore your own home directory somewhere to make sure the process works. That's another thing I'm not a fan of, backup software that is very convoluted to use and does not let you restore files to an alternate location. We used Backupexec when I was doing server support and it was a pretty decent program. We used Tivioli before that, and wow you need a PHD to run that. LOL. We always cringed at the thought of ever having to do a restore because nobody knew how to do it. Pretty sad lol. Eventually the backups just plain stopped working... was nice when we upgraded to the new system.
Click to expand...
This is why I use linux for my file servers. Backup to NAS using a clever combination of tar, bz2 and bash.

Even have the backups integrated with nagios, so I am alerted as to when something fails. Then I do a quarterly restore on our backups, to verify the process. Simply browse to the backup location, open the set in 7zip and restore. I have even automated this and plugged it in to nagios, so if I don't get around to it, the system will handle it automatically and alert me if something is fooked.

I dearly love nagios.
 
But lets assume the buildings gone and servers and all equipment in it is gone. As the It guy whats your plans?
 
TCM said:
Don't compress backups. A single bit flip takes out a whole block size worth of data.
Click to expand...
Which is preferable to not backing up at all. My data sets tend towards the large side of things ( for a small business ), and I work in a budget constrained shop where I am not going to get a couple 10TB data devices to fiddle with.

The loss of a couple megs of data is a preferable risk. Especially when that data may exist in a previous backup set.
 
Most of my backups are full file copies via rsync, but I do have a couple jobs that compress, like the SQL backups. My offsite backups are also compressed because even compressed it takes about a day to run that job. Oh how I wish we'd get fiber here one day. :p
 
You think backing up a single corporate environment is a pain, wait until you're managing backups for thousands upon thousands of enterprise routers of varying make and model across the continental US, many of which are from previous acquisitions and use different login schemes, and/or TACACS, etc.

I'm trying to get everything standardized, but the number of servers involved in the backup process are all managed by different people on different teams, all of whom seem to feel that the problem is not worth their time.

I've got various IT heads behind my initiative, but even they seem unable to get anyone to put more than a few minutes of time into the problem.
 
You must log in or register to reply here.
Back
Top