How do you Backup your ~10TB rigs?

hotzen

Limp Gawd
Joined
Jan 29, 2011
Messages
349
I wonder what to use to properly backup at least some Terabytes of a running NAS-box?
 
Replicate to another NAS would probably be most ideal. Of course that's cost prohibitive for a lot of folks.
 
"cost prohibitive" indeed ;)
everybody always tells you "a raid does not compensate for a backup" but actually backuping a NAS seems to be a whole different job...
 
could go LTO4 tapes, too... use another RAID setup and another set of drives... something like that.
 
if you've got time, just burn it to single sided DVD's. only take a couple thousand discs and approx 6 months non-stop to complete. i didn't factor in toilet breaks, naps or meal times. perhaps add a couple of days on top for that...
 
So it seems none of you is actually doing backups? ;)

Is it possible to use ZFS-Snapshotting + RaidZ/RaidZ2 and ignoring to create backups without having nightmares?
 
Main PC has 21TB in RAID5. Periodically, I manually do a resync to an offline backup on a JBOD chassis (Norco 4020) connected by SAS expander, to another set of RAID5s with equivalent storage. So yes, in my case basically 2 RAIDs.
 
If you really have 10TB of irreplaceable data, then at a minimum you should invest in another 10+ TB system for backup. And possibly a third, offsite.

In the more likely case that most of the 10TB data is replaceable, then I think the thing to do is to separate out the irreplaceable data, which can hopefully fit on a single HDD, and then get two HDDs, and backup to those in rotation, with one always kept offline and offsite or at least in a fireproof waterproof safe.
 
Sneaker-net is another option too, particularly for the "replaceable" portion of your data.
 
If you really have 10TB of irreplaceable data, then at a minimum you should invest in another 10+ TB system for backup. And possibly a third, offsite.

In the more likely case that most of the 10TB data is replaceable, then I think the thing to do is to separate out the irreplaceable data, which can hopefully fit on a single HDD, and then get two HDDs, and backup to those in rotation, with one always kept offline and offsite or at least in a fireproof waterproof safe.

I do the second location NAS backup.
 
I have my file server back up to another file server (using rsync on Solaris). Once you get past a certain point, its pretty much the only option.

My next step is to be able to backup off site.
 
Is there any tutorial on how to backup/restore ZFS-snapshots using e.g. an external Harddisk?
 
I mirror everything to a backup server. I know that sounds terribly expensive, but there's a way to do it that makes it a little easier. Every other time I buy a hard drive for one of my computers, I buy another higher capacity drive for my backup server. With 2TB drives going for $60-70, it isn't too bad. If you do it incrementally, then the total cost is spread out over time. Generally, I try to keep my highest capacity drives in my backup server so that i get maximum storage density there using the fewest number of disks. I also use WHS v1 (with data duplication turned off) for my backup server, so rotating different sized drives in and out is very easy. The idea is to only buy drives for your backup server as you need them, which is usually the same time that you buy drives for your other computers (or when you add a new computer to the mix). Doing it over time maximizes your savings because the cost of storage is constantly going down.

The cost of the actual backup server (without the drives) is pretty minimal. I built mine from spare parts. If you have an extra motherboard with 5-6 sata ports, then you are golden. If not, you can get a cheap AM2 board on Newegg or a cheap HBA card (IBM BR10i is a good one for about $40-50 on ebay) and you'll be set for a while. WHS v1 needs very little actual processing power and only 1-2GB RAM. After you set it up, you can throw it in a closet and run it headless. RDP in whenever you need to swap out a drive. I like to use sata/sas backplanes on my backup server to make it easy to swap drives in and out, but that's an unnecessary expense.

As an example, I recently purchased two 320gb drives to use as an OS mirror in one of my servers. Right around the time i bought them, I also got a 2TB drive to rotate into my backup server. That keeps my backup storage capacity ahead of my production storage capacity for a while longer, which is all I need.

I know it sounds expensive to do things this way, but it really isn't. Besides, you really aren't backed up if you only have one copy of everything and it is sitting on a RAID.
 
I have my 36 TB (usable) system backed up to my 18 TB (usable) backup server. I only have about 12-14 TB of data which I really don't want to lose out of the 23TB I currently have on my 36 TB system.
 
Is there any tutorial on how to backup/restore ZFS-snapshots using e.g. an external Harddisk?

zfs snapshots aren't backups, they're pointers to existing data and copies of changed data.

there is no magic way to backup data when you're talking multiple terabytes and more. you either buy expensive tapes or you buy additional harddisks and keep them on a physically separate system and optionally a physically separate location.
 
2TB is about 500 DVDs. It takes about 30 minutes to rip a DVD. So 250 hours of labor. To recreate.

A spare hard drive plus 1000 minutes to copy is a lot cheaper than my labor to rerip.

I just plug a hard drive into an e-SATA device and copy.
 
2TB is about 500 DVDs. It takes about 30 minutes to rip a DVD. So 250 hours of labor. To recreate.

Do you have to continuously stroke the DVD player while its ripping in order to get it to work? Otherwise, it should take about 1 minute of "labor" to rip a DVD, even if it takes 30 minutes for the rip to complete.
 
zfs snapshots aren't backups, they're pointers to existing data and copies of changed data.

there is no magic way to backup data when you're talking multiple terabytes and more. you either buy expensive tapes or you buy additional harddisks and keep them on a physically separate system and optionally a physically separate location.
What about zfs snapshots and zfs send to another box?
 
Do you have to continuously stroke the DVD player while its ripping in order to get it to work? Otherwise, it should take about 1 minute of "labor" to rip a DVD, even if it takes 30 minutes for the rip to complete.
Since most of my time is not spent at my computer, to get the ripping done requires I stop doing other work every so often. Making that other work much less efficient.

30 minutes per DVD seems about right.
 
What about zfs snapshots and zfs send to another box?

Still need another box to backup to, also running ZFS. I'm not sure if there are issues with different zpool versions either. I personally use ZFS snapshots and rsync since it is more universal. Even if I decided I wanted my backup server to run Server 2008, I could schedule synctoy to pull data.
 
I don't backup, which is pretty much the norm for folks using this stuff at home. Unless you have lots of disposable income; it's just not cost feasible.

Granted, I do store my media in two locations, one on my workstation (split up over multiple directories/drives and copied over to my ZFS server to one "share" via robocopy/batch files), and the other data on my server is either stuff I don't care about, or backups of other systems, like a backup of my workstation (using a backup program... ShadowProtect in my case).

I do want to eventually use my existing disks and implement a low powered ZFS-based server that will probably just contain a replication of the media FSes
 
i backup my critical data locally on 2TB rotating external I have 3 of them.... this is done weekly and also online via crashplan.

also critical data is synced to multiple PCs using dropbox, live sync and sync toy.

I have 4 PCs that have all my music, photos and documents on them.... movies are only on my RAID 5 array and external backups.

I only have 6tb of live data and then 6tb of backups.

6x1TB for live
3x2TB for backups + crashplan and multiple PCs
 
I might as well ask this since its a thread about storing massive amounts of data...

Does anyone know a backup service that provides sftp access at a reasonable amount?

heck, I'd be curious if anyone offers it at any cost. It seems like most folks want you to transfer using their proprietary software.
 
I might as well ask this since its a thread about storing massive amounts of data...

Does anyone know a backup service that provides sftp access at a reasonable amount?

heck, I'd be curious if anyone offers it at any cost. It seems like most folks want you to transfer using their proprietary software.

There are inexpensive companies like Bluehost.com that support sftp. It wouldn't be much of a stretch to arrange a backup strategy with companies like them.
http://helpdesk.bluehost.com/index.php/kb/article/000248
 
The most important thing to me is pics and docs. I back those up to DVD, put them in a fire proof safe, and re-burn every 5 years.

Movies and music I'm more worried about disk failure.
 
The only things that I make sure I have thoroughly backed-up, are non-recreatables (things that can't be practically recreated, if needed). Movies and music, millions of people have backups of those. If I NEEDED them, they still exist in this world. Wedding pictures and video for clients, that can be tough. No one really wants to do a second wedding for the sake of documentation. Personal projects and portfolio work, would be a pain in the arse to recreate all of that, especially years and years of work. When it comes down to it though, I honestly only have about a hundred GBs of stuff I really need to keep. Terabytes? Absolutely not.

You should really go through that whole process and figure out if all those terabytes warrant doubling or tripling your current data storage system.
 
I might as well ask this since its a thread about storing massive amounts of data...

Does anyone know a backup service that provides sftp access at a reasonable amount?

heck, I'd be curious if anyone offers it at any cost. It seems like most folks want you to transfer using their proprietary software.

I'm currently starting a backup company that will offer backup/network disks with SFTP. I need a few beta testers so if you're interested send me a private message.
 
I'll have a 12.5TB rig today if I get the desire to shut down my box and unplug all the cables and lift the heavy beast onto my desk (rig is almost 60lbs) and install the drive I have sitting on my desk.
Might install the 4GB of ram sitting here too.

My important stuff is on 3 of the drives, I don't use raid, just individual drives, and almost all of the data is easily replaceable.
 
I still have all my originals so backup isnt essential. A second 16Tb backup server just isnt worth the money for my use.

Raid however doesnt cost much more and offers some protection against hardware failure (Software/virus issues are unlikely on a LAN only Linux box).
 
I currently have a thread on here detailing my ESXi setup with Solaris 11 Express and napp-it running the NAS side of things. Total usable storage currently is 8TB from a raidz array of 5x2tb hard drives. Of that 8TB, 50% is full and only 1TB is critical information that I care about (the rest is movie rips). With that, I have an additional set of 5 x 1TB drives that will be going into the NAS as soon as I become comfortable that the new system is solid. When that happens, the old WHS server will get a set of 3 x 2TB drives in raidz for a usable 4TB of space and will become a secondary backup system housed in my detached garage that's roughly 30ft away from the house. Nightly rsync will occur to offload a copy of the most critical information to the backup server. Wake-on-lan will be incorporated into the job so that I can power down the backup server when it's not in use to save energy and spindle life.
 
I'm surprised nobody uses my method among you guys. Be generous! Help other people (family is best!) set up their own servers, and "give" them your data! All the data that is personal or that you cannot give away like that, backup to online server somewhere else or take HDD copies off-site.

RaidZ-2 and snapshots DO solve a lot of problems in addition to this. For everything else (fires and theft) any backups MUST be off-site.

This has worked for me for years.
 
I'm surprised nobody uses my method among you guys. Be generous! Help other people (family is best!) set up their own servers, and "give" them your data! All the data that is personal or that you cannot give away like that, backup to online server somewhere else or take HDD copies off-site.

I forgot to mention that I do this as well. My friend and I tend to build similar setups year after year. With that, we've backed up each other's data in the past for an offsite copy.
 
I'm surprised nobody uses my method among you guys. Be generous! Help other people (family is best!) set up their own servers, and "give" them your data! All the data that is personal or that you cannot give away like that, backup to online server somewhere else or take HDD copies off-site.

Crashplan is great for offsite storage. In addition to the methods I described earlier, I have CrashPlan backups with two friends that live in another state. It is secure and free. If I ever have a catastrophe, I can ask one of them to mail me the drive for the restoration process instead of downloading 100+GBs. I've been using it for a few years now and it works really well.
 
I don't have 10TB+ of data, only about 3TB, lot of which is actually backups of backups or VMs that are preallocated. What I do is have a removable drive dock, and I have drives devoted to backups and I rotate them every now and then. Each drive has a separate job so the job that runs is determined by the drive in the dock. I have to do this manually but it's as easy as putting the drive in and running a command, then I get an email when it's done. These are more or less for if my raid craps out.

I also have lot of local backups on my main raid array.

For offsite backups, I just rsync the latest and most important data to my dedicated host.

Once I get to the point where I have more data then I will get a bigger capacity hosted server. My main issue though is my upload sucks. I'd almost have to backup on several hard drives and ship it out somewhere or something. Uploading 1GB takes me about 2 days. that criples my internet so I can't use it.
 
The wife and I have been using crashplan for a while (free version). We are probably going to move to the "paid" version family plan soon enough because of how good this service/utility is. I mean $120/year for unlimited cloud storage for up to 10 machines = ROFL. If you go 4 years...$6/month...damn.

I am still making a push towards tape on critical data (photo's, work, etc)..but crash plan really takes need for duplication away from WHS and allows much cheaper "raid" setups. Right now going with onboard RAID5 with 4x3TB's HDD gets you a ton of space at reasonable performance.
 
I'm surprised nobody uses my method among you guys. Be generous! Help other people (family is best!) set up their own servers, and "give" them your data! All the data that is personal or that you cannot give away like that, backup to online server somewhere else or take HDD copies off-site.

RaidZ-2 and snapshots DO solve a lot of problems in addition to this. For everything else (fires and theft) any backups MUST be off-site.

This has worked for me for years.

I forgot to mention that I do this as well. My friend and I tend to build similar setups year after year. With that, we've backed up each other's data in the past for an offsite copy.

The way I do it too, with the addition of encrypted data. ;)
At least a 100mbit link in between is recommended though, unless you sync very often or don't generate new data at a rapid pace...
And be careful if you don't have an unlimited data plan with both involved ISPs. :eek:
 
Production data store is an LSI 9260-4i with (4) 3TB Hitachi drives in RAID 5.
This is backed up to the same machine with the onboard Intel ICH10R/DO with (4) 2TB WD Green drives in RAID 5. I have not reached full capacity, so I don't have to worry about the difference in size yet.

High-priority data will fit on 1-2 DVD's, which I perform every couple months. All personal data is stored in an encrypted TrueCrypt container. All email (gmail) is downloaded to an Outlook PST on a nightly basis to keep an offline copy. In case gmail goes down or is unavailable.
 
The wife and I have been using crashplan for a while (free version). We are probably going to move to the "paid" version family plan soon enough because of how good this service/utility is. I mean $120/year for unlimited cloud storage for up to 10 machines = ROFL. If you go 4 years...$6/month...damn.

Crashplan has a 15% off deal right now, so it is down to $5.09 per month for the 4 year family unlimited plan.

http://crashplan.com/mozyonover
 
The wife and I have been using crashplan for a while (free version). We are probably going to move to the "paid" version family plan soon enough because of how good this service/utility is. I mean $120/year for unlimited cloud storage for up to 10 machines = ROFL. If you go 4 years...$6/month...damn.

I am still making a push towards tape on critical data (photo's, work, etc)..but crash plan really takes need for duplication away from WHS and allows much cheaper "raid" setups. Right now going with onboard RAID5 with 4x3TB's HDD gets you a ton of space at reasonable performance.

Wow that sounds good, so is it really unlimited. As for the machine limit, could'nt you have everything backup to one machine then just backup that one machine?
 
crashplan is great, i had it setup to backup all my families PCs and Macs to my WHS but then i rebuilt my WHS and didnt transfer the backups correctly (there is a right way to do it) so now we have to start over, which is a little annoying, but a free program that allows my extended family to use my WHS as cloud backup is awesome

I pay for CP central for my 1 Server and have all my data backed up to that 1 server

but not for my 6TB of movies, just music documents photos and home videos
 
Back
Top