Backup for 5-10TB+ Systems

[JJ Johnson]

First, ask yourself whether or not all of the data is in need of being backed up? If it's mostly pirated pr0n and bad movies that you've downloaded, you can probably easily download it all again if anything happens. Figure out what absolutely needs to be backed up and it may turn out to be a small fraction of that.

After that, the choices are fairly simple. If you have 7 TB of data on disk that needs backing up, then you'll need another 7 TB of disk space for you backup. That could be on external drives, drives plugged in and out of a hotswap bay, or drives in a separate system or systems. If you have 20 TB of data that you think needs backing up, you'll need 20 TB of additional drive space. At some point the amount of space needed makes using external drives too unwieldy, so you will need a second system for backups.

 

[tycoonbob]

First, ask yourself whether or not all of the data is in need of being backed up? If it's mostly pirated pr0n and bad movies that you've downloaded, you can probably easily download it all again if anything happens. Figure out what absolutely needs to be backed up and it may turn out to be a small fraction of that.

After that, the choices are fairly simple. If you have 7 TB of data on disk that needs backing up, then you'll need another 7 TB of disk space for you backup. That could be on external drives, drives plugged in and out of a hotswap bay, or drives in a separate system or systems. If you have 20 TB of data that you think needs backing up, you'll need 20 TB of additional drive space. At some point the amount of space needed makes using external drives too unwieldy, so you will need a second system for backups.

You still are missing the point that has been made multiple times. Backing up locally is great, but does NOT allow you to restore your files if your hardware is stole or destroyed in a natural disaster (fire, tornado, etc).

Not backing up media that could be downloaded off the internet is also not a good excuse. Who is to say all of that content is available out on the interenet (illegally, might I add)? What if I ripped all of my CDs to FLAC and lost the originals...what then? What if I took the time to categorize, tag, and apply proper artwork...while I may be able to redownload the files, I would have to spend all the time tagging and cataloging again. No thanks.

Unlimited cloud backup please.

Also, another difference I found about AltDrive is that unlike CrashPlan, AltDrive can backup files that are in use or open. Not a big deal, but definitely worth noting.

 

[JJ Johnson]

You still are missing the point that has been made multiple times. Backing up locally is great, but does NOT allow you to restore your files if your hardware is stole or destroyed in a natural disaster (fire, tornado, etc).
...
Unlimited cloud backup please.

What else is on your Christmas list to Santa?

You do what you can within a budget and within what is practical. For the many who somehow still think using RAID is protecting their data, it's already infinitely better.

Online backup for tens of terabytes of data will not be feasible until upload speeds are increased many, many times. Hard drive seeding may make it _more_ feasible, but the reality is that it's still a long way off for anyone with a large media library.

 

[m1abram]

What else is on your Christmas list to Santa?

You do what you can within a budget and within what is practical. For the many who somehow still think using RAID is protecting their data, it's already infinitely better.

Online backup for tens of terabytes of data will not be feasible until upload speeds are increased many, many times. Hard drive seeding may make it _more_ feasible, but the reality is that it's still a long way off for anyone with a large media library.

~$50 a year should be affordable for anyone who is dedicating several TB of data that they care about.

My upload speeds are probably better than most at 10mbps, however the seed cost is not too bad. Looks like $125 per TB EDIT: Appears you can only seed up to 1TB.
http://support.crashplan.com/doku.p...seed_your_initial_backup_to_crashplan_central

Yeah that gets pricey once you move into the multi TB range, so I would only seed my most important data and let the less important take its time with my upload limits.

 

[tycoonbob]

With my 5mbps upload it took a few weeks to get all of my data uploaded, but with local backups, that is fine. The first upload is always the worst, and once it's been endured...incremental backups is all that is needed.

I built a $1700 NUS server (not including drive costs). 24 hot swap, which will soon be a giant Raid 60 (once I can justify the cost of the drives). Excellent performance. I have a 8 bay eSATA NAS that is connected for local backups. I don't back up everything locally, just the important stuff (personal files like photos, home videos, documents...archived music, and some other stuff). I then backed up that same stuff to the cloud, then I set up another Backup set for everything else, and let it run. Never noticed it running...didn't have to manage it, just set it and forget it.

 

[SirMaster]

Also, another difference I found about AltDrive is that unlike CrashPlan, AltDrive can backup files that are in use or open. Not a big deal, but definitely worth noting.

Hehe, CrashPlan can too!

I've compared a few backup services and I cannot think of a single feature I've seen elsewhere that CrashPlan does not have as well.

 

[tycoonbob]

Yeah, I was just reading in CrashPlan's FAQ and saw that they use VSS in Windows for that, so feature wise for a cloud backup solution, it's just as good as AltDrive.

 

[odditory]

And not to mention the daughter's tornado room likely cost $300/sf to build,

eh? I thought "tornado room" was just hillbilly code for laying down in a bathtub and praying when the storm done roll in

 

[liamjbenett]

Just coming back to this after having done a trial of crashplan. In theory I think this would work great and I do like the tool a lot but I am going to have to come up with another solution because my upload speed is just not fast enough either for the initial seed or for ongoing backups.

I think ideally I am looking to build a offsite NAS and do manual transfers with another eSATA disk, sneakernet-style. What do you guys think?

 

[GeorgeHR]

Agreed-

And not to mention the daughter's tornado room likely cost $300/sf to build, the average American's wage is 30 bucks an hour, Drives cost $75/TB (and fail! and get killed when they land on the concrete!) and mileage is almost 60 cents a mile.

Crashplan and the likes are quite a good deal if you ask me.

My daughter's tornado room is 8'x26'. About 200sqft. The cost was much less than 200*300 ($60,000). Most likely under $3000. For under $100 I can make a disaster proof container that will hold 6 hard drives - 24TB. But my bank safe deposit box offers better economics.

The median family income is about $50K/year. That appears to be $25/hour (2000 hour year). So $30/hour is above the average for the middle class. $38/hr makes one part of the rich.

Drives cost under $50/TB. I don't drop them.

Business mileage deduction is about 56.5 cents/miles for tax purposes. Our actual cost to drive a mile is about 20 cents. So driving is a profit center for our business.

I suspect your economic analysis of CrashPlan is as poor as it is with regard to my statements.

 

[kac77]

Just coming back to this after having done a trial of crashplan. In theory I think this would work great and I do like the tool a lot but I am going to have to come up with another solution because my upload speed is just not fast enough either for the initial seed or for ongoing backups.

I think ideally I am looking to build a offsite NAS and do manual transfers with another eSATA disk, sneakernet-style. What do you guys think?

That's why some people are saying that if you have TB's of data it's crazy to sit up there and use cloud storage as your main form of backup. GB's sure. You'll probably get most of it back within a day. TB's? Depending on the service days or weeks.

Another problem with the fire, tornado scenario is that if your house get's leveled It's going to take you while regardless. After all what are you restoring your data to if your house get's leveled? The answer? Nothing.

Drives if stored properly and definitely tapes in a safe will survive fire, tornadoes and even water. What you do is you get home insurance which costs like 5 bucks a month. If your server gets sent to the graveyard you call your insurance company. They send you money to rebuild. If you have a local backup then you'll be back up the same day you get your server rebuilt.

Offsite backup is for mission critical stuff that you absolutely cannot do with out. You should always have some form of local backup on hand.

 

[SirMaster]

That's why some people are saying that if you have TB's of data it's crazy to sit up there and use cloud storage as your main form of backup. GB's sure. You'll probably get most of it back within a day. TB's? Depending on the service days or weeks.

But have you considered this?

Why would I suddenly need access to ALL my TBs of data at once? If I lost it all and started the restore process from the cloud, you are right, it would take weeks. But during that time I can also pause it and just download the files(s) I need right now immediately. Most cloud backup services have random access to your data. I can pull down about 12GB an hour from my CrashPlan backups.

For my data and the the way I access and use it this is more than satisfactory.

Heck I think it's super convenient even. If I lost my house with my server and everything, I could still access all my data on CrashPlan's servers from my laptop, iPad, iPhone, etc while I wait for the money to rebuild my server and while I'm still setting my server back up before I even get around to starting the restore process.

 

[kac77]

Why would I suddenly need access to ALL my TBs of data at once?

This would be the case for any solution. Either way the cloud solution would take the longest to get my data back.

If I lost it all and started the restore process from the cloud, you are right, it would take weeks. But during that time I can also pause it and just download the files(s) I need right now immediately. Most cloud backup services have random access to your data. I can pull down about 12GB an hour from my CrashPlan backups.

Again the problem with this is that if you have TB's you'll max out at somewhere around 288 GB per day at the absolute best case scenario (and I doubt you'll hit that high sustained) . Now if you have TB's of data you can already see that even if we are talking about half a TB (even a few games will easily occupy that) your going to be waiting for at least 2 days. If you have more than that you are talking weeks to get back your data. If this works for some then that's great, but it's got some pretty big downsides in my eyes and waiting weeks to get back your stuff is one of them.

For my data and the the way I access and use it this is more than satisfactory.

I'm not saying it isn't. It just isn't a panacea either. It's got a few things going for it, but the main one in my eyes is price. It is the cheapest solution and there's no denying that.

Heck I think it's super convenient even. If I lost my house with my server and everything, I could still access all my data on CrashPlan's servers from my laptop, iPad, iPhone, etc while I wait for the money to rebuild my server and while I'm still setting my server back up before I even get around to starting the restore process.

The chances of an iPhone, iPod or laptop surviving a tornado or fire is pretty slim. During a statewide emergency you'll be lucky if you can even make a phone call. That being said, as I said before for mission critical stuff cloud based storage makes sense. I just wouldn't try to back up TB's to it is all. But for really important documents then yes it's good to have.

 

[tycoonbob]

With AltDrive, I can pull down a little over 20GB per hour. That comes to over 400GB per day. 20 day to pull down 10TB is perfectly fine for me considering what the data is (movies, TV, music, photos, important docs,etc). You can guess that the movies and TV shows take up the majority of space, so that will be the last thing restored. Everything backed up and backed up in a data set, so I can pull down my personal docs data first, then once that is done my photos, then music, etc, etc. Pull down the stuff I need first.

Granted I can only upload about 2GB per hour, or 40GB per day, it will take a while to get 10TB uploaded (which I don't have 10TB of data, more like 5TB). 4 Months or so to get all my stuff backed up to the cloud, which I am perfectly okay with. I have 4TB on-site backups, which backs up what I consider critical, and more. For $5/mo or less, it's perfect for me and a no brainer. AltDrive has been around since 2006, so they must be doing something right.

Oh yeah, each person I refer to AltDrive gets me 2 months free. I have gotten a total of 8 months free so far.

 

[Ruffy]

ZFS Nas on my network acts as a backup target from all machines

Once every 3-4 months I Offload the important data to external drive, which gets put in a fire/water safe.

Beyond that crashplan.

If the place burns down my important data is still relatively safe. If the earth opens up and swallows my apartment up then I have more things to worry about than the recovery speed of crashplan.

 

[docbill]

I've just been doing my own tests with crashplan. Really it does not make sense as a many TB disaster recovery. First of all upload speed. I get about 35 GB per day. Crashplan only promises 10 GB per day, so I count myself lucky. So at the promised rate it would take 100 days to backup a single TB. At the rate I'm getting, a month. So my 12 TB archive would take one full year to backup. Really, that is not bad considering I have a 30/10 Mbps connection, and I'm reserving 20% of that so my interactive usage won't suck.

Worse though is restoring the data. Lets say someone breaks into my house and steals my hard drives. It turns out restores are slower than backups. I'm still measuring how much slower... But it will likely take about 1.5 years to restore that 12 TB of data. To add insult the recovery to door service is only available in the US, only if you have less than 3.5 TB total in your account, and only if you aren't using a private key. Each of these is a show stopper to me.

Realistically crashplan is good if you have less than 1 TB of data, or if you only want it for individual file recovery. e.g. It works great if a file becomes corrupted, overwritten, etc, and you want to restore it. The only issue is if you want to restore everything!

In anycase, until crashplan improves there download speeds, I won't be using their online cloud storage. However, there program still seems useful as an easy way to automatically backup all my families computers to one computer. So I may well still use their software. Even if I do decide to go with another solution like altdata for cloud storage.

 

[ToddW2]

From 2013..... back from the dead

 

[zaniix]

I've just been doing my own tests with crashplan. Really it does not make sense as a many TB disaster recovery. First of all upload speed. I get about 35 GB per day. Crashplan only promises 10 GB per day, so I count myself lucky. So at the promised rate it would take 100 days to backup a single TB. At the rate I'm getting, a month. So my 12 TB archive would take one full year to backup. Really, that is not bad considering I have a 30/10 Mbps connection, and I'm reserving 20% of that so my interactive usage won't suck.

Worse though is restoring the data. Lets say someone breaks into my house and steals my hard drives. It turns out restores are slower than backups. I'm still measuring how much slower... But it will likely take about 1.5 years to restore that 12 TB of data. To add insult the recovery to door service is only available in the US, only if you have less than 3.5 TB total in your account, and only if you aren't using a private key. Each of these is a show stopper to me.

Realistically crashplan is good if you have less than 1 TB of data, or if you only want it for individual file recovery. e.g. It works great if a file becomes corrupted, overwritten, etc, and you want to restore it. The only issue is if you want to restore everything!

In anycase, until crashplan improves there download speeds, I won't be using their online cloud storage. However, there program still seems useful as an easy way to automatically backup all my families computers to one computer. So I may well still use their software. Even if I do decide to go with another solution like altdata for cloud storage.

For a disaster you just pay them to send you the data on hard drives. Considering how cheap it is for unlimited storage I think its hard to beat.

 

[docbill]

For a disaster you just pay them to send you the data on hard drives. Considering how cheap it is for unlimited storage I think its hard to beat.

I just commented on that. They will not ship your data on hard drive for any price if:
1. You want them shipped outside the US. (I live in Canada.)
2. Your total storage in CrashPlan is more that 3.5 TB.
3. You use a private encryption fee...

http://support.code42.com/CrashPlan/Latest/Restoring/Restore-To-Door_Overview

Reading through the FAQ, I don't really understand the reason for restriction 1. It makes no sense. Restriction 2 seem to be based on the fact they use 4TB drives to ship the archive, and they do not want to split it across multiple drives. Restriction 3 indicates to me they either want to send the data unencrypted, or they want to review the data to make sure they aren't shipping something illegal like child pornography.

I could probably live with restriction #3, as while I don't like the idea of them viewing my data, it is probably better than losing it. I could probably use a relatives address in the US to get by restriction #1. However, #2 is the complete show stopper. It means the only they aren't going to ship my 12 TB backup no matter how I beg. So I will need to spend more than a year to do a full restore. Although I suppose I could restore 8.5 TB, and then delete that from their server so I could get them to ship the last 3.5 TB for a cost of $300 to my relative's house. Where I could then spend $150 in fuel for the round trip to pick it up because I wouldn't trust my relatives to repack the drive after they opened by mistake and ship it to me. All and all not a very good backup strategy.

I am however, still tempted by crashplan. Even though it doesn't really seem like a good full emergency restore option. It could help for the Opps, type mistakes where locally I backup my mistake... And chances are over time they will improve their restore speeds. Maybe by the time I finish backing everything up, the restore speeds will be twice as fast?

 

[ToddW2]

Your plan introduces a lot of scary issues DURING the recovery of your unrecoverable data ??? LOL!!

 

[docbill]

It is not a plan, it is an option. Not a very good option at that. A better option would be to allow it to take the 1+ years needed, and prioritized what is downloaded first. In the end, like most people I use only a very small percentage of my data on a regular basis. For my movie archive, I would just need to plan out what I might watch in the next few weeks and prioritize the download of those first. Really it would all be recoverable, which is good. It just wouldn't be walk away and forget about it, until it is done.

 

[PhoenixC5]

Interesting. I upload roughly 400GB each night from my nighly ESXi backup process. It takes about 8 hours or so, but it completes each day without issue. Am I just extremely lucky?