Long Term Storage Solutions

D

Dark_Legacy

Limp Gawd
Joined
Dec 7, 2006
Messages
251
Hey fellow [H]'ers.

I've been slowly expanding my collection of data collected over the years - gamesaves, documents, photos, videos, etc; and it's come to my attention that I've got no real long-term storage for most of this data.

Granted, most of the 3TB collection of it is hosted on an almost consistently-powered on box (my primary, backed by a UPS), but I don't know how long the hard drives are really going to last. I've heard that over time, drives demagnetize, the drive heads lose their lubrication and the drives won't spin. Basically, as per law of Entropy.

However, what would you fellow [H]ardware Enthusiasts suggest for a LONG TERM storage solution - e.g, 10/20 years, so I can show some of these things to my kids?

Note: Capacity of Backup - Minimum 3 TB.
 
no one really knows how long HDDs will last.
it also depends on what you mean by storage solution.

does it need to be usable? for example, buy a 3TB hdd, load it with data, put it in a safe deposit box
tape drives might work best
 
tape is the industry standard. but good luck finding consumer gear to do it.

interesting thought though.
 
Even tapes can break down over time, sometimes much faster than HDDs.

The best HDDs to use would be enterprise-grade drives, but they are also very expensive per GB.

Honestly, every 10 years or so, you should rotate your storage. Do this not only for safety of the data, but also to keep up with industry standards. Who's to say SATA-III will even exist in 10 years?
 
Blueray?

I would just swap out the drives every handful of years if you are worried about the hard drive going bonkers. About $120 every few years is nothing when the contents are priceless.

Now you have the old drives as a backup to the new drives.
 
plunger said:
Blueray?
Click to expand...

That depends on how much data you need to keep. Also optical media in the past has been shown not to store well.

In any case if you have stuff that must not be lost make sure you have more than 1 backup preferably in different locations and finally do not trust that an online backup company has a 0% chance of loosing some or all your data.
 
For pics, it's probably actually worth getting them printed out. I'd still keep the originals on hard drives and rotate them as suggested. Use 2 drives per batch that way if a drive fails when you go to plug it in, at least there's a backup. CD/DVD/bluray discs apparantly degrade over time, though I'm not sure if that's true. There may be archive grade discs you can buy that will last longer, too.
 
I'm doing it with offline NAS (ZFS RAIDZ2). Last week I've changed HDDs from 1TB to 2TB each. I was replacing 5x1TB zpool with spare eSATA port (it takes some time) but I suggest to crate a new zpool and copy data from an old one. It is safe and faster but in my case I should have 10 SATA ports available (2x5). It is fine to check and scrub each month or so or when backuping. And this is purely offline backup solution so live data are on "live" NAS with snapshots,...
 
plunger said:
Blueray?

I would just swap out the drives every handful of years if you are worried about the hard drive going bonkers. About $120 every few years is nothing when the contents are priceless.

Now you have the old drives as a backup to the new drives.
Click to expand...

+1 for this. Bluray is the only media with no organic materiel in it. CD-R, DVD+/-R, Tape and HDD all use organics - and organics decay over time. It might be a long time, but they always decay. 5-20 years lifespan likely.

Bluray is rated for archival storage, 50+ years. There are special BD-R disks manufacturer rated for 200+ years, but I don't know how accurate their testing is (and odds are the manufacturer won't even be around for the warrenty claim...but then again, neither will you!).

Of course, 3TB written to BD-R is 60 disks (more likely 120 disks since 50GB BD-R is still cost prohibitive).
 
I wouldn't put the entire 3TB on the blueray then. I would use blueray as a supplementary backup to hard drives. Everything goes on hard disk since its the cheapest. Family photos and family videos goes on blueray in addition to hard disk. 25GB a disk is a lot of space for this task. Do the blueray every few months and now you have multiple copies. I would even switch up the brands and store copies at different locations.
 
The only people I know who have 3TB of "critical data" are people with huge movie collection that really don't need to be backed up or a ton of family videos in "raw" format? Which bucket are you in?

To people worried about family photos, I solved part of my problem. Don't ever erase your cards, just buy new ones. Unless you have a SLR and just cranking away the images...no purpose in deleting. We probably use about 2-3 4GB cards a year. At that price, it is better to just keep them. They also go on a HDD as well.

We will probably switch to Blu-Ray archiving soon. While still not as awesome as tape, very cheap per GB in ocmparison.
 
How much are blue ray discs now days? Last time I checked they were like 35 bucks a pop, basically expensive enough to make movie pirating not worth it. (who even burns movies anymore anyway?) but maybe this changed.
 
BD-R ~$1-2/disk qty 20.

BD-R DL (50GB) ~$7-20/disk qty 10.

Most BD movies require DL disks. The disk manufacturers are still being pressured by RIAA not to let the price of DL disks fall too fast in order to manage piracy (even though almost all piracy today does not require a disk at all...its done using .ISOs distributed via torrent and played on media players/HTPCs). There is no technical reason for DL disks to be any more than about 150% the cost of single layer disks.

Prices still falling. Expect about 20-30% reduction in 1 year.
 
In my humble opinion, the only safe data is checked data.

So that means a system that regularly checks the integrity of your data.

That could either be hardware RAID or software RAID.

Since I'm a ZFS whore, I'll suggest that.
 
Trepidati0n said:
The only people I know who have 3TB of "critical data" are people with huge movie collection that really don't need to be backed up or a ton of family videos in "raw" format? Which bucket are you in?

To people worried about family photos, I solved part of my problem. Don't ever erase your cards, just buy new ones. Unless you have a SLR and just cranking away the images...no purpose in deleting. We probably use about 2-3 4GB cards a year. At that price, it is better to just keep them. They also go on a HDD as well.

We will probably switch to Blu-Ray archiving soon. While still not as awesome as tape, very cheap per GB in ocmparison.
Click to expand...

I've been told flash memory doesn't retain data all that long, like 10 years?
 
If you want real longevity you should go with punched cards! (just kidding)

Seriously though, I read an article recently - I think from the archival specialists at the Library of Congress - where they tested in a very scientific manner (over my head) all sorts of optical media. From the cheapest DVD-Rs to the most expensive archival DVD and Bluray discs. They concluded that they were ALL unreliable for long term storage over 5 years.

Also remember when storing data for the long term: store a means to get the data out of the media you put it on. Tapes, DVDs, Blurays, or hard drives are worthless if you can't read them. With the way that technology merrily marches along, the drives your using today will not be around to read your media in 10 or 20 years. You almost have to store an entire computer capable of reading the media along with the media.
 
munkle said:
I've been told flash memory doesn't retain data all that long, like 10 years?
Click to expand...

I never said use it for long term backup, I said it is a form of backup. No different than putting food in a freezer that is 0 degrees or -60. You have to draw the line somewhere of being reasonable vs being just plane nuts.
 
well you could put it on tape storage in the underground salt caverns here in kansas. that is how the movie industry stores its stuff...and the government.
 
It doesn't matter what you use for backup, I'll say it again:
Honestly, every 10 years or so, you should rotate your storage. Do this not only for safety of the data, but also to keep up with industry standards.
Click to expand...

Who's to say SATA, tape drives, or Blu-Ray will be even exist in 10 years?
Every decade the data should be moved to a new medium to keep up with standards.
 
you know that pack of a zillion blank dvds you bought when they went on sale? start using them bro
 
duoflea said:
you know that pack of a zillion blank dvds you bought when they went on sale? start using them bro
Click to expand...

Single-side single-layer DVDs only hold 4.7GB of data.
Try backing 10TB+ of data to those DVDs, you will be there for months.

Nice try. :rolleyes:
 
Tape is still the best long term storage solution in my opinion. We have been storing data on magnetic tape for 60ish years, therefore as a physical medium it is very well understood. I have an LTO3 setup that I use for archiving my and others' actual non replaceable data, mainly family movies and family photos, it cost me around $800 to set up with some smart shopping and patience on eBay. 400GB uncompressed per tape and you can get new tapes for about $20 each, so per GB isn't really that bad at all. I personally put my CD collection in FLAC onto a couple of tapes, as backing up several thousand dollars worth of music onto $40 worth of tape just seemed like a no brainer. LTO spec specifies that drives read 2 generations back and write one generation back, so I don't think accessibility should be an issue. I personally don't trust hard drives to necessarily work in 10 years, starting up a motor that has sat that long is far from certain, this is not what these drives are designed for. Flash is constantly evolving with higher density features, too fast to ascertain any useful data on longevity. BluRay may be as stable as claimed, then again it may not. I am very confident that I will be able to retrieve every bit of data from my tapes in 10 years, if only to put that data on a newer generation of tape which I am pretty sure will still be in wide usage at that time. Just my $.02

Dustin
 
Yes, tapes can last over ten years. I'm not sure why everyone is saying blu-ray is the best way to backup data. I have CDs and DVDs which were burned over eight years ago and are still going strong. Blu-Ray has barely been around for four years, making it very unproven for longevity.
 
I also wanted to mention that storage standards, particularly ones that are intended for long term archival don't tend to become obsolete that quickly. Even going back 20 years instead of 10, we had double density floppies and things like DDS and QIC tape as common storage formats. Any of these would be quite easy to get the data off today assuming the data itself was intact on the media. Going back even further to things like the audio cassette storage formats common in the late 70s and early 80s, a working cassette player, decent sound card and a simple piece of software on a modern PC can retrieve this data without much effort.

Dustin
 
Don't forget the transport itself. Tape drives are notorious for having alignment issues. What might be readable on the drive the first time it's used might be inaccessible on the same drive years later. And even a tape drive left sitting on the shelf is no guarantee as corrosion, dust and rot continue to take their toll.

There's a reason some places have tape librarians, personnel that handle re-reading and re-writing data to/from old tapes just to make sure the data is still accessible.

Regardless of media type it's pointless to keep backups if you don't periodically make sure the data can actually be reloaded and verified.
 
We used tapes where I work and they have a very high failure rate, we have to keep multiple copies always. Burning DVD's/BR's are good for a long time.

You can also buy a hdd, fill it up and then disconnect it from your system and store it. I a have a few older drives that I fired up last month to check on them for the first time in 6 years and they still work great.
 
tapes where I work and they have a very high failure rate
Click to expand...

Was this DDS?

For DLT and LTO I have had like 2 tapes fail out of 100+ purchased and for both 5+ years of usage.
 
Last edited:
What you need is a gold-plated copper disc like the one sent into space on Voyager 1. I believe it has an estimated lifespan of 50,000 years in the harsh vacuum of space :)
 
/dev/null

Unlimited storage, unlimited compression, deduplication, replication, redundancy, extremely fast, you can set it up as a nfs share too
 
vraa said:
/dev/null

Unlimited storage, unlimited compression, deduplication, replication, redundancy, extremely fast, you can set it up as a nfs share too
Click to expand...

loool I never thought of setting that up as a share. Brilliant.
 
wkearney99 said:
Don't forget the transport itself. Tape drives are notorious for having alignment issues. What might be readable on the drive the first time it's used might be inaccessible on the same drive years later. And even a tape drive left sitting on the shelf is no guarantee as corrosion, dust and rot continue to take their toll.

There's a reason some places have tape librarians, personnel that handle re-reading and re-writing data to/from old tapes just to make sure the data is still accessible.

Regardless of media type it's pointless to keep backups if you don't periodically make sure the data can actually be reloaded and verified.
Click to expand...

Tape drives certainly do fail, but that has no bearing on the integrity of the data, because the data isn't stored on the drive. You can replace/repair the drive and pop your tapes in and off you go. This isn't so with a hard drive, the transport and the media are intended to operate as a unit, if the transport fails so does the media, barring expensive and difficult recovery especially if it's a motor failure.

As to reliability, LTO has some pretty crazy redundancy as I understand. Even if entire portions of the tape are unreadable, like inches missing, reading the tape is still possible. Also, archiving and incremental backup aren't the same thing, as far as tape (and I suppose drive) wear and tear is concerned. A tape that is written only once in a single session and then properly stored is going to do better than one that's been reused or even written once but in multiple sessions.

Periodic verification is of course important regardless of your backup medium, except maybe an online backup service that performs this server side. Thing is, no matter how often you verify it, if it fails you're still screwed. A hard drive will tend to fail suddenly and like I said, you're screwed. An LTO verify will notify you of recoverable errors, signifying declining health of either the medium or the drive before total failure, while there's enough redundant data available to still recover the data intact.

So while tapes, and their drives do "fail", it's rarely the case that this involves actual unrecoverable loss of data in my experience at least.
 
And even a tape drive left sitting on the shelf is no guarantee as corrosion, dust and rot continue to take their toll.
Click to expand...

Which brings up, make sure you store it properly! the best media in the world wont last if you store it in a damp cold place.
 
I've thought about this subject as well.
Years ago, I made a backup to ~100 CDs. Needing the data about a year later, Brand A worked perfectly while Brand B had serious issues with the outer ~20% of the disc. while my archival conditions are hardly perfect, they were stored out of the sun, etc. I can only conclude that optical media is unreliable, not to mention a hassle when (re)storing large quantities of data.
I have considered tape, but nixed it due to the huge cost and lots of old stories of notorious unreliability, especially when restoring.
Flash memory is a new medium, but has a limited shelf life (cells losing charge) and is cost prohibitive for any reasonable amount of storage.
That leaves magnetic platters.
You talk about storing things for 10-20 years, but, as you say, the amount of stuff you want to store is constantly increasing.
I expect you already have everything stored at home.
You need to buy two additional drives (preferably external) to use as a backup medium.
Every month, you backup everything to one drive. Don't just update, copy everything to ensure that the magnetic charge on the platter is refreshed.
Alternate drives so that one is safe while you update the other. Maybe lightning strikes while you're updating the backup, rendering your efforts null and void.
Store it in a safe place (bank vault). Maybe even two different places.
With this solution you constantly check that the backup media is still working, you have an alternate, slightly older, backup if one should fail, and you are sure that the backup medium is readable by current systems. Make sure to buy drives of different brands - if they are the same, they may fail similarly, at a similar time.
 
Yonzie...in regards to optical media...you can't really, with any degree of intellectual honesty, make that claim. To somehow think that HDD are the superior backup solution just boggles my mind based upon personal experience and not statically proof floors me. I could make the claim that since my 20 DVD-RAMS have never failed after 7 years of constant use, that they are the superior solution....no?

All backup methods are prone to failure. Usually most of those failures are user/operator induced by either using it incorrectly or not using the correct quality. Simple example is people using a consumer grade CD and CD burner and then going with 100 discs in a row and not expect an issue. But I think you, yourself, have stumbled upon and are seeing that independent dual sources" in an non-professional setting is prudent.
 
I totally agree. My anecdote proves nothing. The point I was trying to make was that buying optical media and hoping it will work in ten years is a gamble. One will have to test the discs regularly to check data integrity. If the backup requires 120 BR discs (3TB/25GB) that is (IMO) extremely unlikely to take place.
Of course, that is the case with just about any digital consumer-grade media.
I'm not saying harddrives are statistically less prone to failure, I'm saying that it is (much) easier to check if one or a few harddrives work, compared to optical or tape.
 
Tape is not unreliable. While hard drives have become considerably less reliable in recent years, the opposite is true of modern tape formats. Narrow your research down to the specific tape format you're considering using, not reading fail reports from 10 years ago regarding formats that the industry has tossed in the bin long ago. Your bank records are backed up on tape, for example. Almost guaranteed.
 
You must log in or register to reply here.
Back
Top