Durable hard drive for long term dormant storage?

NoxTek

The Geek Redneck
Joined
May 27, 2002
Messages
9,300
<EDIT>

Bwaahahaha... god I'm getting senile or something, I asked this same question like six months ago...

http://hardforum.com/showthread.php?p=1035845760

</EDIT>

I'm in search of a means to store LOTS of raw 720p (and later 1080p) video and I've come to the conclusion that with the cost per gigabyte hard drives are the only logical choice.

Up until now I've been storing raw video on DVD-Rs but with high definition video and me wanting to keep copies of the original video from the camera, DVD-Rs are starting to be cumbersome to use. I can easily shoot more than 4.7GB worth of raw HD video in a single session.

So I'm trying to figure out what's the best way to go about storing all my raw video - on 'off the shelf' external drives or on regular internal SATA drives. So I'm basically looking for a really rugged proven drive that will do well being stored away in a controlled environment and possibly only powered on once in a VERY long while.

I can't think of any other way to store this stuff in a medium that will stand the test of time... except maybe BD-ROM discs but they are still prohibitively expensive.
 
Last edited:
What you ask for does not exist. Or at least it does not exist in SATA drives. From any manufacturer it is expected that 1%+ of your hard drives will die each year and the amount of usage or age is not always a factor. If you want 100% reliability make more than 1 copy. Also I would never sit a hard drive on a shelf for 5 to 10 years and expect it to work. Your best bet is probably BD-ROM if you want really long shelf times and/or make multiple copies of everything.

Edit: Externals are expected to be less reliable than internals just because of the dirt cheap enclosure that most (if not all) manufacturers use with the drive. The good thing however if the enclosure dies there is usually a regular SATA drive inside.
 
Last edited:
I don't know if I would trust any optical medium for more than a few years, except maybe DVD-RAM. There's just too much variability in the quality of the various dye formulations combined with wildly varying quality on the various burners in the market. I haven't been following BD-R development too closely but I'd be amazed if the situation were all that much better. DVD-RAM is supposedly more robust but you still have the issue of < 5GB capacity per disk, and the write speed is usually much slower too.

Also with high capacity magnetic drives there's the issue of "bit rot" which I think can happen even if the drive is un-powered, i.e. the magnetic domains on the disk surface can drift over time and slowly corrupt the data.

So whatever your strategy is, you're probably going to end up recopying the data at least every few years.
 
How about Dual-layer discs? Not a perfect solution but you will have double the capacity of single layer discs at least.
 
If you are talking about DVDs these are not that good at data retention brand new so I would avoid that. 25 GB BD-R is supposed to be better however.

If I was in this situation I would get 2TB green internal disks from at least 2 different manufacturers and make 2 copies of every thing one copy per manufacturer.
 
I agree with using 2TB hard drives. I don't know if different manufacturers matters. 2 or more copies is always good.
 
I was trying to reduce the chance of a bad batch or series. All manufacturers have had problems over the years.
 
At this point, I'd start recommending 2TB HDDs, tape backups, and BD-ROM discs.

I would recommend against storing any long-term data (10+ years) on a DVD or CD ROM as they both have organic materials which may break down over time if they are not well kept in a good/stable environment, i.e. moisture and temperature control.

BD-ROMs do not have this problem and are very good for long-term storage.

External drives are what I use, but only modern versions which have a true 2.5" SATA drive within, and I only use WD Passports and Seagate GoFlex's.

Do not use the WD Elements drives as the case and chipset are crap.

Tape drives are good as well and can last for many years, but the conditions must be optimal to prevent a breakdown of the tape.


In all honesty, any data stored should be moved to new/updated media every 10 years, not only due to the drives or media breaking down, but due to changing standards as well. 10 years from now, USB and SATA may be long dead and finding a system to use those standards may be very difficult at that time.

For so much storage need though, no solution is going to be cheap and at this point a high-end NAS or a SAN may be necessary.

Bottom line, the cheapest alternative is BD-ROMs and external HDDs.

btw, hey Blue Falcon. ;)
 
Yeah, I fully intend to have some sort of redundancy system in place. HDDs still seem like the only viable option even with the chance of bit rot. In 15+ years of doing this I personally have never seen a drive just up and have corrupt data after sitting several years, at least not without another eletrical/mechanical problem being evident.... and believe me I've seen LOTS of drives after sitting dormant for several years. I've even got an old Connor 20MB IDE drive with my original BBS stuff still on it and perfectly intact. I, however, am not one to taunt that evil diety we all know as 'Murphy'.

<EDIT>

Hey Blue Fox!

</EDIT>
 
I am not so much worried about bit rot but more about bearing failure or other mechanical failure for long term storage. The risks can however be minimized if you can keep the drives at a constant temperature and low to normal humidity.
 
Last edited:
At this point I'd just use ZFSGuru.....

Oh and build 2 boxes, 1 Intel and 1 AMD LOL!!!
 
That is an option if the video needs to be online and it is worth the electricity cost to do so. There was no mention of how many TB the OP currently is using and how much that grows or even a budget for this.
 
That is an option if the video needs to be online and it is worth the electricity cost to do so. There was no mention of how many TB the OP currently is using and how much that grows or even a budget for this.

Well I've been accumulating video for at least 3 years now and I've currently got about 300 DVD-Rs full of HD h.264 video. I'd say at this point I'm archiving on average about 20GB a week, but that number is increasing as I grow more active in my videography activities. No doubt it will increase when I switch to 1080p video (at some point in the future).

My budget is... well.... it might as well be nonexistent compared to some of the money I see people spend here. For example an on-the-spot build of a 10TB NAS box is out of the question. It's more like I have ~$200 every month I can possibly throw to this project.

I'm not opposed to building a machine to keep the drives online 24/7, but I have a hard time believing that would make the data safer than putting it on the drive and then keeping it in the closet packed away safely. I do realize nothing is foolproof or accident proof - 'shit happens'. I just want to find a more practical and accessible way to store all this video I'm accumulating offline. At this point even with a good cataloging system it's getting tough to sort through DVD-Rs to find a particular piece of video.

Jesus, shouldn't we have virtually unlimited holographic storage cubes by now? I mean, aren't Professor Moriarty and the Countess Regina living their lives in one at this very moment??? (lol... TNGFTW!)
 
I would look into 2TB hard drives and something like the following.

http://www.newegg.com/Product/Produ...71&cm_re=sata_external-_-17-153-071-_-Product

I'm not opposed to building a machine to keep the drives online 24/7, but I have a hard time believing that would make the data safer than putting it on the drive and then keeping it in the closet packed away safely.

There will be a lot of cost in this and it would not free you from the need to make a backup. Which could mean you need 2 NAS boxes since a backup should not be on the same machine as the source.
 
I'd say at this point I'm archiving on average about 20GB a week, but that number is increasing as I grow more active in my videography activities. ... It's more like I have ~$200 every month I can possibly throw to this project.

20GB/moth is 1TB/year. About $100 for hard drive space for 2 copies. It appears that money is not an issue for this project.

Buying the largest cheap hard drive - 2TB now, every time you run out of space seems reasonable. As media becomes obsolete you can just do a copy of old to new.
 
Yeah even Blu-ray wouldn't cut it. Have the main copy HDD in your computer on a hot-plug drive bay, and use an external drive as long term archive.
It makes mirroring easier to manage, you can keep the videos on both the main hard drive and main copy drive for a week, and replicate to the external archive drive every week.
All the better if you can use a different safe location for the external drive, to avoid losing both sets of data through theft or accident.

Your biggest problem is probably not storage, but to find out the best and easiest archival and replication system, the folder structure and tags to identify videos, etc.

I'd go straight for 3 TB drives though, it will give you space for all your current 300 DVD-Rs plus one year of raw footage.
Save most of these $200 every month, looks like you may need a more decent setup with multiple drives in the future, or a new computer with many hot-plug drive bays and eSATA.
Also, don't plan on keeping an odd collection of archive drives for 10 years, just replace them with the biggest ones you can find every 2-3 years, they'll be faster, more efficient and twice the capacity. This removes all risk of bit rot and you can beat Murphy's law, and maybe even Moore's law, if your need for storage does not double every 2 years!

In the Foundation series, planet Gaia uses the low intensity but gigantic capacity of rocks' collective memory for long term storage, you should look into it! ^-^
 
I say keep the hard drives running. Hopefully S.M.A.R.T. will continue to make leaps and bounds in the future, so keeping your hard drives monitored means replacing them with a backup before something happens.

I also wanted to ask, how much of the video will be business critical say 5 years from now? Are you making promises to clients that you will always have copies of their video? I feel you though, video is just the worst data to keep up with. Dang near 99% of the stuff isn't recreatable. You could go out and get more footage of some wildlife or even an interview with someone, but weddings are particularly expensive to begin with and don't seem to have that same meaning the second time around.

For me, (and I do some video stuff, but mostly self created work) I'd go the 2TB hard drive route in a powered computer to monitor its health, swap it out with an exact copy every week with an exact duplicate kept in a security deposit or sorts, all while continuing to run your DVD (hopefully soon to be BluRay) archive. At some point though, you need to figure what the upkeep cost is of all this is, and how it affect overall business profits. Every year becomes more expensive with more data to retain, although with cheaper storage costs. I just know for myself, I'd let the client how long I typically keep backup copies. If they want me to keep it longer, they can help me manage the financial stress of it all. Just my two cents.
 
I use 2TB WD Green drives and put them in a fire safe.

To compliment those drives I'm thinking I will also burn the data to Bluray discs and put those in a different place somewhere else. Prices for them are great now compared to when they first came out.

http://www.newegg.com/Product/Product.aspx?Item=N82E16817130070

You can often get them on better deals too if you watch the email ads.

Figure you adding 20GB or so a week, a couple 2TB drives will go a good way, then each week also toss that stuff onto a Bluray and throw it in the safe. :)
 
Back
Top