Solution for storing massive amounts of video (MP4/AVC)? Hard Drives?

NoxTek

NoxTek

The Geek Redneck
Joined
May 27, 2002
Messages
9,300
So for those of you who don't know me, I am a bit of a shutterbug and a videobug. I probably record at least 10 minutes a day just doing my YouTube blog and for special events I've been known to record a couple of hours. For example I've got 30GB (about 80 minutes) of graduation ceremonies video I'm trying to figure out what to do with.

This video is all recorded at 720p (1280x720) / 30FPS at a bitrate of around 20 - 24FPS by my Canon Powershot SX20 IS. It's recorded using AVC Lite (MPEG4) by the camera.

After several tests I've discovered that the 24mbit bitrate that the camera uses seems to be the 'magic' number where quality loss is minimal, so it looks like I'm just going to have to start storing the raw video files without compressing them any further if I want to hang on to source video that's as pure as possible for any future projects. As you might see this is going to lead to a whole lot of needed 'archival' storage.

So I'm trying to figure out the most cost effective (yet reliable) means of storing such large amounts of data. Right now I'm leaning towards just purchasing SATA hard drives, using them to store the data, and then storing them in some sort of padded anti-static containers for long term archival storage. Hard drives are just so damned cheap now even for 1+ terabyte drives. But I'm wondering what the feasibility and reliability of using this method are, and more importantly if I'm overlooking some other obvious and more reliable method of storage.

I've considered getting a Blu-Ray drive and using BD-R discs to store this data but right now the prices are still way too high for my liking on both the drives and especially the media.

Any ideas?
 
How much are you trying to store? With hard drive costs, it is not that expensive to buy for a capacity now, then add capacity with larger drives later (when they get cheaper). Think 3TB and 4TB drives will be here by the end of the year.
 
Think this needs some clarification on the scale.

10 min = 3.75gb per day
112.5gb per 30 days (or about 8 months per 1tb of storage)

How many TB do you currently have?
Do you want redundancy at all?

But the most basic answer is an internal 2tb drive and an external 2tb drive to back up to, which would last about 16 months at your daily rate.
 
Hi

Every time I've looked at alternative ways of storing data I've come back to (SATA) hard drives as the only cost effective storage solution for significant amounts of data.

Three other options come to mind but I haven't researched any of these in any great depth:

1. LTO tapes. You can pick a LTO-3 tape drive up pretty cheaply. Quoted lifetime 30 years, as usual your experience may vary (google/bing/whatever "lto tape life expectancy")

2. SSD drives. Less capacity but probably more reliable, thought I haven't personally looked at these in any great depth as the price per megabtye doesn't make them attractive (to me) at the moment.

3. Look at the cloud. Some great deals in the past on online storage. Issues on trust of course and will (probably) be very expensive for the volumes you are looking for.

As Homer pointed out, whatever you estimate, double it for a backup at the very least!
 
I've been using external hard drives for some time & I had just about maxed out the 3 I've got when the Drobo FS showed up. It's a little pricey, but you get expansible storage with redundancy and it's a breeze to use. It's basically a NAS device with 5 empty bays for SATA drives. I've got 3 x 1.5 TB in it now, but it'll hold up to 2TB drives. The 2TB drives are heading towards $99 nowadays, so it's not too bad in the long run.

I'm pretty sure that when the 3TB and 4TB drives show up, they'll release a firmware update to allow those drives, and if so all the better. The really cool thing is that the device can also run applications like FTP and BitTorrent, so you can have things automated if you need them. It also has a cloud component that is either available now or soon. It'll backup all the data to a cloud based storage system in addition to the device, so off-site back-up will be simple.

I've had mine for less than a month and so far it's been worth the investment. Going from 3 powered devices to 1 is a plus and I'm going to try and sell off at least two of the external drives to recoup some of the cash I've put into all this.
 
Well to clarify I'm not necessarily looking for storage that will be online and available in a NAS like scenerio, in fact I'd rather be able to fill up the drive and then store it away in the closet or in another controlled environment. I'm just wondering how long the drive will last in a non-powered static state for a period that could be several years (until a more viable means of storage comes out).

My logical mind tells me that if a drive can last on a retail shelf for months or years it can last in my vault for years and still be powered up years later with no problems. But then again there is that damned Murphy fellow always sticking his nose where it doesn't belong... and he seems to follow me a lot... :D
 
Blue Falcon said:
But then again there is that damned Murphy fellow always sticking his nose where it doesn't belong... and he seems to follow me a lot... :D
Click to expand...

I've had two sets of RAID arrays fail on me in the past and thankfully I was able to get the most important data off the drives. If you want this stuff to last years, you might back up to DVD or BR in addition to the drives at the very least. Or maybe pick up an Amazon S3 account to just store the files in the cloud for a relatively small bit of cash each month/year.

I just found this blog post on the subject of S3 for backups:
http://jeremy.zawodny.com/blog/archives/007624.html
 
If you consider online-backup I would rather use a dedicated backup-service instead of S3. S3 is more work and costs. Also the guy in that link uses a 120W System where you can easily build a system that uses <50W today and have way more storage available than his server had. Also Blue Falcon doesn't want to have it available all the time so the cost of energy are rather low.

I recommend a hdd-based solution, preferably an external hdd enclosure with RAID1-support.
 
He is basing his savings off of a projected 190GB in 5 years? If you have <2TB of data, Atom + 2x 2TB disks in Raid 1 and have the thing power up at midnight / power down at 6am. Save 3/4 of electric costs on <50w.
 
I'm just wondering how long the drive will last in a non-powered static state for a period that could be several years (until a more viable means of storage comes out).
Click to expand...

Make several copies on different disks. The drives may not spin up if not powered for years.
 
Something like WHS is also an alternative, and the disks are active and in use so there are no storage issues. Expandable, and if you build sensibly yourself or get a decent low-powered HP WHS box then it won't be too expensive over time to run.
 
I don't know that hard drives are a great long term storage media. There was an article a while back about the problems hollywood was having in archiving digital media. Reality is that the magnetic domains will wander a bit after time, thus the data can use refreshing. Newer drives may be better about this. Loading up older drives (5-8 years) I have they are unreadable so I would tend to believe this. Ultimately though I think they will be a good solution until something better comes along, and really the entire drive capacity has been growing at a surprising rate compared to what we have been led to believe.
 
I tried reading some DVDs (burned myself) that were 2 years old (or less) and most of them had errors when reading. I still recommend a hard disk based solution if acessed often and maybe tape for long term archiving purposes.
 
haileris said:
1. LTO tapes. You can pick a LTO-3 tape drive up pretty cheaply. Quoted lifetime 30 years, as usual your experience may vary (google/bing/whatever "lto tape life expectancy")
Click to expand...

The drives are expensive new. generally > $1000 US. Tapes can be had for $20 to $40 / per 400GB native tape. Long term the biggest problem with tape is in 20 years will there be a drive that can read the tape? I use LTO2 archives at work now but I have at least 5 technologies of tapes that I have no drive to read the media.

2. SSD drives. Less capacity but probably more reliable, thought I haven't personally looked at these in any great depth as the price per megabtye doesn't make them attractive (to me) at the moment.
Click to expand...

Also these will not make good long term storage as the data retention is only gauranteed a few months to years without being powered up.

3. Look at the cloud. Some great deals in the past on online storage. Issues on trust of course and will (probably) be very expensive for the volumes you are looking for.

As Homer pointed out, whatever you estimate, double it for a backup at the very least!
Click to expand...

There I always worry about bankruptcy, security, bandwidth costs and the long time to get your data. It does not seem very feasible for TBs of data.
 
Last edited:
orpheus88 said:
I tried reading some DVDs (burned myself) that were 2 years old (or less)
Click to expand...

Using good quality media is critical. Also take a look at dvdisaster (free, open source) if you're going to be doing long-term backups to optical media. I would still not rely on it though and would want redundancy of some kind across multiple disks.

Hard disks are cheap, about as cheap (maybe a little cheaper) as blank BluRay, and much faster and more convenient. If you're not interested in NAS and want to keep them offsite in a safe-deposit box or just save power or whatever, I would probably use pairs of mirrored hard disks. Though if you're interested in getting the data offsite sooner, I might go with a dvdisaster-protected BluRay once every week or two, and a single hard disk with the same content for redundancy that'll last you a year or so.
 
Your best bet will probably be build a very low power WHS or Freenas box. Throw in 2 1Tb drives, have file duplication on ( or raid 0 ) and power it on once a week or so minimum.

Obviously I would also backup offsite too, if theyre that important, only you can make that decision. Offsite options could include backing up to the cloud, some internet storage solution. If you use another HD or optical media I reccommend you check for errors/corruption and reburn/recopy as necessary.

Personally, I dont trust optical media. Backing up and recovering over internet is going to be painfully slow. I would NOT trust a HD that I put info on and then stashed away ( would you assume if you bought an auto and stored it for 20 years that it would crank ?? I agree in theory it should, but the problem with theory is we live in reality)

With a small NAS or WHS box you have faster recovery, by powering it on once in a while hopefully you would catch a dying or dead HD , you would be in control of it, it would more than pay for itself quickly, upgrade to more storage easily, etc.
 
jay2472000 said:
Your best bet will probably be build a very low power WHS or Freenas box. Throw in 2 1Tb drives, have file duplication on ( or raid 0 ) and power it on once a week or so minimum.
Click to expand...

I would highly suggest using RAID 1 over RAID 0 for a backup.
 
If you need more storage on your computer get another 2TB SATA drive and always keep a copy of your work on your computer. For backup get a USB or eSATA 2TB drive and store it somewhere safe in your home. Every week or however often you need to to feel comfortable back up the copies stored on your computer to the disk. You could set up software so that all you have to do is plug in the drive and click a button every week. You could keep a third 2TB disk at a relatives ect. and back it up once/thrice a year for extremal high data security.. 2TB disks are about $100 so for $200 (2 USB drives) you get 2TB of reliable and fast backup. It would be inconceivable to have all three drives fail at once. If one of the drives stops functioning one backup just replace it and restore it from one of the other copies.
 
Last edited:
pjkenned said:
I would highly suggest using RAID 1 over RAID 0 for a backup.
Click to expand...

I don't think Jay was meaning to recommend RAID 0, it was probably just a typo. They said "file duplication", which, ignoring for a moment that it's an inaccurate description, I think was intended to mean RAID 1.
 
You must log in or register to reply here.
Back
Top