How does everyone archive data these days?

R

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
I have lot of data on spinning disks that is archive data that I want to keep, but don't really need to access very often nor does it ever change. Just wondering what everyone does these days to archive stuff. I have around 900GB or so.

Simplest is to probably store it on hard drives, but those can fail. Open to other options.

I could probably do a pretty big cleanup on my storage arrays if I archived all this stuff somewhere safe and half decently easy to access. This would also make my backup jobs run faster, as even if the data never changes, rsync has a mind of it's own and sometimes likes to backup stuff that has not changed.
 
EXT HD stored in multiple location for "bulk" (TBs)

Specific stuff that may be more routine or more off-site locations and not as much space needed... M-DISC BLU-RAY (25GB/EACH) -- I don't know the "real life" but the testing/certs are impressive, and it's cheap insurance it seems.
 
Tarsnap, its uber cheap and I generated my own keys so it doesn't even matter if someone gets it
 
Soon will have a SnapRAID set up and use BDs for most important stuff like TC (Veracrypt) containers, pictures, medical records, taxes, and so on. All the above are being put in TC Containers on the BDs. I hope to finish moving all that personal data to BDs in the next month to 2.
 
Simple, two copies on Seagate Archive 8TB HDDs.

I don't know about any archive solution that's easy to access. Worst thing about external HDs is that you have to keep a record on what data is on which disk.
 
Archival data is secure long term storage, what means, that you need
- two or more external copies of data on different places
- data must be verified by checksums
- data must be stored in a way that they cannot be modified by chance or by intention
- you must face the silent data error problems (random biterrors on disks that occur over time)

A common solution can be one or two ZFS NAS systems like a HP Microserver G8
that you use as a filer for your data

- the readonly aspect is done by a versioning with ZFS snaps. They are readonly by design.
- The verification and repair of silent errors is done by a scrubbing process based on realtime
checksums that you must run from time to time. You can run a scrubbing online in the background.
- The two or more copies problem can be done with a second NAS or if you use two
of the 4 bays for a mirror and the remaining for regular backups that you keep elsewhere.
(Use a ZFS pool with checksums from a single disk or a mirror for the second backup )

I would use 2 x 4 TB disks to have enough capacity for versioning the data
ex you store all data on the NAS and do snaps example daily and keep them forever so
you can go back to the data state of a special day long ago.

In my own setup, I use a main ZFS filer with snaps for daily work and two or three ZFS backup
filers on different locations with daily replications and their own snap history.
 
Not the most robust, but I think what I have done is generated a method where I am perfectly safe..

99% of my data stays the same. Videos, tv shows, music, etc. SO I have eight 8TB drives drivepooled together. I use drivepool. I set it for duplication, and I can withstand 2 drive losses together before I lose data on the drivepool.

So twice a day, I have my entire 30TB, (yep), backed up to the cloud using SOS backup. I set it to not delete anything. So if at any point, I did lose let's say 3 drives, or my whole machine to a fire, I could rebuild a new machine, install SOS and bring all of my data back down and make it just as it was. :D
 
Archive, as in things I make safe backups of but don't use on a regular basis, I have started to use Bluray discs. I put all our photos on about 8 25GB discs and keep them at work. I have started to buy the 50GB discs....you can find them for about $2 a disc on Amazon with free shipping if your wait.

It's not as cost effective as a HDD, but like people have said, those harddrives can fail at any minute...1 part goes bad and it's downhill. Optical really only has to worry about time (which is unknown at this point....but should be at least 10 years...and probably more like 20).

Bluray has better specifications for durability then DVD. M-Disc DVD is very good, but the price per gigabyte is not very effective.
 
westrock - They have M-DISC Blu-Ray too, mcuh higher capacity than DVD.
 
ToddW2 said:
westrock - They have M-DISC Blu-Ray too, mcuh higher capacity than DVD.
Click to expand...

yea i use Mdisc and regular. As I start to back up most important stuff to BD. I do 2 Mdisc and 1 regular BD so I have 3 copies so I can ensure that I have 3x buffer against bad writes, bit flips, and decay. I only do this for encrypted containers with photos/videos (ones I have taken), records, and important files. I don't back up games or burned material. I only back up original content that can't be reacquired.

I am debating to switch to 50 GB discs for simplicity and storage effectiveness.

I might switch to 100GB actually

http://www.amazon.com/Verbatim-Disc...8&qid=1449426240&sr=8-1&keywords=m-disc+100gb

sub 20 bucks per disc. at least use these for videos since videos are so large you can get a lot of waste on 25GB discs.

Original content that I have is probably close to 1 TB so to maintain 2 copies on discs would require 20 of these :/ ouch. I might have to rethink my plans now when I think about how much original content I have :/
 
Last edited:
As an Amazon Associate, HardForum may earn from qualifying purchases.
M76 said:
Simple, two copies on Seagate Archive 8TB HDDs.

I don't know about any archive solution that's easy to access. Worst thing about external HDs is that you have to keep a record on what data is on which disk.
Click to expand...

That's kinda what I was leaning towards too... I might just do that. I already have a raid setup and everything for my live data, but the point of this is to get data OFF of that setup so it no longer needs to be part of the backup routine and taking up space. So I might go ahead and just get two of those big drives and dump lot of that data on it. The odds of both drives failing is probably slim so think that should be good enough. Might get two different model drives to be safe. Actually I could maybe just get a NAS enclosure so I can do raid 1 too, but I think I'll go with just doing two separate copies that way I'm not depending on another piece of hardware.

At 8TB I could store all my archive data so I don't have to worry about record keeping. Once it gets past that point I could easily do a recursive dir and then just keep the file somewhere.
 
Red Squirrel said:
That's kinda what I was leaning towards too... I might just do that. I already have a raid setup and everything for my live data, but the point of this is to get data OFF of that setup so it no longer needs to be part of the backup routine and taking up space. So I might go ahead and just get two of those big drives and dump lot of that data on it. The odds of both drives failing is probably slim so think that should be good enough. Might get two different model drives to be safe. Actually I could maybe just get a NAS enclosure so I can do raid 1 too, but I think I'll go with just doing two separate copies that way I'm not depending on another piece of hardware.

At 8TB I could store all my archive data so I don't have to worry about record keeping. Once it gets past that point I could easily do a recursive dir and then just keep the file somewhere.
Click to expand...

http://hardforum.com/showthread.php?t=1884399

started a thread on getting a go to place for M-Disc prices. You can get 100GB discs for sub 16 bucks now. Not too bad for what it is. I am fairly small still for original data so I can get away with the M-Disc set up for sure for only a couple-few hundred bucks.
 
Yeah did not realize optical was at 100GB now. Is optical more reliable than it used to be though? I remember burning CD and DVD's more often before and the success rate was maybe 50%, otherwise it would error out half way through. I'd hate for that to happen when I'm paying that much per disc. Though If I could look into how to split the data logically it would perhaps be more reliable than hard drives, once it's been burned.
 
Red Squirrel said:
Yeah did not realize optical was at 100GB now. Is optical more reliable than it used to be though? I remember burning CD and DVD's more often before and the success rate was maybe 50%, otherwise it would error out half way through. I'd hate for that to happen when I'm paying that much per disc. Though If I could look into how to split the data logically it would perhaps be more reliable than hard drives, once it's been burned.
Click to expand...

i had about 80-90% on DVDs only burned like 6 BDs and 100% so far. But yea a failed disc costing 16 bucks will piss me off.
 
Currently an external hard drive that doesn't get used much for the bulk. Dropbox for the non-sensitive docs and school work. Encrypted flash drive in my safe for the rest of it.
 
Red Squirrel said:
Yeah did not realize optical was at 100GB now. Is optical more reliable than it used to be though? I remember burning CD and DVD's more often before and the success rate was maybe 50%, otherwise it would error out half way through. I'd hate for that to happen when I'm paying that much per disc. Though If I could look into how to split the data logically it would perhaps be more reliable than hard drives, once it's been burned.
Click to expand...

I'd be more concerned with the endurance of the written discs. Those early 2x DVD-R discs that I've written, were almost completely unreadable after 5 years. Same story with CD-R discs, but in that case, the later high speed discs died first, the early ones from almost 20 years ago are still readable.

But for me that's not feasible. Unless an optical disc comes around with the bare minimum of 1TB capacity, I don't care. I don't want to write hundreds of discs. Even with 1TB capacity it would take more than a dozen discs to store my data, double that for redundancy.
 
yeah m-disc here also. And dropbox + MEGA. I know those aren't supposed to be backups but there you go. Can't have my kids childhood disappear. I should use my 35mm minolta more often though.
About early cd-r...I had a few audio cd's in my car over a winter and the layers separated. Fucking awful.
 
M76 said:
I'd be more concerned with the endurance of the written discs. Those early 2x DVD-R discs that I've written, were almost completely unreadable after 5 years. Same story with CD-R discs, but in that case, the later high speed discs died first, the early ones from almost 20 years ago are still readable.

But for me that's not feasible. Unless an optical disc comes around with the bare minimum of 1TB capacity, I don't care. I don't want to write hundreds of discs. Even with 1TB capacity it would take more than a dozen discs to store my data, double that for redundancy.
Click to expand...

thats why you use the most crucial data for optical. office files, pdfs, emails, records (taxes, medical), and photos. I have 1000s of photos that are RAW and JPEG and I still dont have even close to 1TB. Add 1080P videos I might have 1 TB now. These are all home original content....stuff I made. Not movies I have ripped. Do you honestly have TB of word docs and PDFs and home photos?

or are you referring to office stuff?
 
I used to use an ESATA HDD dock. I used 2 HDD's (2 copies) per filesystem, each HDD had a ZFS on there and I send/received snapshots around. This stopped working for me as soon as I had ZFS filesystems that contained more data then a single HDD could hold. I now have a secondary offsite server like _Gea suggested.

If you are considering blurays, don't! Use 2 USB3.0 external drives. In the end much cheaper. I recommend making 2 copies and storing one offsite (at your parents place or something) in case the building burns. Imagine having to tell the wife all the baby pictures are gone because you skimped on a 100 buck harddrive...

You say you have 900 gigs, that is very doable with externals. Update your 2 backups every month or so.
 
ZFS also allows for duplicating data on one drive (or more duplicates if you choose). This would allow you to have some redundancy on the data while maintaining a single portable drive. It would not protect against physical failure though.
 
As long as Blu-ray discs is not of the LTH type (which, AFAIK, only exist for the 25GB version), any blu-ray discs should last significantly longer than non-MDisc DVD's due to the nature of the recording medium.

The main reason for a lot of the DVD's to go bad is because they use organic dye layer to retain the data, and organic dyes are prone to UV light degradation. M-Discs and HTL Blu-ray discs uses inorganic layer to record the data, and thus are much more robust to environmental changes.

LTH Discs on the other hand also uses organic dye, and thus share the UV and environmental weakness of their DVD-R's, while being also cheaper. For short-term storage, LTH discs should, though some players may require firmware update because all Blu-ray drives originally was only designed to read HTL, which is the reverse of LTH. For long-term, avoid LTH discs. I do not believe Multi-Layered LTH discs exist.

French Culture & Communications ministry did a stress test on Blu-ray discs for archival storage in 2012: http://www.archivesdefrance.culture.gouv.fr/static/6187

One of the most reliable brand of 25GB discs turn out to be Panasonic's. Verbatim's 25GB HTL discs were decent but still not as good as Panasonic, while their LTH discs was much worse due to their organic layer.

I personally have started using BD-R's to backup my video files, but I have stuck strictly to the 25GB discs, and never writing them to capacity, as I wanted to avoid the outer edges of the disc, where the disc is most prone to failure. There is no way to avoid the edges in a multi layer disc due to the way the data is burnt onto the disc, however a quality BD-R should not immediately run into that issue.

This is the type I use:

http://www.amazon.com/PANASONIC-Blu-ray-Recordable-Spindle-Printable/dp/B005RY6JEU

Despite it being rated only for 1x ~ 4x, I have burnt to 6x without having any issues.

Also, a word of advice.

Any Blu-ray burner can burn M-Disc BD-R's because BD-R originally was inorganic type disc, and M-Disc BD-R's fully comply with BD-R standards, so no special drive is required. However M-Disc DVD would require a special M-Disc DVD burner, which usually is on the front cover of the drive (if it doesn't have the words M-DISC stamped, it probably won't be able to burn M-Disc DVD's). Non M-Disc DVD burners usually do not have a laser powerful enough to etch the inorganic layer of M-Discs, hence why special burner is needed.
 
Last edited:
As an Amazon Associate, HardForum may earn from qualifying purchases.
Sigh...

And how fast exactly is 6x blu ray? Google says 1x = 36 MBps. Times 6, divide by 8 (bits/bytes conversion) is ... A whopping 27 MB second ... USB 2.0 external drive is faster.

(Lets ignore for a second that bluray drives tend to be CAV, meaning that they will only hit 6x on the outside of the disc.)

Let's imagine we have to restore the OP's 400 gig from BD. That is (400000/27)/(60*60) = 4.12 hours, not counting the switching of the discs. At least four frikkin' hours you have to sit there and swap BD discs.

So blurays are slow, expensive, too small and a hassle if you have a lot of data.

2 500 gig external USB 2.0 drives would be already heaps better for the OP then bluray. And E-SATA will eat the BD's for lunch.

Let us not delay the inevitable demise of optical media, they suck huge donkey nads and should go the way of the dodo as fast as possible. If by some miracle it happens quickly, it will be the only thing that optical media has EVER done quickly!!
 
If I'm not mistaken, bit rot could still affect those hard drives, even if they were offline and offsite, however it cannot affect optical media which is smaller, easier to store and may even last longer with better data integrity. If the goal is to archive the data for some period of time so that is it useable, it would seem to me that optical media is the best way to go about it, unless I'm completely mistaken. Is tape even better perhaps?
 
rsq said:
Sigh...

And how fast exactly is 6x blu ray? Google says 1x = 36 MBps. Times 6, divide by 8 (bits/bytes conversion) is ... A whopping 27 MB second ... USB 2.0 external drive is faster.

(Lets ignore for a second that bluray drives tend to be CAV, meaning that they will only hit 6x on the outside of the disc.)

Let's imagine we have to restore the OP's 400 gig from BD. That is (400000/27)/(60*60) = 4.12 hours, not counting the switching of the discs. At least four frikkin' hours you have to sit there and swap BD discs.

So blurays are slow, expensive, too small and a hassle if you have a lot of data.

2 500 gig external USB 2.0 drives would be already heaps better for the OP then bluray. And E-SATA will eat the BD's for lunch.

Let us not delay the inevitable demise of optical media, they suck huge donkey nads and should go the way of the dodo as fast as possible. If by some miracle it happens quickly, it will be the only thing that optical media has EVER done quickly!!
Click to expand...

BDs are long term storage...not regular back ups. They are used to prevent total loss. They are the back up of your back up.....The hope is to never need those BDs but if you have a total failure of your NAS or a natural disaster those BDs will have your most critical and precious data backed up. No one is advocating using BDs as your primary back up solution.

So your point is completely moot.

Example:
I will have a NAS with 30-40TBs. My critical data will be about 1 TB. This critical data rarely changes or not drastically. If my house burns or my NAS blows the fuck up. I will have my most important data on those BDs as a last resort fail safe. It is unlikely I will ever use them but in the heaven forbid moment I do need them. I will be thankful I used them because I will still have my most critical data and will gladly take a day to restore data that would have been completely lost otherwise.
 
I'm not convinced that writable optical media is likely to have a long life. AFAIK it is not designed for long term archival. When I first started taking digital photos I used CD-R backup and lost some due to the disc being no longer readable. Later i moved to using external HDDs like WD MyBooks. However, I then came to realize these things have the lowest quality junk HDDs in them ("Green" range). I had quite a few and they did fail. At least there is some warning like chkdsk failures. I looked into the "archive" range of HDDs but as far as I can see, these are just normal HDDs with a lower run time life expectancy.

Instead of this nonsense I've moved to use LTO tape which is designed for the purpose. The tapes are rated at 30 years archive life and seem to be much more suited to being transported than HDDs. The capacity is much larger - 1.5Tb for LTO5 vs 25Gb for a BD. The tapes themselves are relatively cheap. With this I am able to do a double backup (onsite & offsite) of the entire collection of HDDs I have online plus keep more only on tape. It has saved me keep buying more and more HDDs. I've backed up about over 15Tb but in duplicate, so 30Tb overall.

Still, this was a pain to setup and not suitable for everyone. If you have multiple Tbs of data and are filling racks full of storage enclosures then it probably is! If anyone is interested I can give info on how to setup.
 
CMG said:
I'm not convinced that writable optical media is likely to have a long life. AFAIK it is not designed for long term archival. When I first started taking digital photos I used CD-R backup and lost some due to the disc being no longer readable. Later i moved to using external HDDs like WD MyBooks. However, I then came to realize these things have the lowest quality junk HDDs in them ("Green" range). I had quite a few and they did fail. At least there is some warning like chkdsk failures. I looked into the "archive" range of HDDs but as far as I can see, these are just normal HDDs with a lower run time life expectancy.

Instead of this nonsense I've moved to use LTO tape which is designed for the purpose. The tapes are rated at 30 years archive life and seem to be much more suited to being transported than HDDs. The capacity is much larger - 1.5Tb for LTO5 vs 25Gb for a BD. The tapes themselves are relatively cheap. With this I am able to do a double backup (onsite & offsite) of the entire collection of HDDs I have online plus keep more only on tape. It has saved me keep buying more and more HDDs. I've backed up about over 15Tb but in duplicate, so 30Tb overall.

Still, this was a pain to setup and not suitable for everyone. If you have multiple Tbs of data and are filling racks full of storage enclosures then it probably is! If anyone is interested I can give info on how to setup.
Click to expand...

There are like 3 kinda of BDs as far as i know. LTH which are usually organic and have lower shelf life, HTL that usually isn't organic, and MDisc. MDisc and the non organic ones have long shelf life. For archiving I am going to be using MDisc.

I am also doing 2 copies of my data to ensure its survival.
 
I use WHS 2011 to backup all of my systems and it also acts as my file server. All files are stored on two separate drives on this box using StableBits Drive Pool Extender. I have a second server at home that has a copy of all of these files as well.

For offsite I have a 5TB external drive that acts as my offsite backup. I also subscribe to Backblaze and have uploaded all of my data files there.
 
I have an Amazon cloud where I load pictures that I don't want to lose.

For archiving, I use a spare 1 TB hard drive, and use Always Sync http://allwaysync.com/
I use an adapter via USB3 for the copies and running Always Sync. If my 1 TB drive craps out, I have a perfect working copy that I slap in. I would then order another 1 TB drive for the backup.

For my boot drive, I keep 2 copies using Acronis True Image on the same 1 TB drive.
 
Local raid with checksums (but not ECC ram :( )
USB HDD
Onedrive encrypted VC partition.
 
You must log in or register to reply here.
Back
Top