Hi All,
I'm a long time reader and first time poster. Some posts here really helped me build my last file server. Thanks! Now I have a new problem hope to get some advise on.
I have a very LARGE (for home user) data collection for my personal data-mining project. Basically financial data go all the way back including PDFs, txts, pics, that were available from various public sources. The total size today is about 50+ TBs, spanning 20x 3TB Mixed WD and Seagate drives (I fill each drive to 90%) and growing at 5TB/month rate. Compression doesn't help much as a lot of files are PDF scans.
My home-grown Norco 4224 FlexRAID dual parity file server is close to be completely filled, and I'm about to build a second JBOD enclosure to attach to the head file server. However, given the shear amount of data that I have (which I was actually shocked when I realized how much crap I accumulated in the last two years), I'm beginning to worry that I may one day lose it all and have no way to get them back. So I'd like to design a backup/restore system in case of fire/flood/system corruption etc.
Here are a few points:
1. The data is not too important. I can theoretically get them back in case I lost them but I have to pay more subscription fees (which cost major $$$). The bottle neck is the download speed. It took me TWO YEARS to accumulate this much data. There is no way to speed up this process because the restriction is limited on the other side (e.g. SEC only allow bulk download at nights and limits speed per IP).
2. I'm a HOME user, I don't make money on these things (at least not directly, I trade stocks), so my budget is limited (<10K total), so no SANs or expensive pre-boxed NAS solutions (e.g. Drobo). Can't afford them. The file server is off most the time, no need for 24x7 operation. I'm the only user.
3. I live in a Condo so I can't have LOUD servers or fans running all the time.
I've only recently heard about LTO tapes. I'm surprised they still exists. But after some research, it looks like they fit my requirements for Backup / Archive. But I've never heard of them before and know very little about them. The media seems cheaper (~$40/1.5TB vs ~$130/3TB disk), but the drive is a large expense (~$2000 for LTO5).
I did some calcs, if I don't count the cost of HBA cards or another case/motherboard/PSU etc, the break-even between the tape and disk solutions is at ~70TB. Beyond that tape is cheaper.
If I do count all that (i.e. build another fileserver for replication and offline storage), the breakeven is ~40TB.
So economically it makes sense.
But what are the down-sides? Is this a good idea for my situation? Should I rent a bank-deposit box to store all these (40+) tapes?
How are the supports for these tape drives? Say If I get a Quantum/HP/IBM/Dell one from Newegg, how's the support from Quantum/HP/IBM/Dell?
Also since LTO6 just came out, should I hold out a few month to get into LTO6 instead of LTO5 right now? How fast would LTO6 drive and media price go down?
Again, any insights comparing the two solutions for HOME users are extremely appreciated.
Sorry for the long post.
I'm a long time reader and first time poster. Some posts here really helped me build my last file server. Thanks! Now I have a new problem hope to get some advise on.
I have a very LARGE (for home user) data collection for my personal data-mining project. Basically financial data go all the way back including PDFs, txts, pics, that were available from various public sources. The total size today is about 50+ TBs, spanning 20x 3TB Mixed WD and Seagate drives (I fill each drive to 90%) and growing at 5TB/month rate. Compression doesn't help much as a lot of files are PDF scans.
My home-grown Norco 4224 FlexRAID dual parity file server is close to be completely filled, and I'm about to build a second JBOD enclosure to attach to the head file server. However, given the shear amount of data that I have (which I was actually shocked when I realized how much crap I accumulated in the last two years), I'm beginning to worry that I may one day lose it all and have no way to get them back. So I'd like to design a backup/restore system in case of fire/flood/system corruption etc.
Here are a few points:
1. The data is not too important. I can theoretically get them back in case I lost them but I have to pay more subscription fees (which cost major $$$). The bottle neck is the download speed. It took me TWO YEARS to accumulate this much data. There is no way to speed up this process because the restriction is limited on the other side (e.g. SEC only allow bulk download at nights and limits speed per IP).
2. I'm a HOME user, I don't make money on these things (at least not directly, I trade stocks), so my budget is limited (<10K total), so no SANs or expensive pre-boxed NAS solutions (e.g. Drobo). Can't afford them. The file server is off most the time, no need for 24x7 operation. I'm the only user.
3. I live in a Condo so I can't have LOUD servers or fans running all the time.
I've only recently heard about LTO tapes. I'm surprised they still exists. But after some research, it looks like they fit my requirements for Backup / Archive. But I've never heard of them before and know very little about them. The media seems cheaper (~$40/1.5TB vs ~$130/3TB disk), but the drive is a large expense (~$2000 for LTO5).
I did some calcs, if I don't count the cost of HBA cards or another case/motherboard/PSU etc, the break-even between the tape and disk solutions is at ~70TB. Beyond that tape is cheaper.
If I do count all that (i.e. build another fileserver for replication and offline storage), the breakeven is ~40TB.
So economically it makes sense.
But what are the down-sides? Is this a good idea for my situation? Should I rent a bank-deposit box to store all these (40+) tapes?
How are the supports for these tape drives? Say If I get a Quantum/HP/IBM/Dell one from Newegg, how's the support from Quantum/HP/IBM/Dell?
Also since LTO6 just came out, should I hold out a few month to get into LTO6 instead of LTO5 right now? How fast would LTO6 drive and media price go down?
Again, any insights comparing the two solutions for HOME users are extremely appreciated.
Sorry for the long post.