Need help in organizing and making accessible 20 hard drives

dennismv

n00b
Joined
Jun 27, 2005
Messages
38
I am looking for a solution to an issue ...

TL;DR: I have 20 hard drives full of various data, often not connected to any device, just laying around, and I need a solution to organize data on them, make that data accessible, and organize a back up solution to that data.

And ideally I want to do this without hiring a full time staff of professional data center folks.

Details..
I'm a bit of a pack rat when it comes to storing data, and the way I dealt with it so far was .. when my PC's HDD gets sufficiently full, I put in a new one into my PC, copy essential data that I need onto it, and put the old one on the shelf. I usually got drives in pairs, so I had two of everything. Then I repeat this for a few years. Past year's photos/etc are typically not accessible, and are in few many different places, I don't even know for sure on which hard drive or what folder exactly on what hard drive. I need a good starting platform to lay out all the data in front of me and so I can start organizing it. I also want to make this data accessible to not only me but to others on my LAN. Thus it is not only storage needs that I have, but layout and fair processing power - enough to copy/move files around for a while, and actively work on them sometimes, and not just "store them and forget them".

So current situation is that I have these drives (unconnected):
  • 1TB WD1001FALS
  • 320Gb WD3200
  • 320Gb WD3200AAJB
  • 320Gb 6A320YO Maxtor (2 drives)
  • 2Tb WD20EARX
  • 60Gb WD600 (2 drives)
  • 80Gb D540X-4K (Maxtor)
  • 130.6Mb ST3145A
  • 300Gb BANC1B70 Maxtor (2 drives)
  • 60Gb WD600
  • 1080Mb ST31082A (Seagate)
  • 250Gb WD2500JB
  • 500Gb WD5000AAKS
  • 164.7Gb HDS722516VLSA80 (Hitachi)
Plus, 3 more drives currently occupying various machines

All of these drives with an exception of maybe one or two are FULL, like totally full, in some cases maybe with 5-10% to spare.

The naive approach: for a while I thought I could organize them one at a time - buy a hard drive that is 2 times larger than my current one, put stuff on that, and things will be good. Except .. they weren't - eventually that drive became full and I bought another one 2x larger, repeat. And here we are with that approach.. Even if I get a 20Tb Drive now if one exists, it will probably help me, but then I might end up here later adding the 20Tb drive to my list, plus what do I do with my existing drive eh?
I also wanted to try a thing to where I connect one, go through it, organize it, label it, and then ... put it back on the shelf.. That didn't work very well either as it's back on the shelf again... and how do you organize or use something that is not accessible?

So I am looking for something that will help me see all drives and work with them, storage large enough to where I can manipulate/move around/work on data and have that data be accessible to others as well, and back my data up.

Any advice on organizing things will help too!

Can you help & suggest things from general soft guidelines to specific hardware solutions or anything in between?
 
Well, separate from the question I would put them up for $5 or $10 each and replace them with a single 3TB drive for under $100. That said, for MacOS Neofinder is a good choice, and for Windows WinCatalog should work fine.
 
Grab yourself three things: a 6 TB drive, an external USB SATA caddy, and a file-level deduplication utility. Put the 6 TB drive in your PC and copy on the first several hard drives. Then run the deduplication utility. Repeat until all drives have been copied and the 6 TB drive has been fully deduplicated. Now organise that drive. You are now ready to look at the sharing / NAS / etc possibilities.
 
You'll need an IDE-USB converter as well as at least 2 of your drives are 40-pin.

You definitely want to get a new HDD to store all the data because some of your drives are old enough to potentially give you issues on startup - I have seen drives that won't spin up because the lubrication on the main spindle dried up completely. I literally used a rubber mallet to whack them on the front to get drives in that state to spin up - the procedure was actually written in the HP internal service notes.
 
The naive approach: for a while I thought I could organize them one at a time - buy a hard drive that is 2 times larger than my current one, put stuff on that, and things will be good. Except .. they weren't - eventually that drive became full and I bought another one 2x larger, repeat. And here we are with that approach.. Even if I get a 20Tb Drive now if one exists, it will probably help me, but then I might end up here later adding the 20Tb drive to my list, plus what do I do with my existing drive eh?

I fail to see any problem with this approach. If you copy all your stuff whenever you get a bigger drive, then the old drives are just backups. Why make things more complicated than necessary?
 
Build or buy a NAS.
Thanks
I specifically avoided saying NAS in my post to avoid biasing replies towards suggesting one but I think a NAS will help me for two reasons:
  • online-accessible storage (set my drives up online, them being on when my computer is off)
  • more more data security - RAID with various options
I'm not sure if I want to buy all new drives for one, since I have so many now. But I am not sure if I can or should reuse them.

I see about 6TB listed there plus whatever you didn't include.
The whatever drives didn't include is the approximately 5Tb (2 x 2Tb drives, 1Tb drive and a 120Gb SSD)
I say 16Tb NAS will last me long and nicely. I might even fit myself into 8Tb NAS, if I use the Mdisks and treat carefully but that will not be future-proof. But best is if I have 12Tb accessible data storage with whatever RAID configuration. If I go Synology route, their https://www.synology.com/en-us/support/nas_selector suggests DS416j, which is under 300 + whatever drives I fit into it. If I get 4x4Tb drives + that NAS, it should run me under $850.

About getting a 6Tb Drive ... I forgot to mention I did get an external Seagage 8Tb USB3 drive but it is painfully slow and more so for backup storage. But it does work. Perhaps I can try that first... I'm sure without deduplicator I will max it out easily though. I'd have to copy data onto it in parts and run dedup (while swearing at how slow it is).

You'll need an IDE-USB converter as well as at least 2 of your drives are 40-pin.
I still have access to an awesome older motherboard with IDE connectors.. to where I can plug those drives directly and copy them over the conventional way.
I think my main issue is .. habitually lacking space, and not having drives online + incredible clutter on the drives. Once I organize the clutter I think my life will become a lot easier. I have been afraid to for some reason ..hahhah. I mean when you have a drawer full of drives, it does feel dreadfully disconcerting.

Questions:
Why sell the drives? I mean ... ya I can get some cash for them, but if I let them go for $5-10 bucks, won't I do better by keeping them and using them for off-site backups? Maybe I can even reuse some for the NAS... although I don't know if it's best to just get all new drives for that.

Why triple parity RAID? Why not dual? How many drives will I need to achieve that? i.e should I get 6 or 8 bay NAS from the get go instead of a 4-bay?

Build or Buy - I will probably buy, since I don't want to deal with a DYI-server-NAS (needs building, needs software installation, distros, etc, needs more maintenance, needs monitor & keyboard, needs some upkeep & I just want a thing that makes my life more simple)

I fail to see any problem with this approach.
well .. problem is that I have 20 drives using this approach, lol ... and no sufficiently large space for all of them. I always got drives that are not big enough, and I didn't know what to do with my old drives, so they kept on accumulating & scaring me, whenever I thought about trying to somehow organize them. They are not nicely organized right now they are just full on stuffed with junk and various many projects I began and either finished or left as-is. So part of my concern is perhaps the data organization itself. I want to do something to where I can actually find stuff later rather then go OMG it's one big clutter pile. And that perhaps been my reason for postponing things & not knowing how to approach this mess..

I think my plan of action is
  • try the 8Tb drive + dedup
  • I'd have to convince myself it is necessary as it's over $800, but I think it will definitely help me with online storage to get a NAS
 
Murphy's Law of hard drives says that no matter how big a drive you buy, you will find a way to fill it up.

(also, the drive always fails the day before you do a backup)
 
Triple-parity RAID is because you seem to want the data pretty badly if you refuse to destroy drives - it ensures you have to break 3 drives simultaneously to get into real trouble.

Most of your drives are old enough and small enough that selling is a waste of time, once you copy the data break all of them below 500GB with a hammer to break your habit of hoarding.

USB 3.0 shouldn't be that slow on a mechanical hard drive (SSD is another story) - any chance you plugged the drive into a USB 2.0 port/hub?

The NAS not only handles a lot of your duplication issues (you save data on a drive because you aren't sure it isn't on another drive), by consolidating your space it lowers your wasted free space - instead of 10GB chunks free on 10 drives you have 100GB free on the NAS. This is on top of the increased data integrity by going dual or triple parity RAID.
 
For your use case (and behavioral norms) consider two separate systems. RAID etc. help address system failure, but your past strategy (physical segmentation) suggests you're also worried about human error e.g. accidentally deleting files, malware, physical loss of the whole unit. As someone who definitely is that way, in addition to the above good advice I would recommend looking at two parallel options. One could be a NAS and the other could be a drive hanging off a PC. Or one could be local storage and the other could be cloud based. Regardless, you may want to get a couple of parallel systems in place and some kind of replication strategy.
 
Unstoppable Copier, great for mass file transfers. and it will tell you if there was an issue with any file without stopping the copy
http://www.roadkil.net/program.php?ProgramID=29

Copy files to your new NAS setup or to a larger drive setup. Then backup.

I would recommend building a ZFS NAS if you can afford it, depending you your hard ware you can get refurbished 2 TB drives for $40 now. Or use Mirror zRaid with 2 or 3 Large drives and added new mirror vdevs as you need.

Also cloud backup or offsite backup. I recently resetup my arrary after 3 drives died on the same day with 11x4TB Raidz3 setup x4 vdevs + hotspare (total 45 4TB drives) without losing any data.
Right now best price per GB seems to be 4TB, but going for 6 or 8 TB means you need fewer drives (and less power, heat, noise) I kept to 4 TB since I already had 30 drives and did not want to split my array.

best example. it seams you have about 4.5 TB of data (if every drive is full) you can have a Mirror ZFS setup using only 6 2 TB drives. so about $250 for 6TB of usable space and 200-300 on the base hard ware (used quad core intel 1366 + 16/32GB Ram, 650 Watt PSU, Case, CPU Cooler, Motherboard, HBA) so a total 450-550 for the setup. Built systems will cost you twice that.

Buying a prebuilt system is ok, but it will skyrocket your price per GB. Also since each drive is mirrored or has a backup drive. if one drive fails you will not lose data.
 
Last edited:
I would do the NAS setup and copy the data to it (de-duping as you go), but then get a grip on your organization habits and avoid saving everything.

Do you have a dresser drawer full of things like car repair receipts from 1992? lol
I'm guessing that kind of thing has carried over to how you save data as well.

It may also be wise to save any critical data to M-Disk archive backups.
The M-Disks are like DVDs that last a very long time.

There are inexpensive optical drives that support M-Disk.
The blanks do cost a bit more though.

"M-DISC's design is intended to provide greater archival media longevity.[3][4] Millenniata claims that properly stored M-DISC DVD recordings will last 1000 years."

https://en.wikipedia.org/wiki/M-DISC


.
 
Back
Top