Saving Money by Using Used Hard Drives

Zarathustra[H]

Zarathustra[H]

Extremely [H]
2FA
Joined
Oct 29, 2000
Messages
42,437
So,

I have always purchased brand new hard drives, simply because I am concerned about data loss, but I was curious how well a used set of drives could work.

So, in October last year I decided to do an experiment, and I figured I might as well share my results with everyone here.

I put my used decommed server board to use in my Testbench system, popped a SAS HBA in, and decided to buy some cheap used hard drives from eBay to see how well they would work for me.

I went poking around on ebay. I wanted to get a lot from one seller, and I wanted to go with a seller that was well reviewed, and appeared to focus on decommed enterprise gear (as enterprise stuff tends to ahve been better treated on average than consumer stuff)

I wound up getting a lot of 9 HP branded 4TB 7200rpm HGST HUS726040AL5214 drives with 12GB/s SAS interfaces, for a total of just north of a hundred bucks. I figured, if I was going to get used drives, at least I should get the brand that is best known for its reliability, so HGST it was.

Once they arrived, I checked what I got.

Drive data output (whatever the SAS equivalent of SMART is called) showed me they all had ~39k hours (~4.45 years) on them despite being about 9 years old based on manufacture date.

I also read the drive data. The results. There was a smattering of ECC corrected errors here or there. Some a little better, some a little worse. This is to be expected, but doesn't mean there was ever a data error. Eight of the nine drives had zero "Elements in Grown Defect List". Essentially, my understanding is that this is the key data to look at when it comes to SAS drives.

The results:

1739590361721.png


Then I dd:ed them full of zeroes to make sure they were clean, and then did a full "badblocks" test that writes random data to the entire drives in four passes.

(This took a little while)

None of the drives had any read or write badblocks errors.

Then I pulled the drive data again.

1739590718147.png


Still not that bad. No new "Elements in Grown Defects List" (which would signal it being time to replace the drive)

Drive 7 did show a significant growth in ECC errors though. These are not critical. That means that the drive caught the errors and fixed them itself, but it is still a sign that something might be going on.

In the grand scheme of things, this was really not bad for drives I did not trust at all.

I notified the seller of the "Elements in Known Defect List" for drive 5, and he gave me a refund of 1/9th of the order.

I considered Drive 5 to be bad at this point and decided not to use it. I then had 8 remaining drives.

Of these drives I decided to select the 6 best ones (which I deemed to be drives 1, 2, 4, 6, 8 and 9 and built a ZFS pool out of them. The remaining two drives (3 and 7) I put aside as spares, should I have a drive failure. Based on the errors they had, I decided to make drive 3 the first spare, and drive 7 the second spare.

Since I did not know what to expect, I decided to go with a single 6 drive RAIDz3 pool (in other words, it can survive three drive failures before guaranteed data loss). Normally I go with RAIDz2, but I felt it was pertinent to add a little more redundancy with used drives.

It has now been four months.

These drives have not been running 24/7 like the drives in my main server do, but rather they sit in my testbench/nackup workstation. Roughly once a week I power it on, and read backups from the main server. I also do some drive image dumps to them, and stuff like that.

In total I have probably written about 14TB of raw data to the drives at this point. Because I am still a little paranoid, I have been doing a "zpool scrub" to check for errors approximately once a week. I am happy to report that in that time I have not had ZFS report even a single flipped bit.

This is a rather limited sample size, from just one seller, and one model of one brand (now defunct) of drives, so make of it what you will, but I guess this little experiment goes to show, that maybe one doesn't need to fear used drives after all.

At least as long as you thoroughly test the drives before first use, use decent redundancy and are willing to expect some fallout and keep some spares handy.

I'm probably not about to replace all of my drives in my main server with used ones any time soon, but it is good to know that drives destined for secondary or backup servers can probably be sourced cheaper used.

I hope someone finds this useful and worth the time it took me to do this brief writeup.
 
Last edited:
Zarathustra[H] said:
At least as long as you thoroughly test the drives before first use, use decent redunancy and are willing to expect some fallout and keep some spares handy.
Click to expand...

Nice read. Couldn't have said it better haha. I am dealing with multiple failed drives at the moment. Thank GOD I didn't put them in the same vdev haha. Would have been toast.

I think it's the last time I buy multiple drives in the same order/batch for failure purposes. I can tolerate some failure but damn 2 disks at the same time kind of stings lol.
 
Zarathustra[H] said:
I considered Drive 5 to be bad at this point and decided not to use it.
Click to expand...

FWIW, with 10 defects, I'd consider it fine as long as you have monitoring and a backup/availability plan, I don't consider it bad until it hits 100 (or is growing at 10-20 a day), and I've had enterprisey sata drives work fine at 500-1000. Background for my numbers is I was pulled in at work when we had some issues with read/write errors on a couple thousand drive fleet; all of the failing drives had thousands of bad sectors according to smart, but we weren't monitoring. After starting monitoring, we went from having to deal with drives actively failing to having warnings and being able to evacuate data and get drives replaced on our schedule. Our SSD drives didn't fail nearly as often, but also we only had a couple instances where SMART data indicated a pending problem before a real problem.

If you don't want to monitor though, 10 seems like a good place to stop.
 
toast0 said:
FWIW, with 10 defects, I'd consider it fine as long as you have monitoring and a backup/availability plan, I don't consider it bad until it hits 100 (or is growing at 10-20 a day), and I've had enterprisey sata drives work fine at 500-1000. Background for my numbers is I was pulled in at work when we had some issues with read/write errors on a couple thousand drive fleet; all of the failing drives had thousands of bad sectors according to smart, but we weren't monitoring. After starting monitoring, we went from having to deal with drives actively failing to having warnings and being able to evacuate data and get drives replaced on our schedule. Our SSD drives didn't fail nearly as often, but also we only had a couple instances where SMART data indicated a pending problem before a real problem.

If you don't want to monitor though, 10 seems like a good place to stop.
Click to expand...
Agreed. OP is overthinking it and being too much paranoid. I only buy used drives nowadays, I've saved thousands of dollars, yet to have one carefully selected and bought used drive to die on me.
 
Ten months ago I bought 13 (one spare) 10tb HGST SAS drives. They have been on pretty much the whole time and I have yet to have any problems.
 
All I buy are used drives. I almost always go to Goharddrive.com to get them. And always make sure they have 5 year warranties. A simple email with serial number to them saying a drive has errors or wont spin up, etc and I get a label that day or early the next day. Why buy new when I rarely have failures and, in my opinion, all these drives are proven with the hours on them. You actually really never know what you are getting from new drives.

So used is the way to go.
 
Last edited:
As an Amazon Associate, HardForum may earn from qualifying purchases.
toast0 said:
If you don't want to monitor though, 10 seems like a good place to stop.
Click to expand...

I've always heard it suggested that any non-zero value for "Elements in Grown Defect List" means that drive is done, and should no longer be used.
 
Zarathustra[H] said:
I've always heard it suggested that any non-zero value for "Elements in Grown Defect List" means that drive is done, and should no longer be used.
Click to expand...

I think that's too conservative, drives have spare sectors for a reason. A small amount of reallocated sectors isn't a big deal, especially since it passed a full disk read/write test. But, if you want to set 0 as your threshold, that's ok. It's probably true that a drive is more likely to go from 1 -> 2 than from 0 -> 1, and there's always a chance it's an unrecoverable read, which is no fun.
 
High RAID levels also make used drives much more attractive.

Just be sure that you do practice exchanging drives so that you don't have surprises in the real cases. And that you have a spare on the shelf.
 
Used to think used drives were dumb but I've bought a few 14-20TB drives without issue in recent years. Had good luck with them. I'm looking at them now to build a replacement software RAID 1 NAS/file server with used 18-22TB drives. I've always had a machine for IT "nice to have" data and near-line backups of backups. Current box is failing and it's 9+ years old.

If a server or workstation is looking unhealthy we can copy data there to move things around or just as a precaution. When people leave the company and no-one wants their data... we sometimes know someone is going to come asking about it eventually so it gets squirreled away there. I don't want to back it up routinely but not ready to delete it. Not a big loss if we lose the data.
 
You must log in or register to reply here.
Back
Top