Dying drive: Where do you start?

[Coldblackice]

If you have a drive that looks to be on its way out, with SMART warnings popping and intermittent usability, where do you start, in terms of an emergency data-evac?

(...while promising to rail on yourself later for not backing up)?

I've got a 3TB WD Red drive that looks to be dying, and I want to maximize what I can pull off of it before it's too late. Any suggestions for the best ways/tools to go about this?


-----

Do you:

• Start copy/pasting files and folders to another drive from within a booted Windows (assuming the drive in question is just a non-OS/primary storage drive)?
• Boot to a Linux distro and start dd'ing the drive, bit by bit?
• Use some pre-boot imaging software like Acronis to try and image the drive?
• ...etc.

I'm curious what others do when this happens (assuming no backup) -- do you quickly try to copy by file/folder, or do you try to image in some way?

My experience is that, when a drive's on its way out, there's no time for imaging -- the drive will probably give up the ghost before an image could complete, rendering the attempt void. But with file + folder copying, not only can you selectively target the most important things, but if a certain file is unreadable, you can jump to another.

But perhaps I could better maximize data recovery by doing something else. I'm interested in maximizing the amount of data I can pull off, rather than selectively targeting particular specific data -- so if there's any type or tool of imaging process that can work with bad/unreadable sectors without getting stuck, and/or would still result in a useful image, even if it's only halfway complete because the drive dies outright halfway through, I'd love to hear suggestions.

 

[drescherjm]

Here is what I would do (and have done several times at work and otherwise):

1. Unplug the drive for now. The more you use it the lower the chance of recovery.
2. Purchase a 3TB or larger drive.
3. Use ddrescue (available on most linux livecd/usb media like sysrescuecd) to make a bit for bit copy of the bad drive to the new. ddrescue will copy as much of the bad drive to the new drive as possible.
4. Check the copy for errors.

 

[patrickdk]

use ddrescue, and use the log option. Run the first pass without retries. Then run a pass doing like 3 retries. Next power down the drive and let it cool off for awhile. Then try again. It will only attempt the parts it couldn't read correctly before.

Normally I found letting the drive cool helps it get more data. Normally I can recover all data, or only missing like 8k of data, using this method. Assuming the drive still spins.

 

[tdowning]

If SMART is throwing errors, you know the drive is on it's way out. BTW, you do know WD Red Drives are supposed to be only used in RAID arrays?

Aside from that, I've never used ddrescue but it sounds like a good option. If there are errors, Spinrite6 may be able to correct them, but if SMART is throwing warnings, Spinrite will probably warn you that the drive has set the "Imminent Failure" flag, and that since it's running on borrowed time, use of spinrite may push it over the edge into total failure.

If spinrite is NOT able to recover all the data you will see a red "U" in the drive map in the log. This indicates that spinrite recovered all of the data that was recovered correctly, and used statistical analysis of over 1000 read retries to guess what the corrupt bits were, (which is usually only a string of several bytes.

If there are enough spare sectors to spare out all the failing ones, any cloning tool should be able to clone the drive, (but unrecoverable sectors may contain corrupted data.) and if the system files were not corrupted, you should be able to continue using the new drive.

 

[bexamous]

IMO most important thing is do not let disk spin down, keep power attached and turn off any power saving stuff that'll spin it down.

 

[phis6]

You can also try UFS explorer (not free) if you find the procedure of ddrescue to be a bit more complex for you.

 

[TGK]

Here is what I would do (and have done several times at work and otherwise):

1. Unplug the drive for now. The more you use it the lower the chance of recovery.
2. Purchase a 3TB or larger drive.
3. Use ddrescue (available on most linux livecd/usb media like sysrescuecd) to make a bit for bit copy of the bad drive to the new. ddrescue will copy as much of the bad drive to the new drive as possible.
4. Check the copy for errors.

I second this. You can also request western digital to cross ship you a replacement if the drive is under warranty and use that to copy to.

 

[Coldblackice]

Here is what I would do (and have done several times at work and otherwise):

1. Unplug the drive for now. The more you use it the lower the chance of recovery.
2. Purchase a 3TB or larger drive.
3. Use ddrescue (available on most linux livecd/usb media like sysrescuecd) to make a bit for bit copy of the bad drive to the new. ddrescue will copy as much of the bad drive to the new drive as possible.
4. Check the copy for errors.

use ddrescue, and use the log option. Run the first pass without retries. Then run a pass doing like 3 retries. Next power down the drive and let it cool off for awhile. Then try again. It will only attempt the parts it couldn't read correctly before.

Normally I found letting the drive cool helps it get more data. Normally I can recover all data, or only missing like 8k of data, using this method. Assuming the drive still spins.

Ah, ddrescue! I remember reading about a special dd variant for situations like this, but couldn't remember its name -- I thought it was a separate piece of software.

Awesome, thanks both for the suggestion.

If SMART is throwing errors, you know the drive is on it's way out. BTW, you do know WD Red Drives are supposed to be only used in RAID arrays?

I did -- I came across two of them on a special deal, but got mopped financially afterward, having to sell one of the twins. I read that, even though designed for RAIDs, it would be alright as a single drive -- is this not the case?

Aside from that, I've never used ddrescue but it sounds like a good option. If there are errors, Spinrite6 may be able to correct them, but if SMART is throwing warnings, Spinrite will probably warn you that the drive has set the "Imminent Failure" flag, and that since it's running on borrowed time, use of spinrite may push it over the edge into total failure.

If spinrite is NOT able to recover all the data you will see a red "U" in the drive map in the log. This indicates that spinrite recovered all of the data that was recovered correctly, and used statistical analysis of over 1000 read retries to guess what the corrupt bits were, (which is usually only a string of several bytes.

If there are enough spare sectors to spare out all the failing ones, any cloning tool should be able to clone the drive, (but unrecoverable sectors may contain corrupted data.) and if the system files were not corrupted, you should be able to continue using the new drive.

Awesome, thanks. I've heard that Spinrite is great, but only as a last-ditch this-drive-is-getting-flushed-down-the-toilet-right-after-this effort, as it'll ride it into the ground -- is that what you mean, as far as pushing it over the edge?

IMO most important thing is do not let disk spin down, keep power attached and turn off any power saving stuff that'll spin it down.

Do you mean when I re-plug the drive back in to attempt recovery? Or were you meaning as soon as you see SMART errors/problems popping, don't let the drive power down?

And when I do power it up again to attempt recovery (as soon as I get a replacement drive to dump to), does that mean it'd be best to try to grab the data all in one go, rather than getting it in spurts?

You can also try UFS explorer (not free) if you find the procedure of ddrescue to be a bit more complex for you.

Thanks for the suggestion.

I second this. You can also request western digital to cross ship you a replacement if the drive is under warranty and use that to copy to.

Great idea. It is still under warranty, thankfully, although I'm pretty paranoid now with using a WD Red drive again -- the Newegg reviews are atrocious. I'm really concerned this drive is suddenly dying after 4-5 months, too. It seems drives usually die either right at the start, or after lots of time. I've never had a drive die at a time like this. I really wish I knew what the problem is. I just don't get how an immobile, well-cooled drive suddenly starts melting-down.

 

[TGK]

Not sure if this will make you feel better but I have 8 WD red 3TB drives that are flawless. You may need to disable the smart parking feature though. One revision turned it back on to 8 seconds similar to the green drives. You will see a high load unload smart value if you got one of them. There are other threads here which describe how to disable it or increase the time.

Also, none of them are in a raid array, they are stand alone zfs pools with mhddfs to unify the directories.

 

[tdowning]

Awesome, thanks. I've heard that Spinrite is great, but only as a last-ditch this-drive-is-getting-flushed-down-the-toilet-right-after-this effort, as it'll ride it into the ground -- is that what you mean, as far as pushing it over the edge?

Another Wall of Text Warning:

There is a great deal of information about the development and workings of Spinrite on Steve Gibson's website: www.grc.com

Steve Gibson does a weekly podcast on twit.tv with Loe Laporte called "Security Now" Occasionally Spinrite and other issues of data reliability come up.

Among such topics was a study of hard drive reliability by Google. For many applications, Google uses standard consumer hard drives, and what they found is that you couldn't rely on a "Failure Imminent" flag to be set before a drive failed. However if the flag did get set, you knew for a fact it was time to replace that drive.

In general, Spinrite will disable anything that the HDD does in the background.
The rest is dependent on the operating level:
Level 0 (zero) will simply make a read-only pass of the entire drive. If the HDD encounters an error, and decides to relocate a sector, if the HDD can get one last good read of the sector, the relocation will proceed, otherwise nothing will happen.
Level 2, will make a read-only pass of the HDD, but if the HDD encounters an error, Spinrite will bring it's "Dynastat" recovery mode into action. Dynastat will make 1024 attempts to read the bad sector, and before every attempt, it will attempt to re-calibrate the head, swing the head to a random sector on the disk and then back to the sector under investigation. Steve describes this as "Bringing the head in at different velocities to the sketchy sector" and that this can cause the head to be closer or farther from the center of the track on any given pass. If at any time the HDD reports that it successfully read the sketchy sector, the Dynastat data is discarded, and the one "Good" read is used in the data relocation. If after 1024 retries there are no "Good" reads of the sector, Dynastat uses statistical analysis of what it did get on all 1024 reads to determine what the uncertain data is most likely to be, (which is usually only a section that is a few bytes long)
Level 4 will exercise every data sector of the HDD, by reading the data that is there, Inverting that data, writing it back to the HDD, reading back the inverted data, writing the original data back, and verifying the original data. If at any point an issue arises dynastat will kick in as above, if it's the first read phase, otherwise Spinrite already has the sector data in RAM and can use that for the relocation. this thorough workout of all sectors will make the SMART subsystem aware of any sketchy sectors, and allow them to be spared out before they become critical.

Many people have reported that regularly using Spinrite before issues arose has kept very old HDDs usable for a very long time.

TL: DR Summary

Spinrite is a great recovery tool. I would say, that if Spinrite can't let you get your data off a drive, your only choice is to send it to DriveSavers or the like. If the Imminent failure flag has been set on a drive, Spinrite will warn you to get what you can off the drive before using Spinrite. If the Imminent failure flag is set there is a chance that spinrite will push it "Over the edge" and completely kill the drive. On drives that have not set the "Imminent Failure" flag, the risk of it killing the drive is near zero.

I personally have no experience with the ddrescue program others have mentioned, and therefore cannot say anything about it. I can say that drives that I tried to clone, and the clone failed, (with Ghost or other programs) running spinrite would recover a sector or two, and relocate them, and then the clone would proceed with no issues.

Addendum: Sector relocation:
Logically, you can think of the data storage on your hard drive as a single piece of linear tape. When the factory lays out the sectors on the platters it leaves intentional multi-sector gaps. for the sake of argument, lets say that there are 100 data sectors followed by 5 spare sectors.
0-1-2-3-...-98-99-100-s1-s2-s3-s4-s5
If sector 5 becomes sketchy it relocates such that that 105 sector block looks like this:
0-1-2-3-4-BAD-5-6-7-...-98-99-100-s2-s3-s4-s5
In other words, sectors 5-100 get shifted farther down the tape by 1 sector.

If many sectors go bad in a short period of time, or the HDD is running out of spare sectors, that is what causes the imminent failure flag to be set.

Final thoughts:

Spinrite Is valuable as a last-ditch-save for your data, but you can also use it preemptively, which can prevent small HDD issues from becoming huge issues. HDD retry issues were easy to spot in windows 95/98 because the HDD light would be on solid, and even the mouse cursor would stop moving, since it had to kick back to DOS for the HDD operations. it is much harder to identify this situation on windows XP and forward, which makes another case for using it preemptively.