Stress test new HDDs?

[Snowdog]

A clean SMART report doesn't mean the drive won't fail, I think people understand that. Certain SMART indicators are, however, correlated with drive failure probability. The point of running badblocks isn't the badblocks report itself its seeing if it trips any SMART indicators.

Someone previously suggested I just run one-pass badblocks to save downtime. How would that be significantly better than simply doing a Full format for potentially finding bad sectors and getting SMART indications of it?

 

[drescherjm]

Someone previously suggested I just run one-pass badblocks to save downtime. How would that be significantly better than simply doing a Full format for potentially finding bad sectors and getting SMART indications of it?

The 1 pass badblocks a little better since it tests the drives retention over a some time period where the full format does not. These days I have seen quite a few drives that can not read back the sectors they wrote a few hours ago (drive amnesia). I assume this is caused by a head problem.

 

[drescherjm]

On large drives I always test before I install them even into a raid array.

I do at minimum 4 pass badblocks on every single drive I receive (new or RMA). However for raid arrays I try to test a little more rigorous if possible. If I have a test server available after the badblocks has completed on every drive I put the drives into the test server create a raid6 array, put btrfs on the raid array and fill the filesystem with data then do several btrfs scrubs and raid array consistency checks and maybe some other md5deep testing. This is similar to what I do when I get a new server. I usually stress test it for 2 weeks or so between the memory testing / raid array testing before any valuable data hits the machine even though I know I have backups I try to be sure I will only rarely need to restore.

 

[drescherjm]

The problem with my recently failed Seagate. All of those were ZERO. I couldn't detect anything amiss with a third party SMART util.

Yet windows was complaining every 15 minutes that I had driver errors, and I needed to backup the drive ASAP. Seagate Tools failed it's own SMART test (which doesn't show any actual values).

But still, nothing on third party SMART utils.

I can say that I have never seen that. However I have only had a few hundred drives. I would be interesting to look at the raw SMART data if you have that just to see if any other attribute looks unusual. Although with Seagate you may not be able to tell since some of their attributes are not real values.

 

[devman]

I can say that I have never seen that. However I have only had a few hundred drives. I would be interesting to look at the raw SMART data if you have that just to see if any other attribute looks unusual. Although with Seagate you may not be able to tell since some of their attributes are not real values.

Its not all that uncommon. When Google published the results of their massive harddrive reliability study one of the points made in the conclusion was that a large fraction of the failed drives failed without any anomalies in there SMART data.

Despite those strong correlations, we find that
failure prediction models based on SMART parameters
alone are likely to be severely limited in their prediction
accuracy, given that a large fraction of our failed drives
have shown no SMART error signals whatsoever. This
result suggests that SMART models are more useful in
predicting trends for large aggregate populations than for
individual components. It also suggests that powerful
predictive models need to make use of signals beyond
those provided by SMART.

http://static.googleusercontent.com...ch.google.com/en/us/archive/disk_failures.pdf

Basically a clean SMART report can't be reliably used an indicator that a drive will not fail, but a bad SMART report can be correlated strongly with drive failure.

 

[Snowdog]

I can say that I have never seen that. However I have only had a few hundred drives. I would be interesting to look at the raw SMART data if you have that just to see if any other attribute looks unusual. Although with Seagate you may not be able to tell since some of their attributes are not real values.

Here is the snapshot I screen grabbed, before removing the drive for return:
http://img546.imageshack.us/img546/9969/sea3tbcrystalsmart.png

Read Error Rate, Seek Error Rate look like nonsense values. They looked like that from the moment it was installed.

The above is from a drive Windows was complaining about every 15 minutes until I removed it and it failed Seagate tools "SMART" test and Diagnostic Short Test.

 

[drescherjm]

http://static.googleusercontent.com...ch.google.com/en/us/archive/disk_failures.pdf

Basically a clean SMART report can't be reliably used an indicator that a drive will not fail, but a bad SMART report can be correlated strongly with drive failure.

I have read and quoted that report several times here mostly about annual failure rates. Anyways I do understand and admit that my sample size is way too small to be statistically significant so I may have just been lucky that I have been able to predict failure extremely reliably on the servers / drives that I have monitored for years. In addition to monitoring these parameters in the SMART I do weekly raid checks / scrubs. That is what triggered the drive last week to be kicked out of the array and at a very bad time since at that I was on my way to the airport to fly to Los Cabos Mexico when it was kicked out.

 

[drescherjm]

Here is the snapshot I screen grabbed, before removing the drive for return:
http://img546.imageshack.us/img546/9969/sea3tbcrystalsmart.png

Read Error Rate, Seek Error Rate look like nonsense values. They looked like that from the moment it was installed.

The above is from a drive Windows was complaining about every 15 minutes until I removed it and it failed Seagate tools "SMART" test and Diagnostic Short Test.

Interesting thanks.

 

[JoeComp]

I have read and quoted that report several times here mostly about annual failure rates. Anyways I do understand and admit that my sample size is way too small to be statistically significant so I may have just been lucky that I have been able to predict failure extremely reliably on the servers / drives that I have monitored for years. In addition to monitoring these parameters in the SMART I do weekly raid checks / scrubs. That is what triggered the drive last week to be kicked out of the array...

In my experience, a large number of failures and pre-failures can be detected by simply writing to and reading every sector of the drive. That is why the badblocks / SMART attributes method is effective. That is also why your scrubs are effective (of course, they would be more effective if they wrote every sector, but that is usually not practical once the HDD is in production).

I also suspect the lack of reading and writing to every sector is why google reported that some drives failed without any SMART attributes showing problems. Likely most of those types of failures were for HDDs that did not frequently get all sectors written and read.

 

[SemiLucid]

Hi so what do you guys think is best practice in this regard, considering there is a question over SMARTs reliability... See Google's study. Perhaps we need to supplement SMART along with other tools.

Hmm. I remember writing about that yesterday but today I do not see it in this thread. Time to search..

Edit: Anyways here are the attributes that I am concerned at most:
"Reallocated_Sector_Ct" "Current_Pending_Sector" "Offline_Uncorrectable" "UDMA_CRC_Error_Count" "Hardware_ECC_Recovered"

Assuming we can place some faith in reading SMART attributes, what thresholds would indicate a bad drive for each of the above attributes? Do the HDD manufacturers give any indication... I have never seen any listed anywhere.

 

[Paradoxine]

What should I use to test a HDD on windows? I don't mind how long it takes but I want a thorough, trusted test.

 

[anaxagorasbc]

does solaris 11.1 have a tool similar to badblocks? Or what about the manufacturers tools? like seatools?

 

[drescherjm]

Or what about the manufacturers tools? like seatools?

These are basically a GUI frontend to the SMART long and short tests that are builtin to the drives firmware + SMART display program that may or may not hide important values.

 

[UhClem]

Hey, thanks for even trying to help out all us dipshits. Shame we're too stupid to understand. I suppose you could educate, but then again, you stated it wasn't your role to educate. I guess that's why you haven't responded to requests for alternatives.

Sorry to "disappoint" you ... I was traveling.

As for the educate thing ... rather than just provide someone with a solution, and deprive them of the opportunity/satisfaction of figuring it out themselves, I'd rather stimulate their thought process by giving them a key "starting point", and let them ride their own wave. (I always preferred things that way, as the recipient.)

Case in point:
I tried, several times, to "lead" JoeComp & drescherjm away from their prescribed usage of badblocks and SMART reports, by pointing out that they are destroying evidence. Their suggestion is to perform:

smartctl -a /dev/sdX > Smart_Before_X.txt
badblocks -wv /dev/sdX
smartctl -a /dev/sdX > Smart_After_X.txt

Better (and what I was hoping they would "see") is:

smartctl -a /dev/sdX > Smart_Before_X.txt
for i in 0xaa 0x55 0xff 0x00
do
badblocks -wv -t $i /dev/sdX
smartctl -a /dev/sdX > Smart_After_X_$i.txt
done

Hence, the exact same drive exercising takes place, in the same elapsed time, but you now have 4 times as much information regarding the "dynamic medical history" of the tested drive. And you have not lost visibility of flaky sectors that pop up during the read phase of a pass (and bump Current_Pending_Sector) but then get "cleared up" during the write phase of the following pass (decrementing C_P_S) [of course, hard-case bad sectors get Reallocated].

But, even these additional SMART reports are not sufficient, because SMART's error logging is very limited, and gets wasted by duplicated log-entries due to driver retries; result is that you have little/no information on LBA#s that are flaky (crucial in "seeing" patterns). To address this, a comprehensive stress test report would want to integrate details of each drive error reported to the kernel (by scanning the relevant portion of var/log/syslog*).

Please note that I have corrected (and mostly retracted) my criticism of badblocks in my initial post [link]. Summary: drive reports UNCorrectable error to kernel driver, but kernel only reports that error to the calling read() (from badblocks) IF the error persists through MAX_RETRIES+1 (6) attempts. Some drive errors are transient, initially, but may very well progress to a more persistent/serious condition. (Feel that lump?/Bend over and cough. )

By getting these early warnings, and assessing their potential (negative) impact, one can minimize faults (and loss of data) during eventual (array) rebuilds. For instance, a mildly flaky drive could be rotated out of front-line service and used instead as a backup spare [cf: Injured Reserve on a sport team].

--UhClem

 

[drescherjm]

I do look at the SMART every once and a while during the running of badblocks. I do not record it cleanly as your suggestion but my scrollback buffer on putty is > 20 thousand lines.

 

[TheDingy]

I do look at the SMART every once and a while during the running of badblocks. I do not record it cleanly as your suggestion but my scrollback buffer on putty is > 20 thousand lines.

Write it out to a file so that you can check it with grep are awk... Much better than just scrolling back..