Constant RAID errors on different drives....already replaced Mobo, RAM, and CPU!

T

ThePerformer

Limp Gawd
Joined
Jun 9, 2003
Messages
284
This is getting rediculous. I have been getting strange errors with my system for MONTHS now. I've solved all of them except one: the constant RAID errors which already cost me 500GB of data that I'll never see again.

When the other strange errors started, they were accompanies by RAID errors on my RAID 5 array of 4 Seagate 500GB drives. Several times a week, one of the drives would get an error on it, I would have to restart my computer and rebuild the drive. The exact drive which would fail was random. It was not a single drive failing. So then I reinstalled Vista. Still constant failures. Then I replace the motherboard and CPU and reinstall Vista. Still constant failures. I change the array to RAID 0+1 thinking that maybe the RAID 5 is just crappy. Still errors. THEN, I replace the RAM which finally seemed to calm it down. The other odd errors are gone and the hard drives were stable for two weeks straight.

In the last 5 days, I have had two different drives in the RAID 0+1 array fail (thankfully not at the same time), and just now, a drive in my RAID 1 system array (WD Raptors) failed as well! It's rebuilding right now.

WTF!!?!? This can't be hardware since it's all been replaced and this can't be software since it's all been reinstalled. It can't possibly be that I have 4+ bad drives.....HDs rarely fail as it is anyways and these are some pretty sweet drives.

So what the heck!?!? What could this possibly be?? I have a Lian Li case that is supposed to have massive cooling capabilities and the drives are not even warm to the touch. They even all use different cabling!!

I really need everyone's help. I HAVE to figure out what's going on because otherwise this extremely expensive system is basically useless. I've already replaced everything that could cause errors. Please help me guys. Thank you!

PS: MB BIOS is already up to date, nothing specific sets it off. Will add more thoughts as I get them.
 
Have you run a low level diagnostic on all the drives yet? RAM test?

Unless the above is an issue or the controller on the mobo is flakey then it sounds like you shouldn't run RAID. :(
 
The RAM has been entirely replaced with new modules. I have not run a low level diagnostic on the disks. How would I go about doing that?

Another drive just failed. That's two different drives in an hour. And I just updated the BIOS to the latest version which was released just a few days ago. Ridiculous.
 
I just ran the Seagate diagnostic software. All 4 drives passed without issues. Furthermore, the highest temperature was 32deg which is far below any warning levels.

This is insane. Please, any thoughts or help would be VERY useful. Thanks!
 
Well the only time anything improved was when you replaced the RAM.

Double check the memory timings with a fine toothed comb. I know you just updated to the fresh Mobo BIOS, but there still might be an issue with the Mobo reading incorrect SPD readings off the RAM.

Might want to try swapping SATA cables, who knows man ... The Shuttle SN25P came with buggy, poor quality SATA cables. Luckily this issue was nailed down before I picked one up. It took months before someone pointed the finger at the faulty cables.
 
Huh. I'll try changing the cables, but each drive uses a cable from a different source, so it's unlikely that that is causing the issue.
 
I don't know, but it kind of sounds like your RAID controller or software is erroneously thinking a drive is faulty and kicking it from the array. Unless I have been misinformed, this can happen when a drive is taking too long to perform error recovery (such as normal reallocation of a bad sector); the drive doesn't respond quickly enough, and the RAID controller thinks the drive has died. I think I had the same problem with some miscellaneous desktop drives under Linux software RAID-5 about a year ago; the problem went away on its own, though, before I figured out what what was (probably) happening.

Run a SMART diagnostic tool on your drives and look at the Reallocated_Sector_Ct and Reallocated_Event_Count. For example, this is probably the one that gave me trouble earlier. 41 reallocated sectors is annoying, but the VALUE of 96 (really a "cooked" score) is still well above the manufacturer's pre-fail threshold of 10. If the VALUE started dropping quickly, then I'd think about replacing the drive.
Code: 
smartctl -A /dev/sdb | egrep 'ID|Reallocated'
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   096   096   010    Pre-fail  Always       -       41
196 Reallocated_Event_Count 0x0032   096   096   000    Old_age   Always       -       41
Anyway, take notes and see if the Reallocated_Event_Count increases whenever a particular drive gets kicked.

Some drives are designed to let a RAID controller handle the error recovery (from the other drives) and reallocate in the background if needed, so as to be able to respond quickly enough that the RAID controller doesn't think the drive has died.
http://www.techworld.com/storage/features/index.cfm?featureid=1019
http://www.google.com/search?q="time+limited+error+recovery"

It seems vaguely possible to me that your drives had a scattering of already-bad sectors when you received them, and every once in a while you end up trying to write to one and the drive gets kicked. You can try using a program like badblocks to run a read-write test on your drives. This ought to cause the drive to find and reallocate all currently bad sectors once and for all. I don't know if that'll work, but it's worth a try.


Also, since it hasn't yet been specifically suggested in this thread, run memtest86+ on your RAM.

-Corey
 
I ran memtest on the RAM before it was replaced and it passed with flying colors. After I replaced the RAM, my system stopped BSODing, my GPU stopped crashing every 20 minutes, and my hard drives were fine for 2 weeks. I don't trust MemTest considering that it told me my RAM was fine when it clearly wasn't.

I'll look into the sectors though......thanks!!!
 
I would suspect the PSU as the likely candidate. Perhaps it is failing and cannot supply enough power and a drive is blipping from the array.
 
Hmmmm.....interesting idea. It's an 850W power supply though. Should be able to handle even more than I'm throwing at it now. Not sure how I could test that theory.

Wouldn't the processor and GPU crash before a random single hard drive though? I feel like I'd see tons of other issues if it was the PSU.
 
ThePerformer said:
Hmmmm.....interesting idea. It's an 850W power supply though. Should be able to handle even more than I'm throwing at it now. Not sure how I could test that theory.

Wouldn't the processor and GPU crash before a random single hard drive though? I feel like I'd see tons of other issues if it was the PSU.
Click to expand...

Nope. I had a PSU that couldn't handle the HDDs I had on it. It would be fine for building the array, but as soon as I started to load files onto it, a few drive would hiccup and the array would fail. Luckily the highpoint controller was smart enough to disable access and the array came right back as soon as I rebooted.
 
Dew said:
Nope. I had a PSU that couldn't handle the HDDs I had on it. It would be fine for building the array, but as soon as I started to load files onto it, a few drive would hiccup and the array would fail. Luckily the highpoint controller was smart enough to disable access and the array came right back as soon as I rebooted.
Click to expand...

Huh.......if that's possible, that could be causing this. It does seem to happen a little more regularly when I'm converting videos for example. I can unplug one of my 8800 GTXs to give it some more power and see if that helps.
 
I hate to say it, but I wonder if it's the case itself. I have had plenty "ghost in the machine" type issues solved because I switched out the case---whether there was a short, or some other weirdness, it tends to resolve a lot of vague issues.
 
Well, I have more bad news:

1) I took out one of my two 8800 GTXs to reduce the load on the CPU. A disk crashed within 15 minutes of starting up my system. So no go there.
2) I replaced all of the cables with different cables. Disk went bad in about 30 minutes. No change there.

So neither reducing the power nor replacing the cables affected it. As I have stated above, the MB, CPU, and RAM have already all been changed and the physical drives have passed the seagate diagnostics.

I can't possibly see how a fault with the case could somehow force random RAID errors. It just doesn't make sense.

Lastly, I have found a way to pretty much reliably force a disk to fail! When I run SUPER (video encoding software), it invariably kicks a disk offline within about 30 minutes. Otherwise, the computer will sit for hours on end without a disk having any problems. I've also used Lightroom and data intense applications but SUPER is the only one that reliably causes problems. What about it could cause RAID errors? Thanks!
 
Is this an overclocking issue (Saturn rig in sig?) ? Can you dial it back to stock settings and see if any change?
 
[LYL]Homer;1031976717 said:
Is this an overclocking issue (Saturn rig in sig?) ? Can you dial it back to stock settings and see if any change?
Click to expand...

It has been 100% stock speeds and voltages since the errors started several months ago. The new mobo, RAM, and CPU have never seen a single cycle above their stock speeds.
 
You may have found your issue. Best way to test though would be to get a PCI or PCI-E raid card and give it a go. Unfortunately you'll end up with a format and re-install.
 
tommithy said:
You may have found your issue. Best way to test though would be to get a PCI or PCI-E raid card and give it a go. Unfortunately you'll end up with a format and re-install.
Click to expand...

Well, I can leave the RAID 1 array on the MB and put the RAID 5 array onto the controller card. That's what I'm going to do. Gonna get a RocketRAID since it's Vista compatible.
 
I've had two RR2320's in service for over a year running 2TB R5 arrays each with no issues.
 
Beaten to it.

Yes, it's the nVidia RAID. Data loss is a fact, not an 'if.' If you're running nVRAID, get off it or don't come crying here. It's buggy half-software garbage and I suspect there's some significant errors in their parity generation on the silicon. e.g. no software fix.

Areca and LSI are the best you can get period.
 
You must log in or register to reply here.
Back
Top