Adaptec 5805 RAID 6 Rebuilding

N

nry

Limp Gawd
Joined
Jul 10, 2008
Messages
409
Had my RAID 6 array for over a year now with no issues. Has been powered up and down as required (system always had power though).
Today started the machine up and Ubuntu wouldn't start!

I have 8x Hitachi 5K3000 3TB drives in RAID6, the following is from the command line config...

So device 4 is rebuilding here (although no HDD led's are flashing which they usually do?)
Is there anyway to find out how long this will take? Or get some sort of status

Really not a fan of the Adaptec software, my Areca card is much more user friendly!!

Code: 
Controllers found: 1
----------------------------------------------------------------------
Controller information
----------------------------------------------------------------------
   Controller Status                        : Optimal
   Channel description                      : SAS/SATA
   Controller Model                         : Adaptec 5805
   Controller Serial Number                 : 8B26106A941
   Temperature                              : 68 C/ 154 F (Normal)
   Installed memory                         : 512 MB
   Copyback                                 : Disabled
   Background consistency check             : Enabled
   Automatic Failover                       : Enabled
   Global task priority                     : Medium
   Performance Mode                         : Default/Dynamic
   Stayawake period                         : Disabled
   Spinup limit internal drives             : 3
   Spinup limit external drives             : 0
   Defunct disk drive count                 : 0
   Logical devices/Failed/Degraded          : 1/0/1
   SSDs assigned to MaxCache pool           : 0
   Maximum SSDs allowed in MaxCache pool    : 8
   MaxCache Read Cache Pool Size            : 0.000 GB
   MaxCache flush and fetch rate            : 0
   MaxCache Read, Write Balance Factor      : 3,1
   NCQ status                               : Enabled
   Statistics data collection mode          : Enabled
   --------------------------------------------------------
   Controller Version Information
   --------------------------------------------------------
   BIOS                                     : 5.2-0 (18937)
   Firmware                                 : 5.2-0 (18937)
   Driver                                   : 1.1-5 (2461)
   Boot Flash                               : 5.2-0 (18937)
   --------------------------------------------------------
   Controller Battery Information
   --------------------------------------------------------
   Status                                   : Charging
   Over temperature                         : No
   Capacity remaining                       : 100 percent
   Time remaining (at current draw)         : 2 days, 1 hours, 45 minutes

----------------------------------------------------------------------
Logical device information
----------------------------------------------------------------------
Logical device number 0
   Logical device name                      : Store1
   RAID level                               : 6 Reed-Solomon
   Status of logical device                 : Suboptimal, Fault Tolerant
   Size                                     : 17141750 MB
   Stripe-unit size                         : 256 KB
   Read-cache mode                          : Enabled
   MaxCache preferred read cache setting    : Enabled
   MaxCache read cache setting              : Disabled
   Write-cache mode                         : Enabled (write-back)
   Write-cache setting                      : Enabled (write-back) when protected by battery/ZMM
   Partitioned                              : Yes
   Protected by Hot-Spare                   : No
   Bootable                                 : Yes
   Failed stripes                           : No
   Power settings                           : Enabled
   Slow down after(Minutes)                 : 1h
   Power off after(Minutes)                 : 3h
   Verify after(Hours)                      : 24h
   Power State                              : Active
   --------------------------------------------------------
   Logical device segment information
   --------------------------------------------------------
   Segment 0                                : Present (Controller:1,Connector:0,Device:0)       MJ1311YNG3WWMA
   Segment 1                                : Present (Controller:1,Connector:0,Device:1)       MJ1311YNG3RJVA
   Segment 2                                : Present (Controller:1,Connector:0,Device:2)       MJ1321YNG0BHSA
   Segment 3                                : Present (Controller:1,Connector:0,Device:3)       MJ1321YNG0A4BA
   Segment 4                                : Present (Controller:1,Connector:1,Device:3)       MJ1311YNG2U06A
   Segment 5                                : Present (Controller:1,Connector:1,Device:2)       MJ1311YNG3YZSA
   Segment 6                                : Present (Controller:1,Connector:1,Device:1)       MJ1311YNG3XULA
   Segment 7                                : Rebuilding (Controller:1,Connector:1,Device:0)       MJ1321YNG0TYWA


----------------------------------------------------------------------
Physical Device information
----------------------------------------------------------------------
      Device #0
         Device is a Hard drive
         State                              : Online
         Supported                          : Yes
         Transfer Speed                     : SATA 3.0 Gb/s
         Reported Channel,Device(T:L)       : 0,0(0:0)
         Reported Location                  : Connector 0, Device 0
         Vendor                             : Hitachi
         Model                              : HDS5C3030ALA630
         Firmware                           : MEAOA5C0
         Serial number                      : MJ1311YNG3WWMA
         Size                               : 2861588 MB
         Write Cache                        : Enabled (write-back)
         FRU                                : None
         S.M.A.R.T.                         : No
         S.M.A.R.T. warnings                : 0
         Power State                        : Full rpm
         Supported Power States             : Full rpm,Powered off,Reduced rpm
         SSD                                : No
         MaxCache Capable                   : No
         MaxCache Assigned                  : No
         NCQ status                         : Enabled
      Device #1
         Device is a Hard drive
         State                              : Online
         Supported                          : Yes
         Transfer Speed                     : SATA 3.0 Gb/s
         Reported Channel,Device(T:L)       : 0,1(1:0)
         Reported Location                  : Connector 0, Device 1
         Vendor                             : Hitachi
         Model                              : HDS5C3030ALA630
         Firmware                           : MEAOA5C0
         Serial number                      : MJ1311YNG3RJVA
         Size                               : 2861588 MB
         Write Cache                        : Enabled (write-back)
         FRU                                : None
         S.M.A.R.T.                         : No
         S.M.A.R.T. warnings                : 0
         Power State                        : Full rpm
         Supported Power States             : Full rpm,Powered off,Reduced rpm
         SSD                                : No
         MaxCache Capable                   : No
         MaxCache Assigned                  : No
         NCQ status                         : Enabled
      Device #2
         Device is a Hard drive
         State                              : Online
         Supported                          : Yes
         Transfer Speed                     : SATA 3.0 Gb/s
         Reported Channel,Device(T:L)       : 0,2(2:0)
         Reported Location                  : Connector 0, Device 2
         Vendor                             : Hitachi
         Model                              : HDS5C3030ALA630
         Firmware                           : MEAOA580
         Serial number                      : MJ1321YNG0BHSA
         Size                               : 2861588 MB
         Write Cache                        : Enabled (write-back)
         FRU                                : None
         S.M.A.R.T.                         : No
         S.M.A.R.T. warnings                : 0
         Power State                        : Full rpm
         Supported Power States             : Full rpm,Powered off,Reduced rpm
         SSD                                : No
         MaxCache Capable                   : No
         MaxCache Assigned                  : No
         NCQ status                         : Enabled
      Device #3
         Device is a Hard drive
         State                              : Online
         Supported                          : Yes
         Transfer Speed                     : SATA 3.0 Gb/s
         Reported Channel,Device(T:L)       : 0,3(3:0)
         Reported Location                  : Connector 0, Device 3
         Vendor                             : Hitachi
         Model                              : HDS5C3030ALA630
         Firmware                           : MEAOA580
         Serial number                      : MJ1321YNG0A4BA
         Size                               : 2861588 MB
         Write Cache                        : Enabled (write-back)
         FRU                                : None
         S.M.A.R.T.                         : No
         S.M.A.R.T. warnings                : 0
         Power State                        : Full rpm
         Supported Power States             : Full rpm,Powered off,Reduced rpm
         SSD                                : No
         MaxCache Capable                   : No
         MaxCache Assigned                  : No
         NCQ status                         : Enabled
      Device #4
         Device is a Hard drive
         State                              : Rebuilding
         Supported                          : Yes
         Transfer Speed                     : SATA 3.0 Gb/s
         Reported Channel,Device(T:L)       : 0,4(4:0)
         Reported Location                  : Connector 1, Device 0
         Vendor                             : Hitachi
         Model                              : HDS5C3030ALA630
         Firmware                           : MEAOA580
         Serial number                      : MJ1321YNG0TYWA
         Size                               : 2861588 MB
         Write Cache                        : Enabled (write-back)
         FRU                                : None
         S.M.A.R.T.                         : No
         S.M.A.R.T. warnings                : 0
         Power State                        : Full rpm
         Supported Power States             : Full rpm,Powered off,Reduced rpm
         SSD                                : No
         MaxCache Capable                   : No
         MaxCache Assigned                  : No
         NCQ status                         : Enabled
      Device #5
         Device is a Hard drive
         State                              : Online
         Supported                          : Yes
         Transfer Speed                     : SATA 3.0 Gb/s
         Reported Channel,Device(T:L)       : 0,5(5:0)
         Reported Location                  : Connector 1, Device 1
         Vendor                             : Hitachi
         Model                              : HDS5C3030ALA630
         Firmware                           : MEAOA5C0
         Serial number                      : MJ1311YNG3XULA
         Size                               : 2861588 MB
         Write Cache                        : Enabled (write-back)
         FRU                                : None
         S.M.A.R.T.                         : No
         S.M.A.R.T. warnings                : 0
         Power State                        : Full rpm
         Supported Power States             : Full rpm,Powered off,Reduced rpm
         SSD                                : No
         MaxCache Capable                   : No
         MaxCache Assigned                  : No
         NCQ status                         : Enabled
      Device #6
         Device is a Hard drive
         State                              : Online
         Supported                          : Yes
         Transfer Speed                     : SATA 3.0 Gb/s
         Reported Channel,Device(T:L)       : 0,6(6:0)
         Reported Location                  : Connector 1, Device 2
         Vendor                             : Hitachi
         Model                              : HDS5C3030ALA630
         Firmware                           : MEAOA5C0
         Serial number                      : MJ1311YNG3YZSA
         Size                               : 2861588 MB
         Write Cache                        : Enabled (write-back)
         FRU                                : None
         S.M.A.R.T.                         : No
         S.M.A.R.T. warnings                : 0
         Power State                        : Full rpm
         Supported Power States             : Full rpm,Powered off,Reduced rpm
         SSD                                : No
         MaxCache Capable                   : No
         MaxCache Assigned                  : No
         NCQ status                         : Enabled
      Device #7
         Device is a Hard drive
         State                              : Online
         Supported                          : Yes
         Transfer Speed                     : SATA 3.0 Gb/s
         Reported Channel,Device(T:L)       : 0,7(7:0)
         Reported Location                  : Connector 1, Device 3
         Vendor                             : Hitachi
         Model                              : HDS5C3030ALA630
         Firmware                           : MEAOA5C0
         Serial number                      : MJ1311YNG2U06A
         Size                               : 2861588 MB
         Write Cache                        : Enabled (write-back)
         FRU                                : None
         S.M.A.R.T.                         : No
         S.M.A.R.T. warnings                : 0
         Power State                        : Full rpm
         Supported Power States             : Full rpm,Powered off,Reduced rpm
         SSD                                : No
         MaxCache Capable                   : No
         MaxCache Assigned                  : No
         NCQ status                         : Enabled


Command completed successfully.
 
Another question:
Is it possible to use the array while its rebuilding to pull data off?

I had planned on moving most of this data to my raid array on my new Areca card...
 
It should be possible to pull data off. RAID6 is supposed to be able to tolerate two disk failures while still making your data available.

I suspect it will take a long time, a couple days maybe. I hope you have backups.
 
This is what I thought, but when ubuntu starts it says waiting for device '/dev/sdb1'...

90% is backed up! just a PITA to transfer 10TB from the remote location!

EDIT:

this is all I see in Adaptec Storage Manager, no percentage, no options to really do anything. Am I missing something here?

ScreenShot2013-02-05at171703_zpsf42c8849.png
 
Last edited:
nry said:
I had planned on moving most of this data to my raid array on my new Areca card...
Click to expand...

The sooner, the better. I'm guessing the rebuild froze because the disk it tried to rebuild is actually bad and its not handling the issue gracefully; one of the many reasons I dumped all my 5-series Adaptecs (2 x 52445, 1 x 5805) years ago and standardized on Areca.

Were this to happen on an Areca where a bad drive is in an unusual enough state to hold up the process without just being failed by the card (can happen in rare instances) then there's a menu option to manually fail the drive (credit to this forum's own houkouonchi for getting Areca to implement that feature, since he was dealing with a lot of remote systems where certain types of drive failures could cause problems for the controller - like the mechanical part has developed defects but the drive controller and SMART continue to insist everything is fine).

If it was me, I'd find a way to physically remove that Device0 on CN1, make sure the data on the array is backed up and then connect failed drive to plain motherboard SATA, look at smart stats, run full read/write surface scan and determine if it needs to be RMA'd or was just some sort of false positive failure. I'd also make sure heat wasn't an issue - which lead to both the drive failure and possibly the RAID controller seizing up.
 
Last edited:
Its currently rebuilding based on 7 of 8 disks in the boot menu.

Took about 3 hours to get to 30% so must be doing something, hopefully I can read the data to pull my 10% of un backed up data off!

Also hoping that I don't have another dead 3TB drive, had 2 dead Hitachi's in January, 1 dead WD and 1 dead seagate! Not a good month for me haha
 
nry said:
Its currently rebuilding based on 7 of 8 disks in the boot menu.

Took about 3 hours to get to 30% so must be doing something, hopefully I can read the data to pull my 10% of un backed up data off!

Also hoping that I don't have another dead 3TB drive, had 2 dead Hitachi's in January, 1 dead WD and 1 dead seagate! Not a good month for me haha
Click to expand...

I would question the temps and/or lack of airflow those failed Hitachi's are being subjected to, because Hitachi failures under normal operating conditions are rare to nonexistent in my experience w/ hundreds of these
 
Will investigate tomorrow
The ones in this system are cooled by 3x 120mm fans so should be ok

Now you mention this I am a little concerned about my server which is on 24/7 as I'm not too sure how well this is cooled! Can't seem to get the SMART data off those drives at the moment (on a 3ware 9690SA card)
 
you might try HDsentinel which supports seeing SMART with some raid controllers.

Smartmontools also is able to look behind a lot of raid controllers.
 
Well I left the controller rebuilding last night and this morning it was back to optimal and all is running ok now.

As for the heat I can't read the SMART value for the drives, but putting my hand on top of the case you can only feel some slight heat. Not the most effective way of measuring but feel pretty confident they are running cool.

Not so sure about my other server through!

Thanks for all the input :) Panic over for now
 
the chips on the circuit board on the bottom of the drives might not be getting enough air flow.
 
On the newer drives I have (WD30EFRX, ST3000DM001) all PCB mounted components are facing towards the drive case with the large ICs using the case as a heatsink. They would receive almost no direct airflow in most rackmount cases. Can't speak for Hitachi or Toshiba, though.
 
omniscence said:
On the newer drives I have (WD30EFRX, ST3000DM001) all PCB mounted components are facing towards the drive case with the large ICs using the case as a heatsink. They would receive almost no direct airflow in most rackmount cases. Can't speak for Hitachi or Toshiba, though.
Click to expand...

Those chips have heating transfer pad to the HD body. You need good airflow to cool HD body.
 
nry said:
Its currently rebuilding based on 7 of 8 disks in the boot menu.

Took about 3 hours to get to 30% so must be doing something, hopefully I can read the data to pull my 10% of un backed up data off!

Also hoping that I don't have another dead 3TB drive, had 2 dead Hitachi's in January, 1 dead WD and 1 dead seagate! Not a good month for me haha
Click to expand...

Are these verified dead drives or just kicked out of an array?
 
Hello from Adaptec by PMC!

I'm sorry to hear you're having an issue with your 5805 controller. The array should be available while it is suboptimal and rebuilding. There should also be a rebuild status displayed if you hover over the array that is in a rebuild state. I'd like to review your full logs and determine if there was indeed a drive anomaly. Please use our online support tool at http://ask.adaptec.com to create a new case and include your support.zip file. Also please indicate the specific version of Ubuntu you are using. Please mark the subject of the case Attn Liz so even if another tech pulls it I can still have a look.

Thanks and best regards,
Adaptec by PMC Technical Support
 
drescherjm said:
Are these verified dead drives or just kicked out of an array?
Click to expand...

All dead, either completely unusable/ really slow to use/ making funny noises.

Adaptec by PMC said:
Hello from Adaptec by PMC!

I'm sorry to hear you're having an issue with your 5805 controller. The array should be available while it is suboptimal and rebuilding. There should also be a rebuild status displayed if you hover over the array that is in a rebuild state. I'd like to review your full logs and determine if there was indeed a drive anomaly. Please use our online support tool at http://ask.adaptec.com to create a new case and include your support.zip file. Also please indicate the specific version of Ubuntu you are using. Please mark the subject of the case Attn Liz so even if another tech pulls it I can still have a look.

Thanks and best regards,
Adaptec by PMC Technical Support
Click to expand...

If I get chance I will upload this. Array is all back online now though with all the original disks!
 
cantalup said:
Those chips have heating transfer pad to the HD body. You need good airflow to cool HD body.
Click to expand...

That is exactly what I implied. The chips itself do not require direct airflow. If the temperature is high enough to damage the chips it is already significantly to high for the rest of the drive.
 
omniscence said:
That is exactly what I implied. The chips itself do not require direct airflow. If the temperature is high enough to damage the chips it is already significantly to high for the rest of the drive.
Click to expand...

the heat does not damage the chips, but will mess-up the internal HD mechanical, such as, bad sectors, stuck heads, and others.

before damage the chips, the overheat will damage the internal HD.
 
cantalup said:
the heat does not damage the chips, but will mess-up the internal HD mechanical, such as, bad sectors, stuck heads, and others.

before damage the chips, the overheat will damage the internal HD.
Click to expand...

I believe heat causes problems with the heads.

With this high percentage of bad disks from 3 manufacturers I would say the problem was either poor shipping, problems caused from excessive heat and/or vibration, a bad power supply or really bad luck..
 
Last edited:
drescherjm said:
I believe heat causes problems with the heads.

With this high percentage of bad disks from 3 manufacturers I would say the problem was either poor shipping, problems caused from excessive heat and/or vibration, a bad power supply or really bad luck..
Click to expand...

mostly damaged the heads, or stuck head or or bumping to the plates

HD is very fragile :D
 
You must log in or register to reply here.
Back
Top