Any reason a drive should just stop or turn off? FreeBSD/ZFS

I

iroc409

[H]ard|Gawd
Joined
Jun 17, 2006
Messages
1,385
While I was away for work, I got a SMART email warning saying it could not open one of the drives in my ZFS array on my file server. I have a daily email set up for disk status, and the drive was just gone. It wasn't in the array, or in /dev, and was missing from the daily run report with SMART status.

I ordered a replacement, replaced the "missing" drive, resilvered, and everything looks fine. I popped the "failed" drive in my eSATA dock on my desktop, and the drive seems to work just fine. Should there be anything in particular I should be looking at on the server itself to get an indication of why the drive disappeared? Last night my Seagate external USB drive also dropped off, but the power supply for it is squealing so I think maybe it just lost power... but it is a similar issue, so I don't want to overlook a larger issue.

The drive that dropped is currently doing a WD diagnostic long test, but it already passed the short test. I guess if it tests find, I'll reinstall it in the array, and find out I bought a new 4TB drive for nothing. :rolleyes:
 
Look at the raw SMART for Current_Pending_Sector, Offline_Uncorrectable, UDMA_CRC_Error_Count and Reallocated_Sector_Ct . Are these all 0?
 
Could be a motherboard or controller issue. I had a board fail slowly where it would just occasionally lose connections to USB and SATA devices. Turned out the thermal tape between the Northbridge and heatsink had failed and was overheating. Didn't notice until I eventually replaced the board when it completely died.
 
Start by posting the SMART data, because for example cabling errors (UDMA CRC Error Count) means the drive itself is not to blame. Not all cabling errors show up in this fashion though.

The kernel log (dmesg) should tell you when and why a device was detached (meaning: becomes invisible to the operating system).
 
OK, it ran the full test on WD DLG, and it came back clean.

For reference, this is a Dell T110ii server with a Sandisk SSD for the boot drive and 4x WD Red 4TB drives. I just tossed the printout from the order a couple of days ago, but I am pretty sure it was purchased new from Dell sometime in early 2014, so a little over a year old.

drescherjm said:
Look at the raw SMART for Current_Pending_Sector, Offline_Uncorrectable, UDMA_CRC_Error_Count and Reallocated_Sector_Ct . Are these all 0?
Click to expand...

It looks like these are all zero.

I went back through the emails to see what I received, and what the error was. The daily output email on 9/25 shows the drive as "OK". 9/26, the drive is not in the list.

9/26 3:29AM received email indicating smartd error stating "unable to open device".

digging through /var/log/messages, was able to find this:

Code: 
Sep 26 03:14:57 servy kernel: ahcich2: Timeout on slot 1 port 0
Sep 26 03:14:57 servy kernel: ahcich2: is 00000000 cs 00000000 ss 00000002 rs 00000002 tfd 40 serr 00000000 cmd 0000c117
Sep 26 03:14:57 servy kernel: (ada2:ahcich2:0:0:0): READ_FPDMA_QUEUED. ACB: 60 08 30 05 b6 40 dc 00 00 00 00 00
Sep 26 03:14:57 servy kernel: (ada2:ahcich2:0:0:0): CAM status: Command timeout
Sep 26 03:14:57 servy kernel: (ada2:ahcich2:0:0:0): Retrying command
Sep 26 03:15:29 servy kernel: ahcich2: AHCI reset: device not ready after 31000ms (tfd = 00000080)
Sep 26 03:15:32 servy kernel: ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
Sep 26 03:15:32 servy kernel: ada2: <WDC WD40EFRX-68WT0N0 80.00A80> s/n WD-XXXXXXXXXX detached
Sep 26 03:15:32 servy kernel: (ada2:ahcich2:0:0:0): Periph destroyed
Sep 26 03:15:37 servy kernel: (aprobe0:ahcich2:0:0:0): NOP. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
Sep 26 03:15:37 servy kernel: (aprobe0:ahcich2:0:0:0): CAM status: ATA Status Error
Sep 26 03:15:37 servy kernel: (aprobe0:ahcich2:0:0:0): ATA status: d1 (BSY DRDY SERV ERR), error: 04 (ABRT )
Sep 26 03:15:37 servy kernel: (aprobe0:ahcich2:0:0:0): RES: d1 04 ff ff ff ff ff ff ff ff ff
Sep 26 03:15:37 servy kernel: (aprobe0:ahcich2:0:0:0): Error 5, Retries exhausted

My office gets pretty hot in the summer. I've been planning to run wires to the finished basement where it's always cooler, but there have been higher priorities. I'm sure this has potentially affected things, as on 9/16 the Seagate external got several airflow warnings/errors.

Thanks to looking at this, I now realize the reason the Seagate USB drive dropped out was because we were playing with circuit breakers that afternoon. For no good reason, the backup drive isn't on UPS, so that answers that mystery.
 
Full SMART info for the drive:

Code: 
smartctl 6.4 2015-06-04 r4109 [x86_64-w64-mingw32-win8] (sf-6.4-1)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD40EFRX-68WT0N0
Serial Number:    WD-XXXXXXXXXXXX
LU WWN Device Id: 5 0014ee 2b49ac199
Firmware Version: 80.00A80
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
Local Time is:    Thu Oct 15 23:55:23 2015 MDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
ATA Security is:  Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(53760) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 537) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x703d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    0
  3 Spin_Up_Time            POS--K   205   171   021    -    6733
  4 Start_Stop_Count        -O--CK   100   100   000    -    31
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   200   200   000    -    0
  9 Power_On_Hours          -O--CK   084   084   000    -    11904
 10 Spin_Retry_Count        -O--CK   100   253   000    -    0
 11 Calibration_Retry_Count -O--CK   100   253   000    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    29
192 Power-Off_Retract_Count -O--CK   200   200   000    -    12
193 Load_Cycle_Count        -O--CK   199   199   000    -    4099
194 Temperature_Celsius     -O---K   119   109   000    -    33
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   200   000    -    0
198 Offline_Uncorrectable   ----CK   100   253   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    0
200 Multi_Zone_Error_Rate   ---R--   100   253   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      6  Ext. Comprehensive SMART error log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  SATA NCQ Queued Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa0-0xa7  GPL,SL  VS      16  Device vendor specific log
0xa8-0xb7  GPL,SL  VS       1  Device vendor specific log
0xbd       GPL,SL  VS       1  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL     VS      93  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     11889         -
# 2  Conveyance offline  Completed without error       00%        10         -
# 3  Conveyance offline  Interrupted (host reset)      90%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       258 (0x0102)
SCT Support Level:                   1
Device State:                        Active (0)
Current Temperature:                    33 Celsius
Power Cycle Min/Max Temperature:     21/42 Celsius
Lifetime    Min/Max Temperature:     21/43 Celsius
Under/Over Temperature Limit Count:   0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (81)

Index    Estimated Time   Temperature Celsius
  82    2015-10-15 15:58    32  *************
 ...    ..( 12 skipped).    ..  *************
  95    2015-10-15 16:11    32  *************
  96    2015-10-15 16:12    33  **************
 ...    ..(  6 skipped).    ..  **************
 103    2015-10-15 16:19    33  **************
 104    2015-10-15 16:20    40  *********************
 ...    ..( 36 skipped).    ..  *********************
 141    2015-10-15 16:57    40  *********************
 142    2015-10-15 16:58    41  **********************
 ...    ..( 30 skipped).    ..  **********************
 173    2015-10-15 17:29    41  **********************
 174    2015-10-15 17:30    40  *********************
 ...    ..( 19 skipped).    ..  *********************
 194    2015-10-15 17:50    40  *********************
 195    2015-10-15 17:51    39  ********************
 ...    ..( 13 skipped).    ..  ********************
 209    2015-10-15 18:05    39  ********************
 210    2015-10-15 18:06    38  *******************
 ...    ..( 36 skipped).    ..  *******************
 247    2015-10-15 18:43    38  *******************
 248    2015-10-15 18:44    37  ******************
 ...    ..( 17 skipped).    ..  ******************
 266    2015-10-15 19:02    37  ******************
 267    2015-10-15 19:03    36  *****************
 ...    ..( 76 skipped).    ..  *****************
 344    2015-10-15 20:20    36  *****************
 345    2015-10-15 20:21    35  ****************
 ...    ..(  4 skipped).    ..  ****************
 350    2015-10-15 20:26    35  ****************
 351    2015-10-15 20:27    34  ***************
 ...    ..(  5 skipped).    ..  ***************
 357    2015-10-15 20:33    34  ***************
 358    2015-10-15 20:34    33  **************
 ...    ..(  6 skipped).    ..  **************
 365    2015-10-15 20:41    33  **************
 366    2015-10-15 20:42    32  *************
 367    2015-10-15 20:43    32  *************
 368    2015-10-15 20:44    32  *************
 369    2015-10-15 20:45    31  ************
 ...    ..(  2 skipped).    ..  ************
 372    2015-10-15 20:48    31  ************
 373    2015-10-15 20:49    30  ***********
 ...    ..(  6 skipped).    ..  ***********
 380    2015-10-15 20:56    30  ***********
 381    2015-10-15 20:57    29  **********
 ...    ..( 10 skipped).    ..  **********
 392    2015-10-15 21:08    29  **********
 393    2015-10-15 21:09    28  *********
 ...    ..(  8 skipped).    ..  *********
 402    2015-10-15 21:18    28  *********
 403    2015-10-15 21:19    27  ********
 ...    ..( 17 skipped).    ..  ********
 421    2015-10-15 21:37    27  ********
 422    2015-10-15 21:38    26  *******
 ...    ..( 19 skipped).    ..  *******
 442    2015-10-15 21:58    26  *******
 443    2015-10-15 21:59     ?  -
 444    2015-10-15 22:00    26  *******
 445    2015-10-15 22:01    26  *******
 446    2015-10-15 22:02    27  ********
 447    2015-10-15 22:03    27  ********
 448    2015-10-15 22:04    28  *********
 ...    ..(  5 skipped).    ..  *********
 454    2015-10-15 22:10    28  *********
 455    2015-10-15 22:11    29  **********
 ...    ..(  3 skipped).    ..  **********
 459    2015-10-15 22:15    29  **********
 460    2015-10-15 22:16    30  ***********
 ...    ..(  9 skipped).    ..  ***********
 470    2015-10-15 22:26    30  ***********
 471    2015-10-15 22:27    31  ************
 ...    ..(  9 skipped).    ..  ************
   3    2015-10-15 22:37    31  ************
   4    2015-10-15 22:38    32  *************
   5    2015-10-15 22:39    31  ************
 ...    ..(  3 skipped).    ..  ************
   9    2015-10-15 22:43    31  ************
  10    2015-10-15 22:44    30  ***********
 ...    ..(  9 skipped).    ..  ***********
  20    2015-10-15 22:54    30  ***********
  21    2015-10-15 22:55    29  **********
 ...    ..( 21 skipped).    ..  **********
  43    2015-10-15 23:17    29  **********
  44    2015-10-15 23:18    28  *********
  45    2015-10-15 23:19    28  *********
  46    2015-10-15 23:20     ?  -
  47    2015-10-15 23:21    28  *********
  48    2015-10-15 23:22    29  **********
 ...    ..( 10 skipped).    ..  **********
  59    2015-10-15 23:33    29  **********
  60    2015-10-15 23:34    30  ***********
 ...    ..(  6 skipped).    ..  ***********
  67    2015-10-15 23:41    30  ***********
  68    2015-10-15 23:42    31  ************
 ...    ..(  7 skipped).    ..  ************
  76    2015-10-15 23:50    31  ************
  77    2015-10-15 23:51    32  *************
 ...    ..(  3 skipped).    ..  *************
  81    2015-10-15 23:55    32  *************

SCT Error Recovery Control:
           Read:     70 (7.0 seconds)
          Write:     70 (7.0 seconds)

Device Statistics (GP/SMART Log 0x04) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2            3  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2            3  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000f  2            0  R_ERR response for host-to-device data FIS, CRC
0x0012  2            0  R_ERR response for host-to-device non-data FIS, CRC
0x8000  4        53550  Vendor specific
 
iroc409 said:
OK, it ran the full test on WD DLG, and it came back clean.
Click to expand...
By doing this you have destroyed the evidence since running such tests can make pending sectors disappear. It will leave no trace in the SMART data.

Next time, secure the SMART data first prior to using any active utility or reading/writing anything from/to the drive.

Because right now, you will never know whether a bad sector was the culprit or not. The evidence is gone!
 
_CiPHER_ said:
By doing this you have destroyed the evidence since running such tests can make pending sectors disappear. It will leave no trace in the SMART data.

Next time, secure the SMART data first prior to using any active utility or reading/writing anything from/to the drive.

Because right now, you will never know whether a bad sector was the culprit or not. The evidence is gone!
Click to expand...

Ah, I didn't realize that. I had already started that path before starting the thread.

Would a pending sector show up as reallocated after a SMART test?
 
It would, but only if the sector has been physically damaged. In most cases, it concerns unreadable sectors without physical damage. This is due to uBER: uncorrectable Bit-Error Rate which increasingly is becoming a problem as data densities grow but ECC errorcorrection stays the same.

So only damaged sectors that have been swapped with reserve sectors would show up as Reallocated Sector Count. Most bad sectors today (90% i estimate) are bad sectors due to insufficient ECC (uBER bad sectors) and not because they are physically damaged.
 
I find drives do this when they are close to death, their firmware freezes and stops responding to commands and has to be power-cycled, but then they seem as if nothing is wrong. They could go tomorrow or 3 months from now, but that drive is on its way out.
 
You must log in or register to reply here.
Back
Top