Scary, almost lost my raid array

[Red Squirrel]

Raid 6 is looking good right about now.

Can anyone explain what would cause this?


First email:

This is an automatically generated mail message from mdadm
running on borg.loc

A Fail event had been detected on md device /dev/md0.

It could be related to component device /dev/sde.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sdb[0] sdh[5] sdg[6](S) sdf[4] sde[7](F) sdd[2] sdc[1]
      3907039744 blocks level 5, 64k chunk, algorithm 2 [5/4] [UUU_U]
      [>....................]  recovery =  0.0% (0/976759936) finish=162793.3min speed=0K/sec
      
unused devices: <none>

5 days later:

This is an automatically generated mail message from mdadm
running on borg.loc

A Fail event had been detected on md device /dev/md0.

It could be related to component device /dev/sdf.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4] 
md_d0 : inactive sde[0](S)
      976762496 blocks
       
md0 : active raid5 sdg[5] sdb[0] sdh[3] sdf[6](F) sdd[2] sdc[1]
      3907039744 blocks level 5, 64k chunk, algorithm 2 [5/4] [UUUU_]
      [>....................]  recovery =  0.0% (0/976759936) finish=162793.3min speed=0K/sec
      
unused devices: <none>

Couple minutes later:

This is an automatically generated mail message from mdadm
running on borg.loc

A FailSpare event had been detected on md device /dev/md0.

It could be related to component device /dev/sdg.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4] 
md_d0 : inactive sde[0](S)
      976762496 blocks
       
md0 : active raid5 sdg[5](F) sdb[0] sdh[3] sdd[2] sdc[1]
      3907039744 blocks level 5, 64k chunk, algorithm 2 [5/4] [UUUU_]
      [>....................]  recovery =  2.0% (20294400/976759936) finish=416.9min speed=38233K/sec
      
unused devices: <none>

Right now all 5 drives are up. This is really weird. Is there any way I can get more diagnostic info on the raid to see what's going on?


This is the details now:

/dev/md0:
        Version : 0.90
  Creation Time : Sat Sep 20 02:15:28 2008
     Raid Level : raid5
     Array Size : 3907039744 (3726.04 GiB 4000.81 GB)
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Mon Jun  6 19:40:44 2011
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 11f961e7:0e37ba39:2c8a1552:76dd72ee
         Events : 0.773870

    Number   Major   Minor   RaidDevice State
       0       8       32        0      active sync   /dev/sdc
       1       8       16        1      active sync   /dev/sdb
       2       8       48        2      active sync   /dev/sdd
       3       8       80        3      active sync   /dev/sdf
       4       8       64        4      active sync   /dev/sde