• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

mdadm raid 6 issue.

S

sonicz

n00b
Joined
May 21, 2013
Messages
28
I managed to get most of my important data saved, lucky me

Now I'm gonna make a new raid, but before I do this I wanna aske if its possible to do the following and if its a stupid approach:

1. Disks are cleaned out.

2. Making a new raid 6 array.

3. Partition the array with parted.

4. Creating ext4 64bit filesystem.

Or should I do the partition on the drives before making the array? - the old way I made partition on each disk, made the array and then used LVM. But have been told that I should avoid using it (leaving out layers on the disks)

But hit me, what would be the best approach on this? - not found anything conclusive by googling - Also please take notice that I need to be able to grow my filesystem.











Old issue
_________________________________________________________________________________________

Hi Everyone

I'm currently having some big troubles with my raid. My old machine crashed and burned = I have no old configs.

The crash apparently also crashed my mdadm raid.

To make a long story short I have tried to assemble my raid with --assemble --force and the disk with a equal or close event Count.

This resulted in 5 out 7 disks. Although I'm not able to Mount the array. I then tried to rebuild the raid by using --add of the 2 last disks. But this failed with mdadm marking one of the assembled disks as faulty. I can then assemble it Again. I have tried rebuild with both disks at the same time, and 1 at a time. With the same result.


Now I'm considering to run:

mdadm --create --assume-clean --level=6 --raid-devices=7 --size=2930133504 /dev/md0 /dev/sdg1 /dev/sdh1 /dev/sde1 /dev/sdb1 /dev/sdd1 /dev/sdc1 /dev/sdf1

But since disk sdg1 keep failing, maybe I should set it as missing? - maybe also sdf1 since it for some reason runs very slow when recovering.

The command is based on the following:

Device Role : Active device 3 sdb1
Device Role : Active device 5 sdc1
Device Role : Active device 4 sdd1
Device Role : Active device 2 sde1
Device Role : Active device 6 sdf1
Device Role : Active device 0 sdg1
Device Role : Active device 1 sdh1

/dev/sdb1:
Events : 2907469
/dev/sdc1:
Events : 2907469
/dev/sdd1:
Events : 2907469
/dev/sde1:
Events : 2907469
/dev/sdf1:
Events : 2904967
/dev/sdg1:
Events : 2907463
/dev/sdh1:
Events : 2907468

/dev/sdb1:
Array State : .AA.A.A ('A' == active, '.' == missing)
/dev/sdc1:
Array State : .AA.A.A ('A' == active, '.' == missing)
/dev/sdd1:
Array State : .AA.A.A ('A' == active, '.' == missing)
/dev/sde1:
Array State : .AA.A.A ('A' == active, '.' == missing)
/dev/sdf1:
Array State : .AA.A.A ('A' == active, '.' == missing)
/dev/sdg1:
Array State : AAAAA.A ('A' == active, '.' == missing)
/dev/sdh1:
Array State : .AA.A.A ('A' == active, '.' == missing)

/dev/sdb1:
Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB)
/dev/sdc1:
Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB)
/dev/sdd1:
Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB)
/dev/sde1:
Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB)
/dev/sdf1:
Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB)
/dev/sdg1:
Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB)
/dev/sdh1:
Used Dev Size : 5860267008 (2794.39 GiB 3000.46 GB)


Any input is welcome, is there something else I can do? - as I find the above command is last resort.
 
Last edited:
But since disk sdg1 keep failing, maybe I should set it as missing?
Click to expand...

Probably. Did you check the SMART raw data for the drives. At work (where I have around a dozen mdadm raid6 arrays) when a drive is kicked out of an array I look at the SMART raw data. I also subject the drive to a 4 pass badblocks which it must pass 2 times before it can go back into the array.

BTW here is a script I use to check the smart on my mdadm arrays:

https://raw.githubusercontent.com/d.../master/Other/shell-scripts/examine_mdraid.sh

Code: 
#!/bin/bash

function process_device()
{
        echo -n ${device}
        hdparm -I /dev/${device} | grep "al Number"
        smartctl --all /dev/${device} | grep -e "Capacity" -e "Reallocated_Sector_Ct" -e "Current_Pending_Sector" -e "Offline_Uncorrectable" -e "UDMA_CRC_Error_Count" -e "Hardware_ECC_Recovered" -e "Power_On_Hours"
        echo
}

for a in /dev/sd?;
do
        device=${a/\/dev\//}
        grep ${device} /proc/mdstat >> /dev/null
        if [ $? == 0 ];
        then
                process_device
        fi
done

With output like this:
Code: 
fileserver1 shell-scripts # sh examine_mdraid.sh
sda     Serial Number:      MN1220F31NWX7D
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0012   096   096   000    Old_age   Always       -       32045
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

sdb     Serial Number:      42D1KCFNF
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
  5 Reallocated_Sector_Ct   0x0033   100   100   050    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   044   044   000    Old_age   Always       -       22755
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0

sdc     Serial Number:      82N2K3UPF
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
  5 Reallocated_Sector_Ct   0x0033   100   100   050    Pre-fail  Always       -       98
  9 Power_On_Hours          0x0032   054   054   000    Old_age   Always       -       18430
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0

sdd     Serial Number:      82N2K3VNF
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
  5 Reallocated_Sector_Ct   0x0033   100   100   050    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   051   051   000    Old_age   Always       -       19995
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0

sde     Serial Number:      MN1220F31GZG7D
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0012   096   096   000    Old_age   Always       -       32046
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

sdf     Serial Number:      42E2KC0NF
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
  5 Reallocated_Sector_Ct   0x0033   100   100   050    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   044   044   000    Old_age   Always       -       22757
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0

sdg     Serial Number:      42D1KCO0F
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
  5 Reallocated_Sector_Ct   0x0033   100   100   050    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   044   044   000    Old_age   Always       -       22757
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0

sdh     Serial Number:      82N2K3URF
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
  5 Reallocated_Sector_Ct   0x0033   100   100   050    Pre-fail  Always       -       613
  9 Power_On_Hours          0x0032   050   050   000    Old_age   Always       -       20009
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0

sdi     Serial Number:      MN1220F33JULPD
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0012   097   097   000    Old_age   Always       -       23846
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

sdk     Serial Number:      Z1X05ABQ
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   082   082   000    Old_age   Always       -       16146
195 Hardware_ECC_Recovered  0x001a   045   024   000    Old_age   Always       -       184103459
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

sdl     Serial Number:      Z1X059LQ
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   082   082   000    Old_age   Always       -       16146
195 Hardware_ECC_Recovered  0x001a   044   021   000    Old_age   Always       -       97657745
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

The only concerning data above is the 98 reallocated sectors on 82N2K3UPF. However these happened when the AC malfunctioned and the server room hit 120F. This drive has been stable at 98 reallocated sectors for over a year now since the incident and I have thoroughly tested it at least 1/2 dozen times.
 
Last edited:
Managed to get access to my data with 5 disks. So for now im gonna Work on damage control :)

Thanks for your reply.
 
I managed to get most of my important data saved, lucky me :)

Now I'm gonna make a new raid, but before I do this I wanna aske if its possible to do the following and if its a stupid approach:

1. Disks are cleaned out.

2. Making a new raid 6 array.

3. Partition the array with parted.

4. Creating ext4 64bit filesystem.

Or should I do the partition on the drives before making the array? - the old way I made partition on each disk, made the array and then used LVM. But have been told that I should avoid using it (leaving out layers on the disks)

But hit me, what would be the best approach on this? - not found anything conclusive by googling :) - Also please take notice that I need to be able to grow my filesystem.
 
You must log in or register to reply here.
Back
Top