mdadm - unable to create degraded RAID5 array

[CC_Pirate]

Hi

Due to some special circumstances I need to create a RAID5 array with two disks, i.e. a degraded array with one drive "missing", using mdadm

I have two 4TB disks. What I've done is booted up Ubuntu 14.04 using a Live-CD. Then I created a single, primary partition on each of them of the type "Linux RAID autodetect".

Then I created the array like this:

sudo mdadm --create --verbose /dev/md1 --level=5 --raid-devices=3 /dev/sda1 /dev/sdc1 missing

which created the array and reported no errors.

When I did a "cat /proc/mdstat" I saw the array was created fine:..

Personalities : [raid6] [raid5] [raid4] 
md1 : active raid5 sdc1[1] sda1[0]
      4294702080 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_]
      
unused devices: <none>

...but no recovery progress bar was displayed. It seemed like it either spent only seconds building the array, or never began the process.

Regardless, I created a filesystem on the array using

sudo mkfs.ext4 /dev/md1

. This operation however, failed the array immediately. Using "dmesg" I saw some superblock read or write errors, and "cat /proc/mdstat" showed that the /dev/sdc1 device had failed, rendering the array useless.

Has anyone got any suggestions as to what might be wrong? I need to get this to work in order to upgrade the storage on my machine..

Thanks!

 

[omniscence]

It can only start to resync the array once you add the third device.
Creating an array just writes a few kilobytes for the superblocks to the disks which takes milliseconds.
And then syncs the array which takes a few hours. Which it can't as the array is degraded.
It did exactly what you want and the outcome is expected.
You know that a degraded RAID5 provides no protection? You could create a RAID0 with almost the same effect.

That that array fails on mkfs is another issue, it would help if you post exactly the dmesg output and the output of 'mdadm --detail /dev/md1'.

EDIT: I read your other thread again... are you going to degrade your old array, connect 2 of your new drives to the machine, build a degraded array from them, resync that array to the third disk and then grow it by the fourth disk? That would be an awful idea... although maybe your easiest choice if there is no additional port or controller or PC available.

What I would do: shrink your array by one disk and then block level copy it to the new disks one by one.

I would think you can also deactivate the array, block level copy 4 of the old disks to the new disks, resulting in a degraded but larger array and shrink it after this by one disk. This should work in theory, never tried something like this. I always could make enough ports available temporarily to avoid such measures. The advantage would be that the old array is unaltered if something bad happens. The filesystem size should not be changed until the process is complete, and after the block level copy you should check that the array correctly got the new size.

 

[CC_Pirate]

It can only start to resync the array once you add the third device.
Creating an array just writes a few kilobytes for the superblocks to the disks which takes milliseconds.
And then syncs the array which takes a few hours. Which it can't as the array is degraded.
It did exactly what you want and the outcome is expected.
You know that a degraded RAID5 provides no protection? You could create a RAID0 with almost the same effect.

That that array fails on mkfs is another issue, it would help if you post exactly the dmesg output and the output of 'mdadm --detail /dev/md1'.

EDIT: I read your other thread again... are you going to degrade your old array, connect 2 of your new drives to the machine, build a degraded array from them, resync that array to the third disk and then grow it by the fourth disk? That would be an awful idea... although maybe your easies choice if there is no additional port or controller or PC available.

Thanks for input.

To clarify, heres what I want to do:
I have 6 SATA ports, and an existing 5x2TB RAID5 array, and in addition a single SSD with the OS installation. I want to move to a 4x4TB RAID5 array.
So the plan is to
1) unplug the SSD and one of the 2TB drives so the existing array runs in degraded mode
2) plug in two 4TB drives into the now free SATA ports
3) boot from a Live-CD and create a new degraded array using the two 4TB drives
4) transfer data from old degraded array to new degraded array
5) plug in third 4TB drive and re-sync new array
6) plug in fourth 4TB drive and grow the new array to four disks
7) unplug and remove all 2TB disks


Step 1, 2 and 3 has succeeded, but as stated I am unable to create a filesystem on the new degraded array.

Do you have another suggestion as to how this could be done?

 

[omniscence]

See my suggestion above for an alternative approach.

What you are were doing sounds correct, however. Please post the exact sequence of commands and the output of dmesg after the array fails. What you did should not cause a failed array.
Please also post the output of 'lsblk', 'blkid', 'smartctl -x /dev/sda' and 'smartctl -x /dev/sdc', I try to get a more precise overview of your system. You may have to install smartctl first in your Live environment.

 

[CC_Pirate]

What I ran was the following:

sudo mdadm --create --verbose /dev/md1 --level=5 --raid-devices=3 /dev/sda1 /dev/sdc1 missing

which successfully created the array. Then I ran

sudo mkfs.ext4 /dev/md1

to create a filesystem on the new array, which caused the array to fail.

Resulting output of `dmesg` after running these commands (which were ran in sequence) can be seen here. Looks like a disk is failing but I dont know if its because Ive done something wrong or if its an actual hardware error. The 4TB disks are new.


`lsblk` produces this:

NAME    MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda       8:0    0   3,7T  0 disk  
└─sda1    8:1    0     2T  0 part  
  └─md1   9:1    0     4T  0 raid5 
sdb       8:16   0   1,8T  0 disk  
└─sdb1    8:17   0   1,8T  0 part  
sdc       8:32   0   3,7T  0 disk  
└─sdc1    8:33   0     2T  0 part  
  └─md1   9:1    0     4T  0 raid5 
sdd       8:48   0   1,8T  0 disk  
└─sdd1    8:49   0   1,8T  0 part  
sde       8:64   0   1,8T  0 disk  
└─sde1    8:65   0   1,8T  0 part  
sdf       8:80   0   1,8T  0 disk  
└─sdf1    8:81   0   1,8T  0 part  
loop0     7:0    0   922M  1 loop  /rofs

which shows the four old 2TB disks and the two new 4TB disks.

Output of `blkid` here.

Output of `smartctl -x /dev/sda` here.

Now as Im nearing the end of my post and running your commands, I think Im seeing something thats indicating a hardware problem with /dev/sdc: Check the output of `smartctl -x -T permissive /dev/sdc` here.

Im going to continue testing to try to find out if the disk is healthy or not.

Thanks for helping me out, I am in deep water here.

 

[TCM2]

If you can afford to juggle your data in this overly convoluted and dangerous way, you surely must have a backup. Just build the new array and restore the backup.

If you neither have a backup or a spare machine on which to build the new array, I'd strongly suggest to stop whatever you're doing and getting either of these first.

My toe nails are rolling up.

 

[omniscence]

Even your working drive already has a pending sector, probably from an incomplete write.
There is a high chance that your power, your SATA cabling or your controller has a problem.
Replace the SATA cables and/or PSU and try again.
I strongly recommend that you sort this out before continuing.

 

[CC_Pirate]

If you can afford to juggle your data in this overly convoluted and dangerous way, you surely must have a backup. Just build the new array and restore the backup.

If you neither have a backup or a spare machine on which to build the new array, I'd strongly suggest to stop whatever you're doing and getting either of these first.

My toe nails are rolling up.

Yes I have a machine at my parents house with a similarly sized array that I rsync important data to at set intervals. That is my backup.

I didnt think initially that this was so risky - Im running both arrays degraded for a period of a few days while the operation is underway, what are the chances of a disk failing exactly then?

But Im starting to realize that there risks are greater than the potential reward of convenience (not having to bring over my backup machine and transferring files over network) here.

Even your working drive already has a pending sector, probably from an incomplete write.
There is a high chance that your power, your SATA cabling or your controller has a problem.
Replace the SATA cables and/or PSU and try again.
I strongly recommend that you sort this out before continuing.

Thanks. Could you point me to what output indicated this error to you? I'm going to do some extensive testing to verify that all disks, ports, controllers and SATA cables are A-OK.

While on the topic - what would you recommend that I use to test the disks under Linux/Ubuntu?

 

[omniscence]

Thanks. Could you point me to what output indicated this error to you? I'm going to do some extensive testing to verify that all disks, ports, controllers and SATA cables are A-OK.

Your drive has a pending sector:

197 Current_Pending_Sector  -O--CK   200   200   000    -    1

That means that there is a sector that the drive tried to read but could not, e.g. the contents did not match its checksum and the data is considered invalid. This could point to a media error, but considering what your other drive shows it is more likely that the drive got interrupted while writing the sector.

While on the topic - what would you recommend that I use to test the disks under Linux/Ubuntu?

I would use 'badblocks -wsv /dev/sdX' to test the disks for media errors before moving data to them. Be aware that this overwrites everything including partition table and raid superblocks, so stop the raid before doing that. Be careful to use the right device, root access is a scary thing with those destructive commands.

There is something wrong with /dev/sdc, which is not just some media errors. Either the drive is defective or some other component in the chain. I would change around cables, ports and drives to so see what exactly causes your error, be aware that you put your already degraded array at risk.
I have been using mdadm for like 14 years now and I always have been able to recover my failed arrays, but that does not mean that you should put it at risk intentionally.

 

[CC_Pirate]

Thanks omniscence.
What I'll do is test all the new disks thoroughly and verify the whole system.
Then I'll plug all 2TB disks back in and run it back to non-degraded mode.

Then I'll bring my backup-server around and use it to aid the data-transfer.

 

[ezno_matrix]

I do the same steps your are suggesting when expanding my mdadm RAID. See here for details.

I have never had issue with creating degraded partitions. Any reason you are partitioning the disks? If the disks will not be used for multiple RAIDs, there is no need to partition.

 

[CC_Pirate]

I do the same steps your are suggesting when expanding my mdadm RAID. See here for details.

I have never had issue with creating degraded partitions. Any reason you are partitioning the disks? If the disks will not be used for multiple RAIDs, there is no need to partition.

Thanks. In theory it should work fine, but folks are trying to scare me here. Granted, I will run without redundancy for at least a few days while the operations run.

I read a guide that suggested I created single primary partitions on the disks of type "Linux RAID autodetect" before including them in an array.. But after all I create a new file system on the new array so I guess the per-disk partitions get wiped then regardless? Thinking about it more it sounds like an unnecessary step, I agree.

Now I'm in the process of finding out if I have any failing disks/cables/power-leads/controllers.

 

[ezno_matrix]

Thanks. In theory it should work fine, but folks are trying to scare me here. Granted, I will run without redundancy for at least a few days while the operations run.

Works fine in practice as well. A lot of [H] users are super paranoid about data lost (not that this is a bad thing).

I read a guide that suggested I created single primary partitions on the disks of type "Linux RAID autodetect" before including them in an array.. But after all I create a new file system on the new array so I guess the per-disk partitions get wiped then regardless? Thinking about it more it sounds like an unnecessary step, I agree.

The "Linux RAID autodetect" partition type was required back in the day when the Kernel could only detect MD disks using the partition type. The current Kernel reads the MD super block so no partition is required. Partition was also needed way way back in the day when different disk manufacture's didn't allocate the same size for similarly identified disks (i.e. IIRC, Seagate's 30GB disks were slightly smaller than Fijitsu's 30GB disks).

Now I'm in the process of finding out if I have any failing disks/cables/power-leads/controllers.

This process is highly dependent on stable hardware. Luckily in your case you still have the original data on the 5x2TB disks. Make sure these are not touched until everything is up and running on the new disks.

 

[CC_Pirate]

Thanks. Im learning some new things here.

Does a long test using 'smartctl' suffice as a test of disk failure? 'badblocks -wsv' is taking years to complete.. I'd like a pre-test that can indicate to me what the most obvious errors are.

 

[omniscence]

The long smart test will just read the whole disk and stop on the first error. In your case it will most likely fail on your pending sector. The badblocks test writes to the complete block range and reads them again, so it can detect bad sectors and the drive can reallocate them. This process will reset your pending sector as a side effect.

At the same time badblocks puts a certain workload on your disks. If it has some weak points you can cause it to fail before entrusting data to it. A few percent of the drives that leave the factory are defective after all.

You have to decide for yourself what level of reliability you need.

 

[CC_Pirate]

It turned out all my new 4TB disks were bad. Should have expected the worst when I saw how poorly they were packaged.

After receiving new disks from the dealer, I stuck to the plan and tried again; degrade existing array to four drives, create new degraded array using two 4TB drives, and copy data over. Then grow the new array with the remaining 4TB drives.

Worked like a charm, no issues. Syncing took half a day and growing to four disks took a good day. Resizing the FS took only a few mins.

I now have a speedy array with lots of space. Thanks for helping out!