upgraded kernel mdadm not starting on boot

M

misterpat

Weaksauce
Joined
Dec 24, 2011
Messages
103
Hello all,

I upgraded to the 3.2.0 kernel and everything seemed to go fine. I was messing around with Folding@home and noticed my raid array wasn't mounted. In fact mdadm wasn't even running.

If I issue this command, It will spit this back to me. This was entered right after it rebooted.

Code: 
login as: root
root@*********** password:
Last login: Sat Jan 21 16:58:20 2012 from 192.168.1.199
[root@Mega-Nas ~]# cat /proc/mdstat
Personalities :
unused devices: <none>
[root@Mega-Nas ~]# mdadm --assemble --scan -v
mdadm: looking for devices for /dev/md0
mdadm: /dev/sda1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 6.
mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 5.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot 7.
mdadm: /dev/sda1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 6.
mdadm: added /dev/sdb1 to /dev/md0 as 1
mdadm: added /dev/sdc1 to /dev/md0 as 2
mdadm: added /dev/sdd1 to /dev/md0 as 3
mdadm: added /dev/sde1 to /dev/md0 as 4
mdadm: added /dev/sdg1 to /dev/md0 as 5
mdadm: added /dev/sdf1 to /dev/md0 as 6
mdadm: added /dev/sdh1 to /dev/md0 as 7
mdadm: added /dev/sda1 to /dev/md0 as 0
mdadm: /dev/md0 has been started with 8 drives.
mdadm: looking for devices for /dev/md/0
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: cannot open device /dev/sdb1: Device or resource busy
mdadm: cannot open device /dev/sdc1: Device or resource busy
mdadm: cannot open device /dev/sdd1: Device or resource busy
mdadm: cannot open device /dev/sdf1: Device or resource busy
mdadm: cannot open device /dev/sdg1: Device or resource busy
mdadm: cannot open device /dev/sde1: Device or resource busy
mdadm: cannot open device /dev/sde1: Device or resource busy
mdadm: cannot open device /dev/sdh1: Device or resource busy
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: cannot open device /dev/sdf1: Device or resource busy
mdadm: looking for devices for /dev/md/0
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: cannot open device /dev/sdb1: Device or resource busy
mdadm: cannot open device /dev/sdc1: Device or resource busy
mdadm: cannot open device /dev/sdd1: Device or resource busy
mdadm: cannot open device /dev/sdf1: Device or resource busy
mdadm: cannot open device /dev/sdg1: Device or resource busy
mdadm: cannot open device /dev/sde1: Device or resource busy
mdadm: cannot open device /dev/sde1: Device or resource busy
mdadm: cannot open device /dev/sdh1: Device or resource busy
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: cannot open device /dev/sdf1: Device or resource busy
[root@Mega-Nas ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sda1[10] sdh1[9] sdf1[11] sdg1[5] sde1[8] sdd1[3] sdc1[2] sdb1[1]
      5860557696 blocks super 1.2 level 6, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]

unused devices: <none>

Notice that after running 'mdadm --assemble --scan -v' and all those errors, I can run 'cat /proc/mdstat' and the array is fine then. I can mount and use it without problem until reboot.

I have been rebooting back to the older kernel, the one the array was built on and the problem still remains.

Any help with this would be greatly appreciated.
 
Last edited:
Array seems fine, it assembles the drives as /dev/md0 and that finishes fine. After that its screwing up and trying to assemble another array, /dev/md/0, which is weird... it tries to grab same disks but they're already in an array at that point and you see error messages.

/dev/md0 is current naming method, /dev/md/0 is kinda old method, often if that exists anymore its just a link to /dev/md0. I don't get whats going wrong, you don't happen to have anything in /dev/mdadm.conf?
 
bexamous said:
I don't get whats going wrong, you don't happen to have anything in /dev/mdadm.conf?
Click to expand...


I see what You mean, Should I delete anything in here?

/etc/mdadm.conf
Code: 
DEVICE /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sdf1 /dev/sdg1 /dev/sde1 /dev/sde1 /dev/sdh1 /dev/sda1 /dev/sdf1 /dev/sdh1
ARRAY /dev/md0 level=raid6 devices=/dev/sda1,/dev/sdb1,/dev/sdc1,/dev/sdd1,/dev/sdf1,/dev/sdg1,/dev/sde1,/dev/sde1,/dev/sde1,/dev/sdh1,/dev/sda1,/dev/sdf1,/dev/sdh1
MAILADDR **********@gmail.com
ARRAY /dev/md/0 metadata=1.2 UUID=58ad9cae:e1e82031:fcfde786:ba99b399 name=Mega-Nas:0
 
Last edited:
I believe /dev/md0 and /dev/md/0 are the same exact thing...maybe it's seeing both and having issues? Comment the first 'ARRAY /dev/md0" out. You should only need that last line...at least that's how it works for my 8 disk raid6 array.

Code: 
ARRAY /dev/md0 metadata=1.2 UUID=5f3eda42:7b9c23ca:fdabdfe5:ce951d6a
ARRAY /dev/md5 metadata=0.90 UUID=c0370395:f845630f:26c640ad:ebf0ffa6

md0 = 4 disk raid10, md5 = 8 disk raid6
 
Yeah seems like source of issues, you have two arrays being described but only have one actual array. Two lines describing arrays: ARRAY /dev/md0 and ARRAY /dev/md/0.

You can probably just comment out last line with /dev/md/0.

Honestly though I'm not even sure mdadm.conf is even needed anymore-- seems like just a source of problems.

These days kernel will assemble any arrays it finds without needing you to tell it about them. Then in /etc/fstab instead of using '/dev/md0' or something use the UUID instead, eg:
UUID=58ad9cae:e1e82031:fcfde786:ba99b399 /mnt/somename ext4 auto 0 0

Won't matter if its assembled as /dev/md0 or /dev/penis.

But no reason to mess with anything if its not broke. I'd try to just comment out that last line in your mdadm.conf with /dev/md/0 and see if it fixes problem. Or comment out the first ARRAY line... either way. Usually if you create array /dev/md0, it creates a symlink on its own from /dev/md/0 to /dev/md0... but if you specifically give it two ARRAY lines it might be confused.
 
Thanks for the help.

I made a backup of the mdadm.conf and edited it to this

Code: 
DEVICE /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1
ARRAY /dev/md/0 metadata=1.2 UUID=58ad9cae:e1e82031:fcfde786:ba99b399 name=Mega-Nas:0

Still getting this after a reboot

Code: 
login as: root
root@************* password:
Last login: Sat Jan 21 22:28:37 2012 from 192.168.1.199
[root@Mega-Nas ~]# cat /proc/mdstat
Personalities :
unused devices: <none>
[root@Mega-Nas ~]# mdadm --assemble --scan -v
mdadm: looking for devices for /dev/md/0
mdadm: /dev/sda1 is identified as a member of /dev/md/0, slot 0.
mdadm: /dev/sdb1 is identified as a member of /dev/md/0, slot 1.
mdadm: /dev/sdc1 is identified as a member of /dev/md/0, slot 2.
mdadm: /dev/sdd1 is identified as a member of /dev/md/0, slot 3.
mdadm: /dev/sde1 is identified as a member of /dev/md/0, slot 4.
mdadm: /dev/sdf1 is identified as a member of /dev/md/0, slot 6.
mdadm: /dev/sdg1 is identified as a member of /dev/md/0, slot 5.
mdadm: /dev/sdh1 is identified as a member of /dev/md/0, slot 7.
mdadm: added /dev/sdb1 to /dev/md/0 as 1
mdadm: added /dev/sdc1 to /dev/md/0 as 2
mdadm: added /dev/sdd1 to /dev/md/0 as 3
mdadm: added /dev/sde1 to /dev/md/0 as 4
mdadm: added /dev/sdg1 to /dev/md/0 as 5
mdadm: added /dev/sdf1 to /dev/md/0 as 6
mdadm: added /dev/sdh1 to /dev/md/0 as 7
mdadm: added /dev/sda1 to /dev/md/0 as 0
mdadm: /dev/md/0 has been started with 7 drives (out of 8) and 1 rebuilding.

I'm going to try changing /dev/md/0 to /dev/md0 and reboot.

EDIT:

I just saw bexamous's post. also going to post fstab once rebooted.
 
Well no more error messages, so that is fixed. Problem now is that its not assembling by default, not sure why that isn't happening.

I think mdadm.conf should just be:
ARRAY /dev/md0 metadata=1.2 UUID=58ad9cae:e1e82031:fcfde786:ba99b399 name=Mega-Nas:0

The DEVICE line is unneeded, by default DEVICE will set to 'partitions' which means it'll scan any partitions it can find on the system. If you set it to some set of drives, it will only scan those drives. Yeah you're current DEVICE line includes all drives, so its not a problem... but its not doing anything useful and will only cause a problem in the future if you add another drive.
 
Ok, here is the current mdadm.conf generated from 'mdadm --detail --scan > /etc/mdadm.conf'

Code: 
ARRAY /dev/md/0 metadata=1.2 spares=1 name=Mega-Nas:0 UUID=58ad9cae:e1e82031:fcfde786:ba99b399

fstab

Code: 
#
# /etc/fstab
# Created by anaconda on Wed Dec 28 19:14:18 2011
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=fadb7c46-2935-4915-aee5-6d4aab91c234 /                       ext4    defaults        1 1
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
UUID=58ad9cae:e1e82031:fcfde786:ba99b399   /mnt/home               ext4    defaults        0  0
 
Last edited:
After updating the array, did you run an update-initramfs -k <kernelver or all> -u ?
 
Robstar said:
After updating the array, did you run an update-initramfs -k <kernelver or all> -u ?
Click to expand...

I actually backed up my data and went back to Debian. On a clean install, it worked great with the LSI 9211 HBA which is the whole reason I went with CentOS. While CentOS is a great OS, it was a major pain in the a$$, at least for me.

I'm going to build a test box to test out upgrading the kernel as to not mess up my good box.

Thanks for all the help with this everyone.
 
You must log in or register to reply here.
Back
Top