mdadm --grow Causes Hard Lock

[parityboy]

I have 7 Samsung 2TB HD204UIs connected to a Dell SAS 6/iR, configured as a RAID 5 + hot spare array and sitting under an LVM volume. The machine is running Ubuntu 10.04.3 LTS (kernel 2.6.32-36-generic), using mdadm version 2.6.7.1

I'm trying to grow the array by including the spare, but every time I try to execute "sudo mdadm --grow /dev/md0 --raid-devices=7", the system immediately hard locks, and needs to be reset.

[54614.003099] Free swap  = 4764160kB
[54614.003101] Total swap = 4803392kB
[54614.014371] 916448 pages RAM
[54614.014375] 689122 pages HighMem
[54614.014377] 13928 pages reserved
[54614.014379] 11443 pages shared
[54614.014382] 238068 pages non-shared
[54614.014414] mdadm: page allocation failure. order:0, mode:0x10
[54614.014420] Pid: 17100, comm: mdadm Tainted: P   M       2.6.32-37-generic #81-Ubuntu
[54614.014424] Call Trace:
[54614.014432]  [<c058aa24>] ? printk+0x1d/0x21
[54614.014440]  [<c01d1a3b>] __alloc_pages_slowpath+0x3bb/0x4b0
[54614.014446]  [<c01d1c6a>] __alloc_pages_nodemask+0x13a/0x170
[54614.014461]  [<f82d82d8>] resize_stripes+0x3d8/0x4a0 [raid456]
[54614.014470]  [<f82d8433>] check_reshape+0x93/0xc0 [raid456]
[54614.014477]  [<f82d849c>] raid5_check_reshape+0x3c/0xd0 [raid456]
[54614.014482]  [<c0483122>] update_array_info+0x282/0x360
[54614.014488]  [<c04883b2>] md_ioctl+0x162/0x710
[54614.014497]  [<c0233e59>] ? blkdev_aio_write+0x79/0x90
[54614.014499]  [<c0488250>] ? md_ioctl+0x0/0x710
[54614.014502]  [<c034264c>] __blkdev_driver_ioctl+0x2c/0x80
[54614.014506]  [<c020411a>] ? __mem_cgroup_commit_charge+0x4a/0xc0
[54614.014508]  [<c0342b49>] blkdev_ioctl+0x1e9/0x750
[54614.014512]  [<c01685e0>] ? autoremove_wake_function+0x0/0x50
[54614.014515]  [<c02f7b84>] ? security_file_permission+0x14/0x20
[54614.014517]  [<c02339f1>] block_ioctl+0x31/0x50
[54614.014519]  [<c02339c0>] ? block_ioctl+0x0/0x50
[54614.014522]  [<c0218911>] vfs_ioctl+0x21/0x90
[54614.014524]  [<c0218bf9>] do_vfs_ioctl+0x79/0x310
[54614.014526]  [<c0209e60>] ? do_sync_write+0x0/0x100
[54614.014528]  [<c0218ef7>] sys_ioctl+0x67/0x80
[54614.014531]  [<c01033ec>] syscall_call+0x7/0xb
[54614.014532] Mem-Info:
[54614.014533] DMA per-cpu:
[54614.014534] CPU    0: hi:    0, btch:   1 usd:   0
[54614.014536] CPU    1: hi:    0, btch:   1 usd:   0
[54614.014537] CPU    2: hi:    0, btch:   1 usd:   0
[54614.014538] CPU    3: hi:    0, btch:   1 usd:   0
[54614.014539] Normal per-cpu:
[54614.014540] CPU    0: hi:  186, btch:  31 usd:   0
[54614.014541] CPU    1: hi:  186, btch:  31 usd:   0
[54614.014543] CPU    2: hi:  186, btch:  31 usd:   0
[54614.014544] CPU    3: hi:  186, btch:  31 usd:   0
[54614.014545] HighMem per-cpu:
[54614.014546] CPU    0: hi:  186, btch:  31 usd:   0
[54614.014547] CPU    1: hi:  186, btch:  31 usd:   0
[54614.014548] CPU    2: hi:  186, btch:  31 usd:   0
[54614.014550] CPU    3: hi:  186, btch:  31 usd:   0

Upon reboot, the system comes back up and resyncs the array over a ~4hr period. I've does this loads of times in the past both on this box with this OS, and on another machine running Gentoo. The fact that it is hard locking tells me it's a kernel/driver bug. Has anyone else ran afoul of this? Can anyone shed any light on this?

 

[drescherjm]

The fact that it is hard locking tells me it's a kernel/driver bug.

I agree. I would just grow the array on a modern kernel and an updated mdadm via a livecd if possible.

BTW, Have you looked at the SMART on the drive you are adding? I usually do a 4 pass badblocks test on every drive before I try to add it to an array.

 

[parityboy]

Thanks for the reply. I came to the same conclusion about using a LiveCD, although it's a bit of a worry that a kernel and/or driver for an LTS release has such a bad bug.

The drive I'm adding is healthy according to SMART, even though the Write Error Rate is measured at 302; I think these Samsungs have poorer platter surfaces compared to other drives. I've never lost any data though.