This was a surprising bug but I unplugged all drives for an array md127.  At first it was just 1 drive and mdadm seemed to notice this.  I unplugged the second drive taking the array offline but mdadm did not realize it was offline and still showed a non-existent disk as being part of it.  This created problems trying to unmount it or even to stop this array with mdadm freezing. 

As for how to fix it I can only think of making sure you are not in a mounted path of the array, unmount it and then manually stop the array but this didn't seem to work until the hung task kicked in.

mdadm --manage /dev/md127 --stop

The resulting kernel messages are what followed:

8586.047108] EXT4-fs error (device md127): __ext4_get_inode_loc:3932: inode #12058625: block 48234528: comm pool: unable to read itable block
[48586.047119] EXT4-fs (md127): previous I/O error to superblock detected
[48586.047207] Buffer I/O error on device md127, logical block 0
[48586.047216] lost page write due to I/O error on md127
[48586.248009] md: super_written gets error=-19, uptodate=0
[48594.634065] md: super_written gets error=-19, uptodate=0
[48594.634173] Buffer I/O error on device md127, logical block 91258880
[48594.634186] lost page write due to I/O error on md127
[48594.634221] JBD2: Error -5 detected when updating journal superblock for md127-8.
[48594.634286] Buffer I/O error on device md127, logical block 91258880
[48594.634299] lost page write due to I/O error on md127
[48594.634329] JBD2: Error -5 detected when updating journal superblock for md127-8.
[48594.635892] EXT4-fs (md127): previous I/O error to superblock detected
[48594.635974] Buffer I/O error on device md127, logical block 0
[48594.635984] lost page write due to I/O error on md127
[48594.837653] md: super_written gets error=-19, uptodate=0
[48851.459776] init: smbd main process (1204) killed by TERM signal
[48966.998219] Buffer I/O error on device md127, logical block 0
[48966.998241] Buffer I/O error on device md127, logical block 0
[48966.998267] Buffer I/O error on device md127, logical block 183110512
[48966.998277] Buffer I/O error on device md127, logical block 183110512
[48966.998295] Buffer I/O error on device md127, logical block 183110526
[48966.998304] Buffer I/O error on device md127, logical block 183110526
[48966.998315] Buffer I/O error on device md127, logical block 0
[48966.998329] Buffer I/O error on device md127, logical block 1
[48966.998337] Buffer I/O error on device md127, logical block 1
[48966.998357] Buffer I/O error on device md127, logical block 183110527
[49172.119413] md: md20 stopped.
[49172.193601] md: bind<sdf1>
[49184.251750] quiet_error: 124 callbacks suppressed
[49184.251764] Buffer I/O error on device md127, logical block 183110512
[49184.251778] Buffer I/O error on device md127, logical block 183110512
[49184.251799] Buffer I/O error on device md127, logical block 183110526
[49184.251808] Buffer I/O error on device md127, logical block 183110526
[49184.251823] Buffer I/O error on device md127, logical block 0
[49184.251831] Buffer I/O error on device md127, logical block 0
[49184.251846] Buffer I/O error on device md127, logical block 1
[49184.251854] Buffer I/O error on device md127, logical block 1
[49184.251869] Buffer I/O error on device md127, logical block 183110527
[49184.251877] Buffer I/O error on device md127, logical block 183110527
[49210.081588] INFO: task mdadm:16228 blocked for more than 120 seconds.
[49210.081605]       Tainted: P           OE 3.16.0-38-generic #52~14.04.1-Ubuntu
[49210.081610] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[49210.081616] mdadm           D ffff88080fd530c0     0 16228  15783 0x00000004
[49210.081627]  ffff88005539fc70 0000000000000082 ffff880804045180 ffff88005539ffd8
[49210.081635]  00000000000130c0 00000000000130c0 ffff880804721460 ffff88005539fd98
[49210.081642]  ffff88005539fda0 7fffffffffffffff ffff880804045180 0000000000000000
[49210.081649] Call Trace:
[49210.081668]  [<ffffffff817699f9>] schedule+0x29/0x70
[49210.081677]  [<ffffffff81768dd9>] schedule_timeout+0x229/0x2a0
[49210.081687]  [<ffffffff81087ef6>] ? __queue_work+0x136/0x320
[49210.081694]  [<ffffffff810881aa>] ? __queue_delayed_work+0xaa/0x1a0
[49210.081701]  [<ffffffff81088509>] ? try_to_grab_pending+0xa9/0x160
[49210.081709]  [<ffffffff8176a4f6>] wait_for_completion+0xa6/0x160
[49210.081718]  [<ffffffff810a1550>] ? wake_up_state+0x20/0x20
[49210.081728]  [<ffffffff811fc97f>] writeback_inodes_sb_nr+0x7f/0xb0
[49210.081735]  [<ffffffff811fc9d5>] writeback_inodes_sb+0x25/0x30
[49210.081745]  [<ffffffff812039cd>] sync_filesystem+0x2d/0xa0
[49210.081753]  [<ffffffff8120af24>] fsync_bdev+0x24/0x60
[49210.081764]  [<ffffffff81366183>] blkdev_ioctl+0x433/0x7b0
[49210.081772]  [<ffffffff8120a5c1>] block_ioctl+0x41/0x50
[49210.081779]  [<ffffffff811e7360>] do_vfs_ioctl+0x2e0/0x4c0
[49210.081788]  [<ffffffff811d96c5>] ? SYSC_newfstat+0x25/0x30
[49210.081795]  [<ffffffff811e75c1>] SyS_ioctl+0x81/0xa0
[49210.081804]  [<ffffffff8176da4d>] system_call_fastpath+0x1a/0x1f
[49210.866657] quiet_error: 260 callbacks suppressed
[49210.866670] Buffer I/O error on device md127, logical block 183110512
[49210.866685] Buffer I/O error on device md127, logical block 183110512
[49210.866705] Buffer I/O error on device md127, logical block 183110526
[49210.866715] Buffer I/O error on device md127, logical block 183110526
[49210.866729] Buffer I/O error on device md127, logical block 0
[49210.866738] Buffer I/O error on device md127, logical block 0
[49210.866753] Buffer I/O error on device md127, logical block 1
[49210.866762] Buffer I/O error on device md127, logical block 1
[49210.866777] Buffer I/O error on device md127, logical block 183110527
[49210.866786] Buffer I/O error on device md127, logical block 183110527
[49218.834738] md127: detected capacity change from 750020722688 to 0
[49218.834759] md: md127 stopped.
[49218.834776] md: unbind<sdc1>
[49218.839214] md: export_rdev(sdc1)
[49275.360612] init: smbd main process (16150) killed by TERM signal
[49330.104883] INFO: task mdadm:16228 blocked for more than 120 seconds.
[49330.104892]       Tainted: P           OE 3.16.0-38-generic #52~14.04.1-Ubuntu
[49330.104895] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[49330.104898] mdadm           D ffff88080fd530c0     0 16228  15783 0x00000004
[49330.104904]  ffff88005539fc70 0000000000000082 ffff880804045180 ffff88005539ffd8
[49330.104908]  00000000000130c0 00000000000130c0 ffff880804721460 ffff88005539fd98
[49330.104911]  ffff88005539fda0 7fffffffffffffff ffff880804045180 0000000000000000
[49330.104916] Call Trace:
[49330.104929]  [<ffffffff817699f9>] schedule+0x29/0x70
[49330.104933]  [<ffffffff81768dd9>] schedule_timeout+0x229/0x2a0
[49330.104939]  [<ffffffff81087ef6>] ? __queue_work+0x136/0x320
[49330.104942]  [<ffffffff810881aa>] ? __queue_delayed_work+0xaa/0x1a0
[49330.104946]  [<ffffffff81088509>] ? try_to_grab_pending+0xa9/0x160
[49330.104949]  [<ffffffff8176a4f6>] wait_for_completion+0xa6/0x160
[49330.104955]  [<ffffffff810a1550>] ? wake_up_state+0x20/0x20
[49330.104960]  [<ffffffff811fc97f>] writeback_inodes_sb_nr+0x7f/0xb0
[49330.104963]  [<ffffffff811fc9d5>] writeback_inodes_sb+0x25/0x30
[49330.104969]  [<ffffffff812039cd>] sync_filesystem+0x2d/0xa0
[49330.104973]  [<ffffffff8120af24>] fsync_bdev+0x24/0x60
[49330.104979]  [<ffffffff81366183>] blkdev_ioctl+0x433/0x7b0
[49330.104983]  [<ffffffff8120a5c1>] block_ioctl+0x41/0x50
[49330.104987]  [<ffffffff811e7360>] do_vfs_ioctl+0x2e0/0x4c0
[49330.104992]  [<ffffffff811d96c5>] ? SYSC_newfstat+0x25/0x30
[49330.104995]  [<ffffffff811e75c1>] SyS_ioctl+0x81/0xa0
[49330.104999]  [<ffffffff8176da4d>] system_call_fastpath+0x1a/0x1f
[49488.994434] md/raid10:md20: active with 1 out of 2 devices
[49488.994492] md20: detected capacity change from 0 to 750020722688
[49489.014456]  md20: unknown partition table
[49505.301904] EXT4-fs (md20): recovery complete
[49505.319010] EXT4-fs (md20): mounted filesystem with ordered data mode. Opts: (null)
[49524.680801] systemd-hostnamed[16729]: Warning: nss-myhostname is not installed. Changing the local hostname might make it unresolveable. Please install nss-myhostname!
[49570.151706] INFO: task mdadm:16228 blocked for more than 120 seconds.
[49570.151723]       Tainted: P           OE 3.16.0-38-generic #52~14.04.1-Ubuntu
[49570.151728] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[49570.151734] mdadm           D ffff88080fd530c0     0 16228  15783 0x00000004
[49570.151745]  ffff88005539fc80 0000000000000082 ffff880804045180 ffff88005539ffd8
[49570.151753]  00000000000130c0 00000000000130c0 ffff880804721460 ffff88005539fda8
[49570.151761]  ffff88005539fdb0 7fffffffffffffff ffff880804045180 0000000000000000
[49570.151768] Call Trace:
[49570.151787]  [<ffffffff817699f9>] schedule+0x29/0x70
[49570.151796]  [<ffffffff81768dd9>] schedule_timeout+0x229/0x2a0
[49570.151806]  [<ffffffff81087ef6>] ? __queue_work+0x136/0x320
[49570.151813]  [<ffffffff810881aa>] ? __queue_delayed_work+0xaa/0x1a0
[49570.151820]  [<ffffffff8108857d>] ? try_to_grab_pending+0x11d/0x160
[49570.151828]  [<ffffffff8176a4f6>] wait_for_completion+0xa6/0x160
[49570.151837]  [<ffffffff810a1550>] ? wake_up_state+0x20/0x20
[49570.151846]  [<ffffffff811fca85>] sync_inodes_sb+0xa5/0x1c0
[49570.151857]  [<ffffffff812039fb>] sync_filesystem+0x5b/0xa0
[49570.151865]  [<ffffffff8120af24>] fsync_bdev+0x24/0x60
[49570.151876]  [<ffffffff81366183>] blkdev_ioctl+0x433/0x7b0
[49570.151883]  [<ffffffff8120a5c1>] block_ioctl+0x41/0x50
[49570.151891]  [<ffffffff811e7360>] do_vfs_ioctl+0x2e0/0x4c0
[49570.151899]  [<ffffffff811d96c5>] ? SYSC_newfstat+0x25/0x30
[49570.151906]  [<ffffffff811e75c1>] SyS_ioctl+0x81/0xa0
[49570.151915]  [<ffffffff8176da4d>] system_call_fastpath+0x1a/0x1f
[49690.175082] INFO: task mdadm:16228 blocked for more than 120 seconds.
[49690.175099]       Tainted: P           OE 3.16.0-38-generic #52~14.04.1-Ubuntu
[49690.175104] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[49690.175110] mdadm           D ffff88080fd530c0     0 16228  15783 0x00000004
[49690.175121]  ffff88005539fc80 0000000000000082 ffff880804045180 ffff88005539ffd8
[49690.175129]  00000000000130c0 00000000000130c0 ffff880804721460 ffff88005539fda8
[49690.175136]  ffff88005539fdb0 7fffffffffffffff ffff880804045180 0000000000000000
[49690.175144] Call Trace:
[49690.175164]  [<ffffffff817699f9>] schedule+0x29/0x70
[49690.175172]  [<ffffffff81768dd9>] schedule_timeout+0x229/0x2a0
[49690.175182]  [<ffffffff81087ef6>] ? __queue_work+0x136/0x320
[49690.175190]  [<ffffffff810881aa>] ? __queue_delayed_work+0xaa/0x1a0
[49690.175197]  [<ffffffff8108857d>] ? try_to_grab_pending+0x11d/0x160
[49690.175205]  [<ffffffff8176a4f6>] wait_for_completion+0xa6/0x160
[49690.175214]  [<ffffffff810a1550>] ? wake_up_state+0x20/0x20
[49690.175224]  [<ffffffff811fca85>] sync_inodes_sb+0xa5/0x1c0
[49690.175234]  [<ffffffff812039fb>] sync_filesystem+0x5b/0xa0
[49690.175242]  [<ffffffff8120af24>] fsync_bdev+0x24/0x60
[49690.175253]  [<ffffffff81366183>] blkdev_ioctl+0x433/0x7b0
[49690.175261]  [<ffffffff8120a5c1>] block_ioctl+0x41/0x50
[49690.175268]  [<ffffffff811e7360>] do_vfs_ioctl+0x2e0/0x4c0
[49690.175277]  [<ffffffff811d96c5>] ? SYSC_newfstat+0x25/0x30
[49690.175284]  [<ffffffff811e75c1>] SyS_ioctl+0x81/0xa0
[49690.175293]  [<ffffffff8176da4d>] system_call_fastpath+0x1a/0x1f
[49810.198423] INFO: task mdadm:16228 blocked for more than 120 seconds.
[49810.198438]       Tainted: P           OE 3.16.0-38-generic #52~14.04.1-Ubuntu
[49810.198443] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[49810.198449] mdadm           D ffff88080fd530c0     0 16228  15783 0x00000004
[49810.198461]  ffff88005539fc80 0000000000000082 ffff880804045180 ffff88005539ffd8
[49810.198469]  00000000000130c0 00000000000130c0 ffff880804721460 ffff88005539fda8
[49810.198476]  ffff88005539fdb0 7fffffffffffffff ffff880804045180 0000000000000000
[49810.198483] Call Trace:
[49810.198503]  [<ffffffff817699f9>] schedule+0x29/0x70
[49810.198512]  [<ffffffff81768dd9>] schedule_timeout+0x229/0x2a0
[49810.198522]  [<ffffffff81087ef6>] ? __queue_work+0x136/0x320
[49810.198530]  [<ffffffff810881aa>] ? __queue_delayed_work+0xaa/0x1a0
[49810.198537]  [<ffffffff8108857d>] ? try_to_grab_pending+0x11d/0x160
[49810.198545]  [<ffffffff8176a4f6>] wait_for_completion+0xa6/0x160
[49810.198554]  [<ffffffff810a1550>] ? wake_up_state+0x20/0x20
[49810.198563]  [<ffffffff811fca85>] sync_inodes_sb+0xa5/0x1c0
[49810.198574]  [<ffffffff812039fb>] sync_filesystem+0x5b/0xa0
[49810.198582]  [<ffffffff8120af24>] fsync_bdev+0x24/0x60
[49810.198593]  [<ffffffff81366183>] blkdev_ioctl+0x433/0x7b0
[49810.198601]  [<ffffffff8120a5c1>] block_ioctl+0x41/0x50
[49810.198608]  [<ffffffff811e7360>] do_vfs_ioctl+0x2e0/0x4c0
[49810.198617]  [<ffffffff811d96c5>] ? SYSC_newfstat+0x25/0x30
[49810.198624]  [<ffffffff811e75c1>] SyS_ioctl+0x81/0xa0
[49810.198638]  [<ffffffff8176da4d>] system_call_fastpath+0x1a/0x1f
 

mdadm, doesn, array, unplugged, drivesthis, surprising, md, offline, existent, disk, unmount, mounted, manually, didn, task, dev, resulting, kernel, ext, fs, __ext, _get_inode_loc, inode, comm, unable, itable, previous, superblock, detected, buffer, super_written, uptodate, jbd, updating, init, smbd, bind, sdf, quiet_error, callbacks, suppressed, info, blocked, tainted, oe, generic, ubuntu, quot, echo, proc, sys, hung_task_timeout_secs, disables, ffff, fd, fc, ffd, fda, fffffffffffffff, ffffffff, dd, schedule_timeout, ef, __queue_work, aa, __queue_delayed_work, xaa, try_to_grab_pending, xa, wait_for_completion, wake_up_state, writeback_inodes_sb_nr, xb, writeback_inodes_sb, sync_filesystem, af, fsync_bdev, blkdev_ioctl, block_ioctl, do_vfs_ioctl, sysc_newfstat, sys_ioctl, system_call_fastpath, capacity, unbind, sdc, export_rdev, raid, active, devices, partition, filesystem, mode, opts, null, systemd, hostnamed, nss, myhostname, installed, hostname, unresolveable, install, fdb, fca, sync_inodes_sb, fb,

Latest Articles