mdadm frozen and doesn't realize array is dead/missing failed due to unplugged drives -

mdadm frozen and doesn't realize array is dead/missing failed due to unplugged drives

This was a surprising bug but I unplugged all drives for an array md127.  At first it was just 1 drive and mdadm seemed to notice this.  I unplugged the second drive taking the array offline but mdadm did not realize it was offline and still showed a non-existent disk as being part of it.  This created problems trying to unmount it or even to stop this array with mdadm freezing. 

As for how to fix it I can only think of making sure you are not in a mounted path of the array, unmount it and then manually stop the array but this didn't seem to work until the hung task kicked in.

mdadm --manage /dev/md127 --stop

 

The resulting kernel messages are what followed:

 

8586.047108] EXT4-fs error (device md127): __ext4_get_inode_loc:3932: inode #12058625: block 48234528: comm pool: unable to read itable block
[48586.047119] EXT4-fs (md127): previous I/O error to superblock detected
[48586.047207] Buffer I/O error on device md127, logical block 0
[48586.047216] lost page write due to I/O error on md127
[48586.248009] md: super_written gets error=-19, uptodate=0
[48594.634065] md: super_written gets error=-19, uptodate=0
[48594.634173] Buffer I/O error on device md127, logical block 91258880
[48594.634186] lost page write due to I/O error on md127
[48594.634221] JBD2: Error -5 detected when updating journal superblock for md127-8.
[48594.634286] Buffer I/O error on device md127, logical block 91258880
[48594.634299] lost page write due to I/O error on md127
[48594.634329] JBD2: Error -5 detected when updating journal superblock for md127-8.
[48594.635892] EXT4-fs (md127): previous I/O error to superblock detected
[48594.635974] Buffer I/O error on device md127, logical block 0
[48594.635984] lost page write due to I/O error on md127
[48594.837653] md: super_written gets error=-19, uptodate=0
[48851.459776] init: smbd main process (1204) killed by TERM signal
[48966.998219] Buffer I/O error on device md127, logical block 0
[48966.998241] Buffer I/O error on device md127, logical block 0
[48966.998267] Buffer I/O error on device md127, logical block 183110512
[48966.998277] Buffer I/O error on device md127, logical block 183110512
[48966.998295] Buffer I/O error on device md127, logical block 183110526
[48966.998304] Buffer I/O error on device md127, logical block 183110526
[48966.998315] Buffer I/O error on device md127, logical block 0
[48966.998329] Buffer I/O error on device md127, logical block 1
[48966.998337] Buffer I/O error on device md127, logical block 1
[48966.998357] Buffer I/O error on device md127, logical block 183110527
[49172.119413] md: md20 stopped.
[49172.193601] md: bind<sdf1>
[49184.251750] quiet_error: 124 callbacks suppressed
[49184.251764] Buffer I/O error on device md127, logical block 183110512
[49184.251778] Buffer I/O error on device md127, logical block 183110512
[49184.251799] Buffer I/O error on device md127, logical block 183110526
[49184.251808] Buffer I/O error on device md127, logical block 183110526
[49184.251823] Buffer I/O error on device md127, logical block 0
[49184.251831] Buffer I/O error on device md127, logical block 0
[49184.251846] Buffer I/O error on device md127, logical block 1
[49184.251854] Buffer I/O error on device md127, logical block 1
[49184.251869] Buffer I/O error on device md127, logical block 183110527
[49184.251877] Buffer I/O error on device md127, logical block 183110527
[49210.081588] INFO: task mdadm:16228 blocked for more than 120 seconds.
[49210.081605]       Tainted: P           OE 3.16.0-38-generic #52~14.04.1-Ubuntu
[49210.081610] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[49210.081616] mdadm           D ffff88080fd530c0     0 16228  15783 0x00000004
[49210.081627]  ffff88005539fc70 0000000000000082 ffff880804045180 ffff88005539ffd8
[49210.081635]  00000000000130c0 00000000000130c0 ffff880804721460 ffff88005539fd98
[49210.081642]  ffff88005539fda0 7fffffffffffffff ffff880804045180 0000000000000000
[49210.081649] Call Trace:
[49210.081668]  [<ffffffff817699f9>] schedule+0x29/0x70
[49210.081677]  [<ffffffff81768dd9>] schedule_timeout+0x229/0x2a0
[49210.081687]  [<ffffffff81087ef6>] ? __queue_work+0x136/0x320
[49210.081694]  [<ffffffff810881aa>] ? __queue_delayed_work+0xaa/0x1a0
[49210.081701]  [<ffffffff81088509>] ? try_to_grab_pending+0xa9/0x160
[49210.081709]  [<ffffffff8176a4f6>] wait_for_completion+0xa6/0x160
[49210.081718]  [<ffffffff810a1550>] ? wake_up_state+0x20/0x20
[49210.081728]  [<ffffffff811fc97f>] writeback_inodes_sb_nr+0x7f/0xb0
[49210.081735]  [<ffffffff811fc9d5>] writeback_inodes_sb+0x25/0x30
[49210.081745]  [<ffffffff812039cd>] sync_filesystem+0x2d/0xa0
[49210.081753]  [<ffffffff8120af24>] fsync_bdev+0x24/0x60
[49210.081764]  [<ffffffff81366183>] blkdev_ioctl+0x433/0x7b0
[49210.081772]  [<ffffffff8120a5c1>] block_ioctl+0x41/0x50
[49210.081779]  [<ffffffff811e7360>] do_vfs_ioctl+0x2e0/0x4c0
[49210.081788]  [<ffffffff811d96c5>] ? SYSC_newfstat+0x25/0x30
[49210.081795]  [<ffffffff811e75c1>] SyS_ioctl+0x81/0xa0
[49210.081804]  [<ffffffff8176da4d>] system_call_fastpath+0x1a/0x1f
[49210.866657] quiet_error: 260 callbacks suppressed
[49210.866670] Buffer I/O error on device md127, logical block 183110512
[49210.866685] Buffer I/O error on device md127, logical block 183110512
[49210.866705] Buffer I/O error on device md127, logical block 183110526
[49210.866715] Buffer I/O error on device md127, logical block 183110526
[49210.866729] Buffer I/O error on device md127, logical block 0
[49210.866738] Buffer I/O error on device md127, logical block 0
[49210.866753] Buffer I/O error on device md127, logical block 1
[49210.866762] Buffer I/O error on device md127, logical block 1
[49210.866777] Buffer I/O error on device md127, logical block 183110527
[49210.866786] Buffer I/O error on device md127, logical block 183110527
[49218.834738] md127: detected capacity change from 750020722688 to 0
[49218.834759] md: md127 stopped.
[49218.834776] md: unbind<sdc1>
[49218.839214] md: export_rdev(sdc1)
[49275.360612] init: smbd main process (16150) killed by TERM signal
[49330.104883] INFO: task mdadm:16228 blocked for more than 120 seconds.
[49330.104892]       Tainted: P           OE 3.16.0-38-generic #52~14.04.1-Ubuntu
[49330.104895] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[49330.104898] mdadm           D ffff88080fd530c0     0 16228  15783 0x00000004
[49330.104904]  ffff88005539fc70 0000000000000082 ffff880804045180 ffff88005539ffd8
[49330.104908]  00000000000130c0 00000000000130c0 ffff880804721460 ffff88005539fd98
[49330.104911]  ffff88005539fda0 7fffffffffffffff ffff880804045180 0000000000000000
[49330.104916] Call Trace:
[49330.104929]  [<ffffffff817699f9>] schedule+0x29/0x70
[49330.104933]  [<ffffffff81768dd9>] schedule_timeout+0x229/0x2a0
[49330.104939]  [<ffffffff81087ef6>] ? __queue_work+0x136/0x320
[49330.104942]  [<ffffffff810881aa>] ? __queue_delayed_work+0xaa/0x1a0
[49330.104946]  [<ffffffff81088509>] ? try_to_grab_pending+0xa9/0x160
[49330.104949]  [<ffffffff8176a4f6>] wait_for_completion+0xa6/0x160
[49330.104955]  [<ffffffff810a1550>] ? wake_up_state+0x20/0x20
[49330.104960]  [<ffffffff811fc97f>] writeback_inodes_sb_nr+0x7f/0xb0
[49330.104963]  [<ffffffff811fc9d5>] writeback_inodes_sb+0x25/0x30
[49330.104969]  [<ffffffff812039cd>] sync_filesystem+0x2d/0xa0
[49330.104973]  [<ffffffff8120af24>] fsync_bdev+0x24/0x60
[49330.104979]  [<ffffffff81366183>] blkdev_ioctl+0x433/0x7b0
[49330.104983]  [<ffffffff8120a5c1>] block_ioctl+0x41/0x50
[49330.104987]  [<ffffffff811e7360>] do_vfs_ioctl+0x2e0/0x4c0
[49330.104992]  [<ffffffff811d96c5>] ? SYSC_newfstat+0x25/0x30
[49330.104995]  [<ffffffff811e75c1>] SyS_ioctl+0x81/0xa0
[49330.104999]  [<ffffffff8176da4d>] system_call_fastpath+0x1a/0x1f
[49488.994434] md/raid10:md20: active with 1 out of 2 devices
[49488.994492] md20: detected capacity change from 0 to 750020722688
[49489.014456]  md20: unknown partition table
[49505.301904] EXT4-fs (md20): recovery complete
[49505.319010] EXT4-fs (md20): mounted filesystem with ordered data mode. Opts: (null)
[49524.680801] systemd-hostnamed[16729]: Warning: nss-myhostname is not installed. Changing the local hostname might make it unresolveable. Please install nss-myhostname!
[49570.151706] INFO: task mdadm:16228 blocked for more than 120 seconds.
[49570.151723]       Tainted: P           OE 3.16.0-38-generic #52~14.04.1-Ubuntu
[49570.151728] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[49570.151734] mdadm           D ffff88080fd530c0     0 16228  15783 0x00000004
[49570.151745]  ffff88005539fc80 0000000000000082 ffff880804045180 ffff88005539ffd8
[49570.151753]  00000000000130c0 00000000000130c0 ffff880804721460 ffff88005539fda8
[49570.151761]  ffff88005539fdb0 7fffffffffffffff ffff880804045180 0000000000000000
[49570.151768] Call Trace:
[49570.151787]  [<ffffffff817699f9>] schedule+0x29/0x70
[49570.151796]  [<ffffffff81768dd9>] schedule_timeout+0x229/0x2a0
[49570.151806]  [<ffffffff81087ef6>] ? __queue_work+0x136/0x320
[49570.151813]  [<ffffffff810881aa>] ? __queue_delayed_work+0xaa/0x1a0
[49570.151820]  [<ffffffff8108857d>] ? try_to_grab_pending+0x11d/0x160
[49570.151828]  [<ffffffff8176a4f6>] wait_for_completion+0xa6/0x160
[49570.151837]  [<ffffffff810a1550>] ? wake_up_state+0x20/0x20
[49570.151846]  [<ffffffff811fca85>] sync_inodes_sb+0xa5/0x1c0
[49570.151857]  [<ffffffff812039fb>] sync_filesystem+0x5b/0xa0
[49570.151865]  [<ffffffff8120af24>] fsync_bdev+0x24/0x60
[49570.151876]  [<ffffffff81366183>] blkdev_ioctl+0x433/0x7b0
[49570.151883]  [<ffffffff8120a5c1>] block_ioctl+0x41/0x50
[49570.151891]  [<ffffffff811e7360>] do_vfs_ioctl+0x2e0/0x4c0
[49570.151899]  [<ffffffff811d96c5>] ? SYSC_newfstat+0x25/0x30
[49570.151906]  [<ffffffff811e75c1>] SyS_ioctl+0x81/0xa0
[49570.151915]  [<ffffffff8176da4d>] system_call_fastpath+0x1a/0x1f
[49690.175082] INFO: task mdadm:16228 blocked for more than 120 seconds.
[49690.175099]       Tainted: P           OE 3.16.0-38-generic #52~14.04.1-Ubuntu
[49690.175104] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[49690.175110] mdadm           D ffff88080fd530c0     0 16228  15783 0x00000004
[49690.175121]  ffff88005539fc80 0000000000000082 ffff880804045180 ffff88005539ffd8
[49690.175129]  00000000000130c0 00000000000130c0 ffff880804721460 ffff88005539fda8
[49690.175136]  ffff88005539fdb0 7fffffffffffffff ffff880804045180 0000000000000000
[49690.175144] Call Trace:
[49690.175164]  [<ffffffff817699f9>] schedule+0x29/0x70
[49690.175172]  [<ffffffff81768dd9>] schedule_timeout+0x229/0x2a0
[49690.175182]  [<ffffffff81087ef6>] ? __queue_work+0x136/0x320
[49690.175190]  [<ffffffff810881aa>] ? __queue_delayed_work+0xaa/0x1a0
[49690.175197]  [<ffffffff8108857d>] ? try_to_grab_pending+0x11d/0x160
[49690.175205]  [<ffffffff8176a4f6>] wait_for_completion+0xa6/0x160
[49690.175214]  [<ffffffff810a1550>] ? wake_up_state+0x20/0x20
[49690.175224]  [<ffffffff811fca85>] sync_inodes_sb+0xa5/0x1c0
[49690.175234]  [<ffffffff812039fb>] sync_filesystem+0x5b/0xa0
[49690.175242]  [<ffffffff8120af24>] fsync_bdev+0x24/0x60
[49690.175253]  [<ffffffff81366183>] blkdev_ioctl+0x433/0x7b0
[49690.175261]  [<ffffffff8120a5c1>] block_ioctl+0x41/0x50
[49690.175268]  [<ffffffff811e7360>] do_vfs_ioctl+0x2e0/0x4c0
[49690.175277]  [<ffffffff811d96c5>] ? SYSC_newfstat+0x25/0x30
[49690.175284]  [<ffffffff811e75c1>] SyS_ioctl+0x81/0xa0
[49690.175293]  [<ffffffff8176da4d>] system_call_fastpath+0x1a/0x1f
[49810.198423] INFO: task mdadm:16228 blocked for more than 120 seconds.
[49810.198438]       Tainted: P           OE 3.16.0-38-generic #52~14.04.1-Ubuntu
[49810.198443] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[49810.198449] mdadm           D ffff88080fd530c0     0 16228  15783 0x00000004
[49810.198461]  ffff88005539fc80 0000000000000082 ffff880804045180 ffff88005539ffd8
[49810.198469]  00000000000130c0 00000000000130c0 ffff880804721460 ffff88005539fda8
[49810.198476]  ffff88005539fdb0 7fffffffffffffff ffff880804045180 0000000000000000
[49810.198483] Call Trace:
[49810.198503]  [<ffffffff817699f9>] schedule+0x29/0x70
[49810.198512]  [<ffffffff81768dd9>] schedule_timeout+0x229/0x2a0
[49810.198522]  [<ffffffff81087ef6>] ? __queue_work+0x136/0x320
[49810.198530]  [<ffffffff810881aa>] ? __queue_delayed_work+0xaa/0x1a0
[49810.198537]  [<ffffffff8108857d>] ? try_to_grab_pending+0x11d/0x160
[49810.198545]  [<ffffffff8176a4f6>] wait_for_completion+0xa6/0x160
[49810.198554]  [<ffffffff810a1550>] ? wake_up_state+0x20/0x20
[49810.198563]  [<ffffffff811fca85>] sync_inodes_sb+0xa5/0x1c0
[49810.198574]  [<ffffffff812039fb>] sync_filesystem+0x5b/0xa0
[49810.198582]  [<ffffffff8120af24>] fsync_bdev+0x24/0x60
[49810.198593]  [<ffffffff81366183>] blkdev_ioctl+0x433/0x7b0
[49810.198601]  [<ffffffff8120a5c1>] block_ioctl+0x41/0x50
[49810.198608]  [<ffffffff811e7360>] do_vfs_ioctl+0x2e0/0x4c0
[49810.198617]  [<ffffffff811d96c5>] ? SYSC_newfstat+0x25/0x30
[49810.198624]  [<ffffffff811e75c1>] SyS_ioctl+0x81/0xa0
[49810.198638]  [<ffffffff8176da4d>] system_call_fastpath+0x1a/0x1f
 


  • Linux how to view video card make and exact model
  • Authentication refused: bad ownership or modes for directory /home/user SSH Public Key Authentication Failed Solution
  • How to Update Linux /usr/share/misc/pci.ids
  • pcimodules and lspci not working alternative solution
  • How to disable Google Fonts in Wordpress
  • Unable to load dynamic library /usr/lib64/php/modules/php_openssl
  • mysqld in Linux hacked
  • W: GPG error: http://archive.debian.org squeeze Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY AED4B06F473041FA NO_PUBKEY 64481591B98321F9
  • cannot mount kvm ntfs image
  • h264 DVR security camera footage cannot be played
  • dhcpd.conf how to secure so only known and allowed clients will be given dhcpd IP address leases
  • Thunderbird E-mail List Blank White but e-mails still clickable and viewable
  • css responsive images
  • responsive table without changing much code solution
  • yum how to install old obsolete packages
  • PHP Howto Store Value of Included File Output Into Variable
  • PHP Migration from 5.3 to 5.4+ and dealing with deprecated functions
  • ffmpeg vidstab to stabilize video
  • userdel user userdel: cannot lock /etc/passwd; try again later.
  • mdadm how to mount inactive array