mdadm frozen and doesn't realize array is dead/missing failed due to unplugged drives -

mdadm frozen and doesn't realize array is dead/missing failed due to unplugged drives

This was a surprising bug but I unplugged all drives for an array md127.  At first it was just 1 drive and mdadm seemed to notice this.  I unplugged the second drive taking the array offline but mdadm did not realize it was offline and still showed a non-existent disk as being part of it.  This created problems trying to unmount it or even to stop this array with mdadm freezing. 

As for how to fix it I can only think of making sure you are not in a mounted path of the array, unmount it and then manually stop the array but this didn't seem to work until the hung task kicked in.

mdadm --manage /dev/md127 --stop

 

The resulting kernel messages are what followed:

 

8586.047108] EXT4-fs error (device md127): __ext4_get_inode_loc:3932: inode #12058625: block 48234528: comm pool: unable to read itable block
[48586.047119] EXT4-fs (md127): previous I/O error to superblock detected
[48586.047207] Buffer I/O error on device md127, logical block 0
[48586.047216] lost page write due to I/O error on md127
[48586.248009] md: super_written gets error=-19, uptodate=0
[48594.634065] md: super_written gets error=-19, uptodate=0
[48594.634173] Buffer I/O error on device md127, logical block 91258880
[48594.634186] lost page write due to I/O error on md127
[48594.634221] JBD2: Error -5 detected when updating journal superblock for md127-8.
[48594.634286] Buffer I/O error on device md127, logical block 91258880
[48594.634299] lost page write due to I/O error on md127
[48594.634329] JBD2: Error -5 detected when updating journal superblock for md127-8.
[48594.635892] EXT4-fs (md127): previous I/O error to superblock detected
[48594.635974] Buffer I/O error on device md127, logical block 0
[48594.635984] lost page write due to I/O error on md127
[48594.837653] md: super_written gets error=-19, uptodate=0
[48851.459776] init: smbd main process (1204) killed by TERM signal
[48966.998219] Buffer I/O error on device md127, logical block 0
[48966.998241] Buffer I/O error on device md127, logical block 0
[48966.998267] Buffer I/O error on device md127, logical block 183110512
[48966.998277] Buffer I/O error on device md127, logical block 183110512
[48966.998295] Buffer I/O error on device md127, logical block 183110526
[48966.998304] Buffer I/O error on device md127, logical block 183110526
[48966.998315] Buffer I/O error on device md127, logical block 0
[48966.998329] Buffer I/O error on device md127, logical block 1
[48966.998337] Buffer I/O error on device md127, logical block 1
[48966.998357] Buffer I/O error on device md127, logical block 183110527
[49172.119413] md: md20 stopped.
[49172.193601] md: bind<sdf1>
[49184.251750] quiet_error: 124 callbacks suppressed
[49184.251764] Buffer I/O error on device md127, logical block 183110512
[49184.251778] Buffer I/O error on device md127, logical block 183110512
[49184.251799] Buffer I/O error on device md127, logical block 183110526
[49184.251808] Buffer I/O error on device md127, logical block 183110526
[49184.251823] Buffer I/O error on device md127, logical block 0
[49184.251831] Buffer I/O error on device md127, logical block 0
[49184.251846] Buffer I/O error on device md127, logical block 1
[49184.251854] Buffer I/O error on device md127, logical block 1
[49184.251869] Buffer I/O error on device md127, logical block 183110527
[49184.251877] Buffer I/O error on device md127, logical block 183110527
[49210.081588] INFO: task mdadm:16228 blocked for more than 120 seconds.
[49210.081605]       Tainted: P           OE 3.16.0-38-generic #52~14.04.1-Ubuntu
[49210.081610] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[49210.081616] mdadm           D ffff88080fd530c0     0 16228  15783 0x00000004
[49210.081627]  ffff88005539fc70 0000000000000082 ffff880804045180 ffff88005539ffd8
[49210.081635]  00000000000130c0 00000000000130c0 ffff880804721460 ffff88005539fd98
[49210.081642]  ffff88005539fda0 7fffffffffffffff ffff880804045180 0000000000000000
[49210.081649] Call Trace:
[49210.081668]  [<ffffffff817699f9>] schedule+0x29/0x70
[49210.081677]  [<ffffffff81768dd9>] schedule_timeout+0x229/0x2a0
[49210.081687]  [<ffffffff81087ef6>] ? __queue_work+0x136/0x320
[49210.081694]  [<ffffffff810881aa>] ? __queue_delayed_work+0xaa/0x1a0
[49210.081701]  [<ffffffff81088509>] ? try_to_grab_pending+0xa9/0x160
[49210.081709]  [<ffffffff8176a4f6>] wait_for_completion+0xa6/0x160
[49210.081718]  [<ffffffff810a1550>] ? wake_up_state+0x20/0x20
[49210.081728]  [<ffffffff811fc97f>] writeback_inodes_sb_nr+0x7f/0xb0
[49210.081735]  [<ffffffff811fc9d5>] writeback_inodes_sb+0x25/0x30
[49210.081745]  [<ffffffff812039cd>] sync_filesystem+0x2d/0xa0
[49210.081753]  [<ffffffff8120af24>] fsync_bdev+0x24/0x60
[49210.081764]  [<ffffffff81366183>] blkdev_ioctl+0x433/0x7b0
[49210.081772]  [<ffffffff8120a5c1>] block_ioctl+0x41/0x50
[49210.081779]  [<ffffffff811e7360>] do_vfs_ioctl+0x2e0/0x4c0
[49210.081788]  [<ffffffff811d96c5>] ? SYSC_newfstat+0x25/0x30
[49210.081795]  [<ffffffff811e75c1>] SyS_ioctl+0x81/0xa0
[49210.081804]  [<ffffffff8176da4d>] system_call_fastpath+0x1a/0x1f
[49210.866657] quiet_error: 260 callbacks suppressed
[49210.866670] Buffer I/O error on device md127, logical block 183110512
[49210.866685] Buffer I/O error on device md127, logical block 183110512
[49210.866705] Buffer I/O error on device md127, logical block 183110526
[49210.866715] Buffer I/O error on device md127, logical block 183110526
[49210.866729] Buffer I/O error on device md127, logical block 0
[49210.866738] Buffer I/O error on device md127, logical block 0
[49210.866753] Buffer I/O error on device md127, logical block 1
[49210.866762] Buffer I/O error on device md127, logical block 1
[49210.866777] Buffer I/O error on device md127, logical block 183110527
[49210.866786] Buffer I/O error on device md127, logical block 183110527
[49218.834738] md127: detected capacity change from 750020722688 to 0
[49218.834759] md: md127 stopped.
[49218.834776] md: unbind<sdc1>
[49218.839214] md: export_rdev(sdc1)
[49275.360612] init: smbd main process (16150) killed by TERM signal
[49330.104883] INFO: task mdadm:16228 blocked for more than 120 seconds.
[49330.104892]       Tainted: P           OE 3.16.0-38-generic #52~14.04.1-Ubuntu
[49330.104895] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[49330.104898] mdadm           D ffff88080fd530c0     0 16228  15783 0x00000004
[49330.104904]  ffff88005539fc70 0000000000000082 ffff880804045180 ffff88005539ffd8
[49330.104908]  00000000000130c0 00000000000130c0 ffff880804721460 ffff88005539fd98
[49330.104911]  ffff88005539fda0 7fffffffffffffff ffff880804045180 0000000000000000
[49330.104916] Call Trace:
[49330.104929]  [<ffffffff817699f9>] schedule+0x29/0x70
[49330.104933]  [<ffffffff81768dd9>] schedule_timeout+0x229/0x2a0
[49330.104939]  [<ffffffff81087ef6>] ? __queue_work+0x136/0x320
[49330.104942]  [<ffffffff810881aa>] ? __queue_delayed_work+0xaa/0x1a0
[49330.104946]  [<ffffffff81088509>] ? try_to_grab_pending+0xa9/0x160
[49330.104949]  [<ffffffff8176a4f6>] wait_for_completion+0xa6/0x160
[49330.104955]  [<ffffffff810a1550>] ? wake_up_state+0x20/0x20
[49330.104960]  [<ffffffff811fc97f>] writeback_inodes_sb_nr+0x7f/0xb0
[49330.104963]  [<ffffffff811fc9d5>] writeback_inodes_sb+0x25/0x30
[49330.104969]  [<ffffffff812039cd>] sync_filesystem+0x2d/0xa0
[49330.104973]  [<ffffffff8120af24>] fsync_bdev+0x24/0x60
[49330.104979]  [<ffffffff81366183>] blkdev_ioctl+0x433/0x7b0
[49330.104983]  [<ffffffff8120a5c1>] block_ioctl+0x41/0x50
[49330.104987]  [<ffffffff811e7360>] do_vfs_ioctl+0x2e0/0x4c0
[49330.104992]  [<ffffffff811d96c5>] ? SYSC_newfstat+0x25/0x30
[49330.104995]  [<ffffffff811e75c1>] SyS_ioctl+0x81/0xa0
[49330.104999]  [<ffffffff8176da4d>] system_call_fastpath+0x1a/0x1f
[49488.994434] md/raid10:md20: active with 1 out of 2 devices
[49488.994492] md20: detected capacity change from 0 to 750020722688
[49489.014456]  md20: unknown partition table
[49505.301904] EXT4-fs (md20): recovery complete
[49505.319010] EXT4-fs (md20): mounted filesystem with ordered data mode. Opts: (null)
[49524.680801] systemd-hostnamed[16729]: Warning: nss-myhostname is not installed. Changing the local hostname might make it unresolveable. Please install nss-myhostname!
[49570.151706] INFO: task mdadm:16228 blocked for more than 120 seconds.
[49570.151723]       Tainted: P           OE 3.16.0-38-generic #52~14.04.1-Ubuntu
[49570.151728] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[49570.151734] mdadm           D ffff88080fd530c0     0 16228  15783 0x00000004
[49570.151745]  ffff88005539fc80 0000000000000082 ffff880804045180 ffff88005539ffd8
[49570.151753]  00000000000130c0 00000000000130c0 ffff880804721460 ffff88005539fda8
[49570.151761]  ffff88005539fdb0 7fffffffffffffff ffff880804045180 0000000000000000
[49570.151768] Call Trace:
[49570.151787]  [<ffffffff817699f9>] schedule+0x29/0x70
[49570.151796]  [<ffffffff81768dd9>] schedule_timeout+0x229/0x2a0
[49570.151806]  [<ffffffff81087ef6>] ? __queue_work+0x136/0x320
[49570.151813]  [<ffffffff810881aa>] ? __queue_delayed_work+0xaa/0x1a0
[49570.151820]  [<ffffffff8108857d>] ? try_to_grab_pending+0x11d/0x160
[49570.151828]  [<ffffffff8176a4f6>] wait_for_completion+0xa6/0x160
[49570.151837]  [<ffffffff810a1550>] ? wake_up_state+0x20/0x20
[49570.151846]  [<ffffffff811fca85>] sync_inodes_sb+0xa5/0x1c0
[49570.151857]  [<ffffffff812039fb>] sync_filesystem+0x5b/0xa0
[49570.151865]  [<ffffffff8120af24>] fsync_bdev+0x24/0x60
[49570.151876]  [<ffffffff81366183>] blkdev_ioctl+0x433/0x7b0
[49570.151883]  [<ffffffff8120a5c1>] block_ioctl+0x41/0x50
[49570.151891]  [<ffffffff811e7360>] do_vfs_ioctl+0x2e0/0x4c0
[49570.151899]  [<ffffffff811d96c5>] ? SYSC_newfstat+0x25/0x30
[49570.151906]  [<ffffffff811e75c1>] SyS_ioctl+0x81/0xa0
[49570.151915]  [<ffffffff8176da4d>] system_call_fastpath+0x1a/0x1f
[49690.175082] INFO: task mdadm:16228 blocked for more than 120 seconds.
[49690.175099]       Tainted: P           OE 3.16.0-38-generic #52~14.04.1-Ubuntu
[49690.175104] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[49690.175110] mdadm           D ffff88080fd530c0     0 16228  15783 0x00000004
[49690.175121]  ffff88005539fc80 0000000000000082 ffff880804045180 ffff88005539ffd8
[49690.175129]  00000000000130c0 00000000000130c0 ffff880804721460 ffff88005539fda8
[49690.175136]  ffff88005539fdb0 7fffffffffffffff ffff880804045180 0000000000000000
[49690.175144] Call Trace:
[49690.175164]  [<ffffffff817699f9>] schedule+0x29/0x70
[49690.175172]  [<ffffffff81768dd9>] schedule_timeout+0x229/0x2a0
[49690.175182]  [<ffffffff81087ef6>] ? __queue_work+0x136/0x320
[49690.175190]  [<ffffffff810881aa>] ? __queue_delayed_work+0xaa/0x1a0
[49690.175197]  [<ffffffff8108857d>] ? try_to_grab_pending+0x11d/0x160
[49690.175205]  [<ffffffff8176a4f6>] wait_for_completion+0xa6/0x160
[49690.175214]  [<ffffffff810a1550>] ? wake_up_state+0x20/0x20
[49690.175224]  [<ffffffff811fca85>] sync_inodes_sb+0xa5/0x1c0
[49690.175234]  [<ffffffff812039fb>] sync_filesystem+0x5b/0xa0
[49690.175242]  [<ffffffff8120af24>] fsync_bdev+0x24/0x60
[49690.175253]  [<ffffffff81366183>] blkdev_ioctl+0x433/0x7b0
[49690.175261]  [<ffffffff8120a5c1>] block_ioctl+0x41/0x50
[49690.175268]  [<ffffffff811e7360>] do_vfs_ioctl+0x2e0/0x4c0
[49690.175277]  [<ffffffff811d96c5>] ? SYSC_newfstat+0x25/0x30
[49690.175284]  [<ffffffff811e75c1>] SyS_ioctl+0x81/0xa0
[49690.175293]  [<ffffffff8176da4d>] system_call_fastpath+0x1a/0x1f
[49810.198423] INFO: task mdadm:16228 blocked for more than 120 seconds.
[49810.198438]       Tainted: P           OE 3.16.0-38-generic #52~14.04.1-Ubuntu
[49810.198443] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[49810.198449] mdadm           D ffff88080fd530c0     0 16228  15783 0x00000004
[49810.198461]  ffff88005539fc80 0000000000000082 ffff880804045180 ffff88005539ffd8
[49810.198469]  00000000000130c0 00000000000130c0 ffff880804721460 ffff88005539fda8
[49810.198476]  ffff88005539fdb0 7fffffffffffffff ffff880804045180 0000000000000000
[49810.198483] Call Trace:
[49810.198503]  [<ffffffff817699f9>] schedule+0x29/0x70
[49810.198512]  [<ffffffff81768dd9>] schedule_timeout+0x229/0x2a0
[49810.198522]  [<ffffffff81087ef6>] ? __queue_work+0x136/0x320
[49810.198530]  [<ffffffff810881aa>] ? __queue_delayed_work+0xaa/0x1a0
[49810.198537]  [<ffffffff8108857d>] ? try_to_grab_pending+0x11d/0x160
[49810.198545]  [<ffffffff8176a4f6>] wait_for_completion+0xa6/0x160
[49810.198554]  [<ffffffff810a1550>] ? wake_up_state+0x20/0x20
[49810.198563]  [<ffffffff811fca85>] sync_inodes_sb+0xa5/0x1c0
[49810.198574]  [<ffffffff812039fb>] sync_filesystem+0x5b/0xa0
[49810.198582]  [<ffffffff8120af24>] fsync_bdev+0x24/0x60
[49810.198593]  [<ffffffff81366183>] blkdev_ioctl+0x433/0x7b0
[49810.198601]  [<ffffffff8120a5c1>] block_ioctl+0x41/0x50
[49810.198608]  [<ffffffff811e7360>] do_vfs_ioctl+0x2e0/0x4c0
[49810.198617]  [<ffffffff811d96c5>] ? SYSC_newfstat+0x25/0x30
[49810.198624]  [<ffffffff811e75c1>] SyS_ioctl+0x81/0xa0
[49810.198638]  [<ffffffff8176da4d>] system_call_fastpath+0x1a/0x1f
 


  • bash Linux how to get first or last letters of a word
  • l2tp ipsec VPN Error Sep 12 18:16:25 vps pluto[7299]: ERROR: asynchronous network error report on eth0 (sport=500) for message to 192.5.6.2 port 20640, complainant 192.5.6.2: Connection refused [errno 111, origin ICMP type 3 code 3 (not authenticated
  • Centos 5.9 Working Vault Repo file
  • Disable SSH Password Authentication to Increase Security and Harden SSH Linux Unix Server Ubuntu Mint Centos Debian
  • ecryptfs how to mount or recover from a backup
  • Linux bash script to see what connected computers respond to ping
  • rsync specify alternate port non-standard port than 22
  • Centos 6.6/6.9 KVM VM Kernel Panic On Boot - Kernel panic - not syncing: Attempted to kill init!
  • Cannot allocate TUN/TAP dev dynamically - FreeBSD/OpenBSD OpenVPN Client error Solution
  • Linux Mint/Ubuntu/Debian apt how to downgrade a package
  • Ubuntu Linux Mint How To Exclude Stop Package from being Upgraded or Installed
  • Intel NUC Lower Power Green Computing Boxes Review/Comparison of J3160 and J3455 Models
  • How to backup entire computer Linux Mint Ubuntu with tar
  • tar how to ignore failed reads and not exit
  • kdenlive titles/text renders as white screen when using .sh script
  • [1035724.274610] [drm:intel_pipe_update_end [i915_bpo]] *ERROR* Atomic update failure on pipe A (start=62076478 end=62076479) time 102 us, min 894, max 899, scanline start 893, end 900 W: Possible missing firmware /lib/firmware/i915/kbl_dmc_ver1.bin
  • ffmpeg how to watermark and concatenate in one command
  • /usr/bin/ld: cannot find -lboost_system-mt-s /usr/bin/ld: cannot find -lboost_filesystem-mt-s /usr/bin/ld: cannot find -lboost_program_options-mt-s /usr/bin/ld: cannot find -lboost_thread-mt-s collect2: error: ld returned 1 exit status make: *** [cag
  • Wine uninstalled broken on Linux Mint
  • ffmpeg trouble concatenating similar but different files