/ mdadm frozen and doesn't realize array is dead/missing failed due to unplugged drives - rtt - IT Resource

mdadm frozen and doesn't realize array is dead/missing failed due to unplugged drives

This was a surprising bug but I unplugged all drives for an array md127. At first it was just 1 drive and mdadm seemed to notice this. I unplugged the second drive taking the array offline but mdadm did not realize it was offline and still showed a non-existent disk as being part of it. This created problems trying to unmount it or even to stop this array with mdadm freezing.

As for how to fix it I can only think of making sure you are not in a mounted path of the array, unmount it and then manually stop the array but this didn't seem to work until the hung task kicked in.

mdadm --manage /dev/md127 --stop

The resulting kernel messages are what followed:

8586.047108] EXT4-fs error (device md127): __ext4_get_inode_loc:3932: inode #12058625: block 48234528: comm pool: unable to read itable block
[48586.047119] EXT4-fs (md127): previous I/O error to superblock detected
[48586.047207] Buffer I/O error on device md127, logical block 0
[48586.047216] lost page write due to I/O error on md127
[48586.248009] md: super_written gets error=-19, uptodate=0
[48594.634065] md: super_written gets error=-19, uptodate=0
[48594.634173] Buffer I/O error on device md127, logical block 91258880
[48594.634186] lost page write due to I/O error on md127
[48594.634221] JBD2: Error -5 detected when updating journal superblock for md127-8.
[48594.634286] Buffer I/O error on device md127, logical block 91258880
[48594.634299] lost page write due to I/O error on md127
[48594.634329] JBD2: Error -5 detected when updating journal superblock for md127-8.
[48594.635892] EXT4-fs (md127): previous I/O error to superblock detected
[48594.635974] Buffer I/O error on device md127, logical block 0
[48594.635984] lost page write due to I/O error on md127
[48594.837653] md: super_written gets error=-19, uptodate=0
[48851.459776] init: smbd main process (1204) killed by TERM signal
[48966.998219] Buffer I/O error on device md127, logical block 0
[48966.998241] Buffer I/O error on device md127, logical block 0
[48966.998267] Buffer I/O error on device md127, logical block 183110512
[48966.998277] Buffer I/O error on device md127, logical block 183110512
[48966.998295] Buffer I/O error on device md127, logical block 183110526
[48966.998304] Buffer I/O error on device md127, logical block 183110526
[48966.998315] Buffer I/O error on device md127, logical block 0
[48966.998329] Buffer I/O error on device md127, logical block 1
[48966.998337] Buffer I/O error on device md127, logical block 1
[48966.998357] Buffer I/O error on device md127, logical block 183110527
[49172.119413] md: md20 stopped.
[49172.193601] md: bind
[49184.251750] quiet_error: 124 callbacks suppressed
[49184.251764] Buffer I/O error on device md127, logical block 183110512
[49184.251778] Buffer I/O error on device md127, logical block 183110512
[49184.251799] Buffer I/O error on device md127, logical block 183110526
[49184.251808] Buffer I/O error on device md127, logical block 183110526
[49184.251823] Buffer I/O error on device md127, logical block 0
[49184.251831] Buffer I/O error on device md127, logical block 0
[49184.251846] Buffer I/O error on device md127, logical block 1
[49184.251854] Buffer I/O error on device md127, logical block 1
[49184.251869] Buffer I/O error on device md127, logical block 183110527
[49184.251877] Buffer I/O error on device md127, logical block 183110527
[49210.081588] INFO: task mdadm:16228 blocked for more than 120 seconds.
[49210.081605] Tainted: P OE 3.16.0-38-generic #52~14.04.1-Ubuntu
[49210.081610] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[49210.081616] mdadm D ffff88080fd530c0 0 16228 15783 0x00000004
[49210.081627] ffff88005539fc70 0000000000000082 ffff880804045180 ffff88005539ffd8
[49210.081635] 00000000000130c0 00000000000130c0 ffff880804721460 ffff88005539fd98
[49210.081642] ffff88005539fda0 7fffffffffffffff ffff880804045180 0000000000000000
[49210.081649] Call Trace:
[49210.081668] [] schedule+0x29/0x70
[49210.081677] [] schedule_timeout+0x229/0x2a0
[49210.081687] [] ? __queue_work+0x136/0x320
[49210.081694] [] ? __queue_delayed_work+0xaa/0x1a0
[49210.081701] [] ? try_to_grab_pending+0xa9/0x160
[49210.081709] [] wait_for_completion+0xa6/0x160
[49210.081718] [] ? wake_up_state+0x20/0x20
[49210.081728] [] writeback_inodes_sb_nr+0x7f/0xb0
[49210.081735] [] writeback_inodes_sb+0x25/0x30
[49210.081745] [] sync_filesystem+0x2d/0xa0
[49210.081753] [] fsync_bdev+0x24/0x60
[49210.081764] [] blkdev_ioctl+0x433/0x7b0
[49210.081772] [] block_ioctl+0x41/0x50
[49210.081779] [] do_vfs_ioctl+0x2e0/0x4c0
[49210.081788] [] ? SYSC_newfstat+0x25/0x30
[49210.081795] [] SyS_ioctl+0x81/0xa0
[49210.081804] [] system_call_fastpath+0x1a/0x1f
[49210.866657] quiet_error: 260 callbacks suppressed
[49210.866670] Buffer I/O error on device md127, logical block 183110512
[49210.866685] Buffer I/O error on device md127, logical block 183110512
[49210.866705] Buffer I/O error on device md127, logical block 183110526
[49210.866715] Buffer I/O error on device md127, logical block 183110526
[49210.866729] Buffer I/O error on device md127, logical block 0
[49210.866738] Buffer I/O error on device md127, logical block 0
[49210.866753] Buffer I/O error on device md127, logical block 1
[49210.866762] Buffer I/O error on device md127, logical block 1
[49210.866777] Buffer I/O error on device md127, logical block 183110527
[49210.866786] Buffer I/O error on device md127, logical block 183110527
[49218.834738] md127: detected capacity change from 750020722688 to 0
[49218.834759] md: md127 stopped.
[49218.834776] md: unbind
[49218.839214] md: export_rdev(sdc1)
[49275.360612] init: smbd main process (16150) killed by TERM signal
[49330.104883] INFO: task mdadm:16228 blocked for more than 120 seconds.
[49330.104892] Tainted: P OE 3.16.0-38-generic #52~14.04.1-Ubuntu
[49330.104895] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[49330.104898] mdadm D ffff88080fd530c0 0 16228 15783 0x00000004
[49330.104904] ffff88005539fc70 0000000000000082 ffff880804045180 ffff88005539ffd8
[49330.104908] 00000000000130c0 00000000000130c0 ffff880804721460 ffff88005539fd98
[49330.104911] ffff88005539fda0 7fffffffffffffff ffff880804045180 0000000000000000
[49330.104916] Call Trace:
[49330.104929] [] schedule+0x29/0x70
[49330.104933] [] schedule_timeout+0x229/0x2a0
[49330.104939] [] ? __queue_work+0x136/0x320
[49330.104942] [] ? __queue_delayed_work+0xaa/0x1a0
[49330.104946] [] ? try_to_grab_pending+0xa9/0x160
[49330.104949] [] wait_for_completion+0xa6/0x160
[49330.104955] [] ? wake_up_state+0x20/0x20
[49330.104960] [] writeback_inodes_sb_nr+0x7f/0xb0
[49330.104963] [] writeback_inodes_sb+0x25/0x30
[49330.104969] [] sync_filesystem+0x2d/0xa0
[49330.104973] [] fsync_bdev+0x24/0x60
[49330.104979] [] blkdev_ioctl+0x433/0x7b0
[49330.104983] [] block_ioctl+0x41/0x50
[49330.104987] [] do_vfs_ioctl+0x2e0/0x4c0
[49330.104992] [] ? SYSC_newfstat+0x25/0x30
[49330.104995] [] SyS_ioctl+0x81/0xa0
[49330.104999] [] system_call_fastpath+0x1a/0x1f
[49488.994434] md/raid10:md20: active with 1 out of 2 devices
[49488.994492] md20: detected capacity change from 0 to 750020722688
[49489.014456] md20: unknown partition table
[49505.301904] EXT4-fs (md20): recovery complete
[49505.319010] EXT4-fs (md20): mounted filesystem with ordered data mode. Opts: (null)
[49524.680801] systemd-hostnamed[16729]: Warning: nss-myhostname is not installed. Changing the local hostname might make it unresolveable. Please install nss-myhostname!
[49570.151706] INFO: task mdadm:16228 blocked for more than 120 seconds.
[49570.151723] Tainted: P OE 3.16.0-38-generic #52~14.04.1-Ubuntu
[49570.151728] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[49570.151734] mdadm D ffff88080fd530c0 0 16228 15783 0x00000004
[49570.151745] ffff88005539fc80 0000000000000082 ffff880804045180 ffff88005539ffd8
[49570.151753] 00000000000130c0 00000000000130c0 ffff880804721460 ffff88005539fda8
[49570.151761] ffff88005539fdb0 7fffffffffffffff ffff880804045180 0000000000000000
[49570.151768] Call Trace:
[49570.151787] [] schedule+0x29/0x70
[49570.151796] [] schedule_timeout+0x229/0x2a0
[49570.151806] [] ? __queue_work+0x136/0x320
[49570.151813] [] ? __queue_delayed_work+0xaa/0x1a0
[49570.151820] [] ? try_to_grab_pending+0x11d/0x160
[49570.151828] [] wait_for_completion+0xa6/0x160
[49570.151837] [] ? wake_up_state+0x20/0x20
[49570.151846] [] sync_inodes_sb+0xa5/0x1c0
[49570.151857] [] sync_filesystem+0x5b/0xa0
[49570.151865] [] fsync_bdev+0x24/0x60
[49570.151876] [] blkdev_ioctl+0x433/0x7b0
[49570.151883] [] block_ioctl+0x41/0x50
[49570.151891] [] do_vfs_ioctl+0x2e0/0x4c0
[49570.151899] [] ? SYSC_newfstat+0x25/0x30
[49570.151906] [] SyS_ioctl+0x81/0xa0
[49570.151915] [] system_call_fastpath+0x1a/0x1f
[49690.175082] INFO: task mdadm:16228 blocked for more than 120 seconds.
[49690.175099] Tainted: P OE 3.16.0-38-generic #52~14.04.1-Ubuntu
[49690.175104] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[49690.175110] mdadm D ffff88080fd530c0 0 16228 15783 0x00000004
[49690.175121] ffff88005539fc80 0000000000000082 ffff880804045180 ffff88005539ffd8
[49690.175129] 00000000000130c0 00000000000130c0 ffff880804721460 ffff88005539fda8
[49690.175136] ffff88005539fdb0 7fffffffffffffff ffff880804045180 0000000000000000
[49690.175144] Call Trace:
[49690.175164] [] schedule+0x29/0x70
[49690.175172] [] schedule_timeout+0x229/0x2a0
[49690.175182] [] ? __queue_work+0x136/0x320
[49690.175190] [] ? __queue_delayed_work+0xaa/0x1a0
[49690.175197] [] ? try_to_grab_pending+0x11d/0x160
[49690.175205] [] wait_for_completion+0xa6/0x160
[49690.175214] [] ? wake_up_state+0x20/0x20
[49690.175224] [] sync_inodes_sb+0xa5/0x1c0
[49690.175234] [] sync_filesystem+0x5b/0xa0
[49690.175242] [] fsync_bdev+0x24/0x60
[49690.175253] [] blkdev_ioctl+0x433/0x7b0
[49690.175261] [] block_ioctl+0x41/0x50
[49690.175268] [] do_vfs_ioctl+0x2e0/0x4c0
[49690.175277] [] ? SYSC_newfstat+0x25/0x30
[49690.175284] [] SyS_ioctl+0x81/0xa0
[49690.175293] [] system_call_fastpath+0x1a/0x1f
[49810.198423] INFO: task mdadm:16228 blocked for more than 120 seconds.
[49810.198438] Tainted: P OE 3.16.0-38-generic #52~14.04.1-Ubuntu
[49810.198443] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[49810.198449] mdadm D ffff88080fd530c0 0 16228 15783 0x00000004
[49810.198461] ffff88005539fc80 0000000000000082 ffff880804045180 ffff88005539ffd8
[49810.198469] 00000000000130c0 00000000000130c0 ffff880804721460 ffff88005539fda8
[49810.198476] ffff88005539fdb0 7fffffffffffffff ffff880804045180 0000000000000000
[49810.198483] Call Trace:
[49810.198503] [] schedule+0x29/0x70
[49810.198512] [] schedule_timeout+0x229/0x2a0
[49810.198522] [] ? __queue_work+0x136/0x320
[49810.198530] [] ? __queue_delayed_work+0xaa/0x1a0
[49810.198537] [] ? try_to_grab_pending+0x11d/0x160
[49810.198545] [] wait_for_completion+0xa6/0x160
[49810.198554] [] ? wake_up_state+0x20/0x20
[49810.198563] [] sync_inodes_sb+0xa5/0x1c0
[49810.198574] [] sync_filesystem+0x5b/0xa0
[49810.198582] [] fsync_bdev+0x24/0x60
[49810.198593] [] blkdev_ioctl+0x433/0x7b0
[49810.198601] [] block_ioctl+0x41/0x50
[49810.198608] [] do_vfs_ioctl+0x2e0/0x4c0
[49810.198617] [] ? SYSC_newfstat+0x25/0x30
[49810.198624] [] SyS_ioctl+0x81/0xa0
[49810.198638] [] system_call_fastpath+0x1a/0x1f


Tags:

mdadm, doesn, array, unplugged, drivesthis, surprising, md, offline, existent, disk, unmount, mounted, manually, didn, task, dev, resulting, kernel, ext, fs, __ext, _get_inode_loc, inode, comm, unable, itable, previous, superblock, detected, buffer, super_written, uptodate, jbd, updating, init, smbd, bind, sdf, quiet_error, callbacks, suppressed, info, blocked, tainted, oe, generic, ubuntu, quot, echo, proc, sys, hung_task_timeout_secs, disables, ffff, fd, fc, ffd, fda, fffffffffffffff, ffffffff, dd, schedule_timeout, ef, __queue_work, aa, __queue_delayed_work, xaa, try_to_grab_pending, xa, wait_for_completion, wake_up_state, writeback_inodes_sb_nr, xb, writeback_inodes_sb, sync_filesystem, af, fsync_bdev, blkdev_ioctl, block_ioctl, do_vfs_ioctl, sysc_newfstat, sys_ioctl, system_call_fastpath, capacity, unbind, sdc, export_rdev, raid, active, devices, partition, filesystem, mode, opts, null, systemd, hostnamed, nss, myhostname, installed, hostname, unresolveable, install, fdb, fca, sync_inodes_sb, fb,

Latest Articles

  • syntax error, unexpected T_SL in PHP Solution
  • grep regular expression match number range between specific numbers
  • bash how to print out lines of text within a range from the first occurrence
  • bash script how to to check LAN computers for open ports
  • MySQL Using mytop Debug Source of High IO and Slow Performance
  • How To Mathematically Convert and Calculate Binary Value To Decimal Value
  • systemd management using systemctl and journalctl to check systemd logs
  • css how to format code in the code tag
  • css br with extra blank line
  • What is /dev/pts and why do we need it in Linux?
  • Linux What is umask (user mask) for file and directory creation permissions and how to calculate umask and change the defaults
  • Linux Permissions and Groups Real Usecase for Group Access
  • Centos 7 python34 how to install the gi library
  • Centos 7 - Convert Minimal to Graphical GUI GNOME or KDE Desktop
  • AMD Set Fan Speed and Other Powerplay Memory/CPU Timings with a Linux script
  • Ethereum Mining Claymore Nanopool Error
  • genisoimage errors with long filenames and deep directory structures
  • Linux Kernel USB Export Errors
  • How to download gajim 0.16.9 XMPP/Jabber client so you can use OMEMO encryption
  • HP DL385 G7 Linux BIOS Update Flash