Linux High IOWAIT updatedb can't be killed and crash with mdadm

high IO wait

424 root 39 19 1900 848 552 D 0.0 0.0 0:00.91 updatedb

root 424 0.0 0.0 1900 848 ? DN Mar11 0:00 /usr/bin/updatedb -f sysfs?rootfs?bdev?proc?cpuset?binfmt_misc?debugfs?sockfs?usbfs?pipefs?anon_inodefs?futexfs?tmpfs?inotifyfs?eventpollfs?devpts?ramfs?hugetlbfs?mqueue?rpc_pipefs?autofs
root 2907 0.0 0.0 1768 492 ? DN Mar15 0:00 /usr/bin/updatedb -f sysfs?rootfs?bdev?proc?cpuset?binfmt_misc?debugfs?sockfs?usbfs?pipefs?anon_inodefs?futexfs?tmpfs?inotifyfs?eventpollfs?devpts?ramfs?hugetlbfs?mqueue?rpc_pipefs?autofs
root 8691 0.0 0.0 1768 492 ? DN Mar12 0:00 /usr/bin/updatedb -f sysfs?rootfs?bdev?proc?cpuset?binfmt_misc?debugfs?sockfs?usbfs?pipefs?anon_inodefs?futexfs?tmpfs?inotifyfs?eventpollfs?devpts?ramfs?hugetlbfs?mqueue?rpc_pipefs?autofs
root 10642 0.0 0.0 4004 696 pts/0 S+ 00:38 0:00 grep updatedb
root 14937 0.0 0.0 1768 492 ? DN Mar16 0:00 /usr/bin/updatedb -f sysfs?rootfs?bdev?proc?cpuset?binfmt_misc?debugfs?sockfs?usbfs?pipefs?anon_inodefs?futexfs?tmpfs?inotifyfs?eventpollfs?devpts?ramfs?hugetlbfs?mqueue?rpc_pipefs?autofs
root 16697 0.0 0.0 1768 488 ? DN Mar13 0:00 /usr/bin/updatedb -f sysfs?rootfs?bdev?proc?cpuset?binfmt_misc?debugfs?sockfs?usbfs?pipefs?anon_inodefs?futexfs?tmpfs?inotifyfs?eventpollfs?devpts?ramfs?hugetlbfs?mqueue?rpc_pipefs?autofs
root 18790 0.0 0.0 1768 496 ? DN Mar18 0:00 /usr/bin/updatedb -f sysfs?rootfs?bdev?proc?cpuset?binfmt_misc?debugfs?sockfs?usbfs?pipefs?anon_inodefs?futexfs?tmpfs?inotifyfs?eventpollfs?devpts?ramfs?hugetlbfs?mqueue?rpc_pipefs?autofs
root 24783 0.0 0.0 1768 500 ? DN Mar14 0:00 /usr/bin/updatedb -f sysfs?rootfs?bdev?proc?cpuset?binfmt_misc?debugfs?sockfs?usbfs?pipefs?anon_inodefs?futexfs?tmpfs?inotifyfs?eventpollfs?devpts?ramfs?hugetlbfs?mqueue?rpc_pipefs?autofs
root 24922 0.0 0.0 1768 492 ? DN Mar17 0:00 /usr/bin/updatedb -f sysfs?rootfs?bdev?proc?cpuset?binfmt_misc?debugfs?sockfs?usbfs?pipefs?anon_inodefs?futexfs?tmpfs?inotifyfs?eventpollfs?devpts?ramfs?hugetlbfs?mqueue?rpc_pipefs?autofs
root 29655 0.0 0.0 1768 496 ? DN Mar19 0:00 /usr/bin/updatedb -f sysfs?rootfs?bdev?proc?cpuset?binfmt_misc?debugfs?sockfs?usbfs?pipefs?anon_inodefs?futexfs?tmpfs?inotifyfs?eventpollfs?devpts?ramfs?hugetlbfs?mqueue?rpc_pipefs?autofs


I noticed that updatedb was running for days and then this dmesg message:

(manually killing the process and also killall updatedb does not stop it or kill it)

INFO: task updatedb:424 blocked for more than 300 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
updatedb D F79C8CF0 6476 424 419 (NOTLB)
e740bdd4 00000086 c30045c0 000e2832 f7acea4c 00000001 f79c8cf0 c30f8800
000e2832 000f4240 556bd125 0024f991 f79c8e14 f7d14000 556bf321 0024f991
00000002 00000000 00000001 00000000 c0419058 f7d14000 c06ecd80 f7a0cdec
Call Trace:
[] __wake_up+0x2a/0x3d
[] io_schedule+0x3b/0x64
[] sync_page+0x0/0x3b
[] sync_page+0x38/0x3b
[] __wait_on_bit_lock+0x2a/0x52
[] __lock_page+0x52/0x59
[] wake_bit_function+0x0/0x3c
[] truncate_inode_pages_range+0x20b/0x260
[] journal_stop+0x208/0x213 [jbd]
[] truncate_inode_pages+0x9/0xe
[] ext3_delete_inode+0x13/0xba [ext3]
[] ext3_delete_inode+0x0/0xba [ext3]
[] generic_delete_inode+0x91/0xfe
[] iput+0x67/0x69
[] d_kill+0x19/0x32
[] dput+0x19f/0x1ac
[] sys_renameat+0x15f/0x1af
[] _atomic_dec_and_lock+0x2a/0x44
[] audit_syscall_entry+0x160/0x192
[] sys_rename+0x11/0x15
[] syscall_call+0x7/0xb
=======================

kill -9 424
strace -p 424

Process 424 attached - interrupt to quit

attach: ptrace(PTRACE_ATTACH, ...): Operation not permitted

==========================
Eventually the system crashed with references to ext3 and inodes etc..
After a reboot it did the following:

md: md0: raid array is not clean -- starting background reconstruction
raid1: raid set md0 active with 2 out of 2 mirrors
md: ... autorun DONE.
md: syncing RAID array md0
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction.
md: using 128k window, over a total of 511999424 blocks.
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting. Commit interval 5 seconds
EXT3-fs: md0: orphan cleanup on readonly fs
ext3_orphan_cleanup: deleting unreferenced inode 97467990
ext3_orphan_cleanup: deleting unreferenced inode 76153118
EXT3-fs: md0: 2 orphan inodes deleted
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.


Tags:

linux, iowait, updatedb, mdadmhigh, io, dn, mar, usr, bin, sysfs, rootfs, bdev, proc, cpuset, binfmt_misc, debugfs, sockfs, usbfs, pipefs, anon_inodefs, futexfs, tmpfs, inotifyfs, eventpollfs, devpts, ramfs, hugetlbfs, mqueue, rpc_pipefs, autofs, pts, grep, dmesg, manually, killall, info, task, blocked, quot, echo, sys, kernel, hung_task_timeout_secs, disables, cf, notlb, bdd, acea, bd, bf, ecd, cdec, __wake_up, io_schedule, ead, sync_page, ee, __wait_on_bit_lock, __lock_page, wake_bit_function, truncate_inode_pages_range, journal_stop, jbd, truncate_inode_pages, xe, ca, ext, _delete_inode, xba, generic_delete_inode, xfe, iput, dd, d_kill, dput, ac, sys_renameat, af, ebb, _atomic_dec_and_lock, audit_syscall_entry, sys_rename, syscall_call, xb, strace, attach, ptrace, ptrace_attach, permitted, references, inodes, etc, reboot, md, raid, array, reconstruction, active, mirrors, autorun, syncing, autodetecting, arrays, _guaranteed_, kb, disc, maximum, idle, bandwidth, fs, readonly, filesystem, enabled, kjournald, interval, orphan, cleanup, _orphan_cleanup, deleting, unreferenced, inode, deleted, mounted, mode,

Latest Articles

  • How To Install OpenProject on Centos 7 Step-by-Step Guide
  • Ubuntu Debian Linux Cannot Install Wine Solution - wine1.6 : Depends: wine1.6-i386 (= 1:1.6.2-0ubuntu14.2) but it is not installable wine1.4 : Depends: wine1.6 but it is not going to be installed
  • How To Install python 3.4 3.5 and up on Linux with wine - Working Solution
  • using Xvfb on virtual remote ssh server to have X graphical programs work
  • ssh Received disconnect from port 22:2: Too many authentication failures
  • named bind errors - DNSKEY: unable to find a DNSKEY which verifies the DNSKEY RRset and also matches a trusted key for '.'
  • OpenVZ vs LXC DIR mode poor security in LXC
  • httpd: Syntax error on line 221 of /etc/httpd/conf/httpd.conf: Syntax error on line 6 of /etc/httpd/conf.d/php.conf: Cannot load modules/libphp5.so into server: /lib64/libresolv.so.2: symbol __h_errno, version GLIBC_PRIVATE not defined in file libc.s
  • Radeon R3 GPU on Debian Crashing
  • MySQL 5.7 on Debian and Ubuntu - How To Reset Root Password
  • SSH and sshfs timeout settings keepalive
  • Linux How To Add User To Additional Group
  • Howto Set Static IP on boot in initramfs for dropbear or other purposes NFS, Linux, Debian, Ubuntu, CentOS
  • Convert and install to LUKS Encrypted Drive Ubuntu 18.04 19.10 Linux Mint and Debian Based Linux
  • Debian and Netplan
  • CentOS 8 how to restart the network!
  • CentOS 8 how to convert to a bootable mdadm RAID software array
  • ADATA USB Thumb Drive Issues
  • KMODE EXCEPTION NOT HANDLED - QEMU/KVM Won't Boot Windows 2016 or 10 Image or Physical Machine
  • Linux Mint / Ubuntu / Debian Mate Disable Guest Session and Hide Usernames on Lightdm Login screen GUI