How to Track Down/Find Cause of Linux high disk iowait/wa

top - 09:34:12 up 2 days, 20:57,  2 users,  load average: 1.83, 1.99, 2.03
Tasks:  59 total,   2 running,  57 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.0%sy,  0.0%ni,  0.0%id, 99.7%wa,  0.0%hi,  0.0%si,  0.0%st

That 99.7% wa is iowait, it means the server is waiting for a process to complete an IO operation or in plain English, there is a delay in reading and/or writing data to the hard drive.

Here are some ways to get an idea of what is going on (in RPM based Distros this package is called "sysstat") and contains the very helpful binary called "iostat".  There's another package I like called "dstat" that does a great job too.

Run iostat -d 5 (it refreshes the data every 5 seconds):

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1              2.07         5.27        28.66    1309570    7120384
sda2              0.00         0.03         0.04       7496       8832

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1             27.15         0.00       357.68          0       1792
sda2              0.00         0.00         0.00          0          0

What we're seeing is that the iowait must be caused primarily by blocks being written to /dev/sda1 which is mounted as / (my root filesystem).  So now we know the main cause is by an application constantly writing to the root partition, but we still don't know what process is directly responsible for this.

If you have a newer kernel it's often possible to view processes by io usage, you can test for this by running: cat /proc/self/io

If the file is not found then you don't have io stats enabled in your kernel.  Then you can check a suspected process by running cat /proc/13/io (replace the 13 with the PID you want to check).

read_bytes: 0
write_bytes: 0
cancelled_write_bytes: 0

 


Tags:

linux, disk, iowait, watop, users, tasks, cpu, sy, ni, wa, server, io, rpm, distros, quot, sysstat, contains, binary, iostat, dstat, refreshes, tps, blk_read, blk_wrtn, sda, primarily, dev, mounted, filesystem, partition, newer, kernel, processes, usage, proc, stats, enabled, pid, read_bytes, write_bytes, cancelled_write_bytes,

Latest Articles

  • Linux How To Create A RamDisk
  • mdadm force resync when resync=PENDING solution
  • Proxmox Breaks Storage/LVM Backing If Killing QEMU-IMG
  • Proxmox trying to acquire lock... TASK ERROR: can't lock file '/var/lock/qemu-server/lock-102.conf' - got timeout
  • Debian 9 SSH root password authentication failure password not working problem / solution
  • QEMU / KVM How To Manually Create Basic Virtual Machine VM
  • Linux wlan0 check all wireless clients
  • PHP Issues With Decoding Strange Smart Quotes And Non-Standard ASCII Characters
  • /etc/iproute2/rt_tables default settings file in Linux Centos 6,7 and most other NIX's
  • bind named error solutions named[2169]: error (no valid DS) resolving / error (broken trust chain) resolving / : error (no valid RRSIG) resolving 'com/DS/IN':
  • iptables how to log ALL dropped incoming packets
  • How To Edit Linux Based NM Network Manager Connection Settings Without GUI
  • Linux Disable IPV6 Centos / Debian / Mint Howto
  • Linux use growisofs to burn a larger file on a BD-R / Bluray Disc
  • Linux partprobe/partx cannot access last and 4th partition
  • DRBD Errors Caused By Physical Corruption
  • mdadm: add new device failed for /dev/sdb4 as 3: Invalid argument solution
  • Linux named / bind how to dump, view and clear the cache!
  • Centos 6 / 7 / 8 How To Change Default nameservers in /etc/resolv.conf when using DHCP / dhclient
  • Adobe Acrobat Reader for Linux to use and view XFA Fillable Forms