How to Track Down/Find Cause of Linux high disk iowait/wa

top - 09:34:12 up 2 days, 20:57,  2 users,  load average: 1.83, 1.99, 2.03
Tasks:  59 total,   2 running,  57 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.0%sy,  0.0%ni,  0.0%id, 99.7%wa,  0.0%hi,  0.0%si,  0.0%st

That 99.7% wa is iowait, it means the server is waiting for a process to complete an IO operation or in plain English, there is a delay in reading and/or writing data to the hard drive.

Here are some ways to get an idea of what is going on (in RPM based Distros this package is called "sysstat") and contains the very helpful binary called "iostat".  There's another package I like called "dstat" that does a great job too.

Run iostat -d 5 (it refreshes the data every 5 seconds):

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1              2.07         5.27        28.66    1309570    7120384
sda2              0.00         0.03         0.04       7496       8832

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1             27.15         0.00       357.68          0       1792
sda2              0.00         0.00         0.00          0          0

What we're seeing is that the iowait must be caused primarily by blocks being written to /dev/sda1 which is mounted as / (my root filesystem).  So now we know the main cause is by an application constantly writing to the root partition, but we still don't know what process is directly responsible for this.

If you have a newer kernel it's often possible to view processes by io usage, you can test for this by running: cat /proc/self/io

If the file is not found then you don't have io stats enabled in your kernel.  Then you can check a suspected process by running cat /proc/13/io (replace the 13 with the PID you want to check).

read_bytes: 0
write_bytes: 0
cancelled_write_bytes: 0

 

Latest Articles

  • AMD Set Fan Speed and Other Powerplay Memory/CPU Timings with a Linux script
  • Ethereum Mining Claymore Nanopool Error
  • genisoimage errors with long filenames and deep directory structures
  • Linux Kernel USB Export Errors
  • How to download gajim 0.16.9 XMPP/Jabber client so you can use OMEMO encryption
  • HP DL385 G7 Linux BIOS Update Flash
  • hwloc-nox set CPU affinity in Linux
  • Firefox An error occurred during a connection to some-ip-or-domain. SSL peer reports incorrect Message Authentication Code. Error code: SSL_ERROR_BAD_MAC_ALERT Solution
  • Proxmox understanding the directory structure and why an NFS datastore appears to be missing files/isos
  • pandoc convert markdown to html
  • Proxmox error uploading an iso solution
  • Cannot install moodle
  • MySQL change for Antelope format to Barracuda error solution
  • vmkping -I vmk1 10.0.2.69 PING 10.0.2.69 (10.0.2.69): 56 data bytes sendto() failed (Host is down)
  • gvfs mount in /run/user cannot be accessed or displayed wrong permissions
  • VMWare vSphere 6.7 Errors Solution 503 Service Unavailable (Failed to connect to endpoint:
  • How To Enable Nested KVM so guests can virtualize with hardware extensions
  • vi error solution E166: Can't open linked file for writing
  • Supermicro IPMI / KVM / BMC Remote Console Screen Resizing Issue - Window Cut Off Solution
  • Linux bash shell doesn't show username, hostname or current path fix solution