How to Track Down/Find Cause of Linux high disk iowait/wa

top - 09:34:12 up 2 days, 20:57,  2 users,  load average: 1.83, 1.99, 2.03
Tasks:  59 total,   2 running,  57 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.0%sy,  0.0%ni,  0.0%id, 99.7%wa,  0.0%hi,  0.0%si,  0.0%st

That 99.7% wa is iowait, it means the server is waiting for a process to complete an IO operation or in plain English, there is a delay in reading and/or writing data to the hard drive.

Here are some ways to get an idea of what is going on (in RPM based Distros this package is called "sysstat") and contains the very helpful binary called "iostat".  There's another package I like called "dstat" that does a great job too.

Run iostat -d 5 (it refreshes the data every 5 seconds):

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1              2.07         5.27        28.66    1309570    7120384
sda2              0.00         0.03         0.04       7496       8832

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1             27.15         0.00       357.68          0       1792
sda2              0.00         0.00         0.00          0          0

What we're seeing is that the iowait must be caused primarily by blocks being written to /dev/sda1 which is mounted as / (my root filesystem).  So now we know the main cause is by an application constantly writing to the root partition, but we still don't know what process is directly responsible for this.

If you have a newer kernel it's often possible to view processes by io usage, you can test for this by running: cat /proc/self/io

If the file is not found then you don't have io stats enabled in your kernel.  Then you can check a suspected process by running cat /proc/13/io (replace the 13 with the PID you want to check).

read_bytes: 0
write_bytes: 0
cancelled_write_bytes: 0

 


Tags:

linux, disk, iowait, watop, users, tasks, cpu, sy, ni, wa, server, io, rpm, distros, quot, sysstat, contains, binary, iostat, dstat, refreshes, tps, blk_read, blk_wrtn, sda, primarily, dev, mounted, filesystem, partition, newer, kernel, processes, usage, proc, stats, enabled, pid, read_bytes, write_bytes, cancelled_write_bytes,

Latest Articles

  • python 3 pip cannot install mysql module
  • QEMU-KVM won't boot Windows 2016 or 2019 server on an Intel Core i3
  • Virtualbox vbox not starting "No suitable module for running kernel found"
  • Bind / named not responding to queries solution
  • Linux Mint How To Set Desktop Background Image From Bash Prompt CLI
  • ImageMagick Convert PDF Not Authorized
  • ImageMagick Converted PDF to JPEG some files have a black background solution
  • Linux Mint Mate Customize the Lock screen messages and hide username and real name
  • Ubuntu/Gnome/Mint/Centos How To Take a partial screenshot
  • ssh how to verify your host key / avoid MIM attacks
  • Cisco IP Phone CP-8845 8800/8900 Series How To Reset To Factory Settings Instructions
  • ls how to list ONLY directories
  • How to encrypt your SSH private key file id_rsa
  • Linux Mint 18 Disable User Name List from showing on Login Screen
  • Firefox Cannot Hit Enter Key In Address Bar and Location History Not Working
  • Cisco Unified Communications Manager / CUCM IP 8.6,10,12 Install Error Solution
  • Ubuntu Debian Mint Linux SSHD OpenSSH Server Not Starting After Reboot Solution
  • nmap how to scan for all ports and not just the 1000 most common ports
  • Windows 7,8,10 and Server 2008, 2012, 2016, 2019 Read Only Attribute Won't Go Away
  • bind / named how to make a wildcard record and retain defined A records