How to Track Down/Find Cause of Linux high disk iowait/wa -

How to Track Down/Find Cause of Linux high disk iowait/wa

top - 09:34:12 up 2 days, 20:57,  2 users,  load average: 1.83, 1.99, 2.03
Tasks:  59 total,   2 running,  57 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.0%sy,  0.0%ni,  0.0%id, 99.7%wa,  0.0%hi,  0.0%si,  0.0%st

That 99.7% wa is iowait, it means the server is waiting for a process to complete an IO operation or in plain English, there is a delay in reading and/or writing data to the hard drive.

Here are some ways to get an idea of what is going on (in RPM based Distros this package is called "sysstat") and contains the very helpful binary called "iostat".  There's another package I like called "dstat" that does a great job too.

Run iostat -d 5 (it refreshes the data every 5 seconds):

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1              2.07         5.27        28.66    1309570    7120384
sda2              0.00         0.03         0.04       7496       8832

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1             27.15         0.00       357.68          0       1792
sda2              0.00         0.00         0.00          0          0

What we're seeing is that the iowait must be caused primarily by blocks being written to /dev/sda1 which is mounted as / (my root filesystem).  So now we know the main cause is by an application constantly writing to the root partition, but we still don't know what process is directly responsible for this.

If you have a newer kernel it's often possible to view processes by io usage, you can test for this by running: cat /proc/self/io

If the file is not found then you don't have io stats enabled in your kernel.  Then you can check a suspected process by running cat /proc/13/io (replace the 13 with the PID you want to check).

read_bytes: 0
write_bytes: 0
cancelled_write_bytes: 0

 


  • How Do you Open/Extract .WARC Internet Archive Files on Linux Ubuntu/Mint/Centos?
  • How To Disable htaccess inheritance or exclude a directory
  • root/home directory has ownership changed to the wrong user/owner mysteriously
  • mdadm and lvm how to completely disable and remove vg/pv/lv
  • sshd[10470]: Authentication refused: bad ownership or modes for directory /root
  • LG Phoenix 2 Escape Disable AT&T Phonebook/Contacts Error Message
  • mdadm frozen and doesn't realize array is dead/missing failed due to unplugged drives
  • Unable to mount location Failed to retrieve share list from server: No such file or directory solution
  • mdadm how to make inactive array active
  • ImageMagick how to trim white space automatically in Linux
  • curl: (1) Protocol "https not supported or disabled in libcurl"
  • Centos 5 OpenSSL does not support TLS 1.2 Apache Error
  • DRBD Split-brain solution
  • How to Properly Secure SSL/TLS Apache Settings against Heartbleed Poodle (TLS) Poodle (SSLv3) FREAK BEAST CRIME
  • K9 Mail Android Cannot See or View E-mails Disappear after reading - with Dovecot server. Solution
  • The folder contents could not be displayed connection refused - solution
  • Setting Up System for First Use... Please Wait... - WHMCS Installer
  • ERROR 2013 (HY000): Lost connection to MySQL server during query
  • if script bash check if socket file (mysql.sock) exists
  • ioncube loader install howto on PHP/Centos