How to Track Down/Find Cause of Linux high disk iowait/wa -

How to Track Down/Find Cause of Linux high disk iowait/wa

top - 09:34:12 up 2 days, 20:57,  2 users,  load average: 1.83, 1.99, 2.03
Tasks:  59 total,   2 running,  57 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.0%sy,  0.0%ni,  0.0%id, 99.7%wa,  0.0%hi,  0.0%si,  0.0%st

That 99.7% wa is iowait, it means the server is waiting for a process to complete an IO operation or in plain English, there is a delay in reading and/or writing data to the hard drive.

Here are some ways to get an idea of what is going on (in RPM based Distros this package is called "sysstat") and contains the very helpful binary called "iostat".  There's another package I like called "dstat" that does a great job too.

Run iostat -d 5 (it refreshes the data every 5 seconds):

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1              2.07         5.27        28.66    1309570    7120384
sda2              0.00         0.03         0.04       7496       8832

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1             27.15         0.00       357.68          0       1792
sda2              0.00         0.00         0.00          0          0

What we're seeing is that the iowait must be caused primarily by blocks being written to /dev/sda1 which is mounted as / (my root filesystem).  So now we know the main cause is by an application constantly writing to the root partition, but we still don't know what process is directly responsible for this.

If you have a newer kernel it's often possible to view processes by io usage, you can test for this by running: cat /proc/self/io

If the file is not found then you don't have io stats enabled in your kernel.  Then you can check a suspected process by running cat /proc/13/io (replace the 13 with the PID you want to check).

read_bytes: 0
write_bytes: 0
cancelled_write_bytes: 0

 


  • zip warning: name not matched: in Linux Solution
  • ls in Linux with full directory path with files howto
  • bash how to check if a symlink exists as a file or directory
  • css .ul and .li how to remove the identation padding/margin from list items?
  • how to extract .xz file in Linux Centos/Debian/Ubuntu
  • Linux how to whois query other gTLDs such as .club solution
  • How to verify SSL SHA-1 Certificate Fingerprnit Signature of your mail/web server to avoid hijacking/man-in-the-middle attacks
  • Linux Ubuntu Mint how to view RAW image files .arw?
  • ecryptfs errors
  • What happens when you unplug 1 or more devices from an mdadm RAID array to simulate a failure in Linux Ubuntu/Centos/Debian?
  • Migrating from Linux Mint 17.2 mdadm RAID array to a new one (because I upgraded to larger drives).
  • Linux how to reformat html file code solution
  • ENOM how to do a 301 redirect to another domain/site
  • ENOM TRANSFER passwords do not match - solution
  • Linux how to check http headers using bash/curl for SEO
  • mdadm: CREATE group disk not found Incrementally started RAID arrays. Incrementally starting RAID arrays...
  • Linux SAMBA does not work with symlinks
  • How to Execute PHP in .html files with Apache in Linux Centos/Debian/Ubuntu etc
  • mdadm how to recover from failed drive
  • yum Packages skipped because of dependency problems: