How to Track Down/Find Cause of Linux high disk iowait/wa

top - 09:34:12 up 2 days, 20:57,  2 users,  load average: 1.83, 1.99, 2.03
Tasks:  59 total,   2 running,  57 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.0%sy,  0.0%ni,  0.0%id, 99.7%wa,  0.0%hi,  0.0%si,  0.0%st

That 99.7% wa is iowait, it means the server is waiting for a process to complete an IO operation or in plain English, there is a delay in reading and/or writing data to the hard drive.

Here are some ways to get an idea of what is going on (in RPM based Distros this package is called "sysstat") and contains the very helpful binary called "iostat".  There's another package I like called "dstat" that does a great job too.

Run iostat -d 5 (it refreshes the data every 5 seconds):

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1              2.07         5.27        28.66    1309570    7120384
sda2              0.00         0.03         0.04       7496       8832

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1             27.15         0.00       357.68          0       1792
sda2              0.00         0.00         0.00          0          0

What we're seeing is that the iowait must be caused primarily by blocks being written to /dev/sda1 which is mounted as / (my root filesystem).  So now we know the main cause is by an application constantly writing to the root partition, but we still don't know what process is directly responsible for this.

If you have a newer kernel it's often possible to view processes by io usage, you can test for this by running: cat /proc/self/io

If the file is not found then you don't have io stats enabled in your kernel.  Then you can check a suspected process by running cat /proc/13/io (replace the 13 with the PID you want to check).

read_bytes: 0
write_bytes: 0
cancelled_write_bytes: 0

 


Tags:

linux, disk, iowait, watop, users, tasks, cpu, sy, ni, wa, server, io, rpm, distros, quot, sysstat, contains, binary, iostat, dstat, refreshes, tps, blk_read, blk_wrtn, sda, primarily, dev, mounted, filesystem, partition, newer, kernel, processes, usage, proc, stats, enabled, pid, read_bytes, write_bytes, cancelled_write_bytes,

Latest Articles

  • Linux and Windows Dual Boot Issue NIC Won't work After Booting Windows
  • Cisco CME How To Enable ACD hunt groups
  • How to install gns3 on Linux Ubuntu Mint
  • How to convert audio for Asterisk .wav format
  • Using Cisco CME Router with Asterisk as a dial-peer
  • Cisco CME How To Configure SIP Trunk VOIP
  • Virtualbox host Only Network Error Failed to save host network interface parameter - Cannot change gateway IP of host only network
  • Cisco CME and C7200 Router Testing and Learning Environment on Ubuntu 20+ Setup Tutorial Guide
  • Abusive IP ranges blacklist
  • How to Install Any OS on a Physical Drive from Windows Using VMware Workstation (Linux, Windows, BSD)
  • CDN Cloudflare how to set and preserve the real IP of the client without modifying application code on Apache
  • CentOS 7 fix Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was 14: curl#6 -
  • Ubuntu Debian How To Install Recommended Packages Automatically
  • How to set Linux Ubuntu Redhat Debian Command Line http https socks proxy for yum apt
  • How to resize a pdf without losing much quality in Linux Mint Ubuntu Debian Redhat Solution
  • qemu: could not load PC BIOS 'bios-256k.bin' solution
  • Proxmox How To Custom Partition During Install
  • Hyper-V Linux VM Boots to Black Screen, Storage, NIC Not Found Issues
  • Ubuntu Mint How to Fix Missing/Broken /dev and /dev/pts which causes terminal to immediately close exit and not work
  • How high can a Xeon CPU get?