How to Track Down/Find Cause of Linux high disk iowait/wa

top - 09:34:12 up 2 days, 20:57,  2 users,  load average: 1.83, 1.99, 2.03
Tasks:  59 total,   2 running,  57 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.0%sy,  0.0%ni,  0.0%id, 99.7%wa,  0.0%hi,  0.0%si,  0.0%st

That 99.7% wa is iowait, it means the server is waiting for a process to complete an IO operation or in plain English, there is a delay in reading and/or writing data to the hard drive.

Here are some ways to get an idea of what is going on (in RPM based Distros this package is called "sysstat") and contains the very helpful binary called "iostat".  There's another package I like called "dstat" that does a great job too.

Run iostat -d 5 (it refreshes the data every 5 seconds):

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1              2.07         5.27        28.66    1309570    7120384
sda2              0.00         0.03         0.04       7496       8832

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1             27.15         0.00       357.68          0       1792
sda2              0.00         0.00         0.00          0          0

What we're seeing is that the iowait must be caused primarily by blocks being written to /dev/sda1 which is mounted as / (my root filesystem).  So now we know the main cause is by an application constantly writing to the root partition, but we still don't know what process is directly responsible for this.

If you have a newer kernel it's often possible to view processes by io usage, you can test for this by running: cat /proc/self/io

If the file is not found then you don't have io stats enabled in your kernel.  Then you can check a suspected process by running cat /proc/13/io (replace the 13 with the PID you want to check).

read_bytes: 0
write_bytes: 0
cancelled_write_bytes: 0

 


Tags:

linux, disk, iowait, watop, users, tasks, cpu, sy, ni, wa, server, io, rpm, distros, quot, sysstat, contains, binary, iostat, dstat, refreshes, tps, blk_read, blk_wrtn, sda, primarily, dev, mounted, filesystem, partition, newer, kernel, processes, usage, proc, stats, enabled, pid, read_bytes, write_bytes, cancelled_write_bytes,

Latest Articles

  • Enable AMDGPU Linux Driver
  • apache symlinks denied even with followsymlinks
  • chown how to change ownership on a symlink
  • how to use ifplugd in Linux to execute a command or script when a NIC cable is unplugged or plugged in
  • dd how to backup and restore disk images including compression with gzip
  • mpv / mplayer with Radeon / AMD GPU Video Card Driver enable VDPAU GPU Accelerated Video Decoding
  • Wordpress Reset Blog User Password from MySQL Using Linux Bash and not PHPMyadmin
  • Ubuntu Linux Mint Debian xorg performance and tear-free tuning for AMD Radeon Based Cards
  • Centos 7 Stopped and Disabled Firewalld and ports still blocked
  • MariaDB / MySQL Reset Root Forgotten Password on Centos 7
  • Centos 7 How to install Mysql/Mariadb
  • PHP 7.2, Apache and Centos 7 How To Install
  • Ubuntu Debian Linux Mint r8169 r8168 Network Driver Problem and Solution
  • Linux 3D Performance benchmarks with glxgears 59-60fps solution
  • How To Install Asterisk 16 17 on Debian Ubuntu Linux
  • Linux Ubuntu Debian Centos How To Make a Bootable Windows 7, 8, 10, 2016, 2019 Server USB from ISO
  • How To Restore Windows MBR Bootsector from Linux using syslinux
  • Linux Ubuntu Cannot Print Large Images
  • Cannot Print PDF Solution and Howto Resize
  • Linux Console Login Screen TTY Change Message