How to Track Down/Find Cause of Linux high disk iowait/wa

top - 09:34:12 up 2 days, 20:57,  2 users,  load average: 1.83, 1.99, 2.03
Tasks:  59 total,   2 running,  57 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.0%sy,  0.0%ni,  0.0%id, 99.7%wa,  0.0%hi,  0.0%si,  0.0%st

That 99.7% wa is iowait, it means the server is waiting for a process to complete an IO operation or in plain English, there is a delay in reading and/or writing data to the hard drive.

Here are some ways to get an idea of what is going on (in RPM based Distros this package is called "sysstat") and contains the very helpful binary called "iostat".  There's another package I like called "dstat" that does a great job too.

Run iostat -d 5 (it refreshes the data every 5 seconds):

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1              2.07         5.27        28.66    1309570    7120384
sda2              0.00         0.03         0.04       7496       8832

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1             27.15         0.00       357.68          0       1792
sda2              0.00         0.00         0.00          0          0

What we're seeing is that the iowait must be caused primarily by blocks being written to /dev/sda1 which is mounted as / (my root filesystem).  So now we know the main cause is by an application constantly writing to the root partition, but we still don't know what process is directly responsible for this.

If you have a newer kernel it's often possible to view processes by io usage, you can test for this by running: cat /proc/self/io

If the file is not found then you don't have io stats enabled in your kernel.  Then you can check a suspected process by running cat /proc/13/io (replace the 13 with the PID you want to check).

read_bytes: 0
write_bytes: 0
cancelled_write_bytes: 0

 


Tags:

linux, disk, iowait, watop, users, tasks, cpu, sy, ni, wa, server, io, rpm, distros, quot, sysstat, contains, binary, iostat, dstat, refreshes, tps, blk_read, blk_wrtn, sda, primarily, dev, mounted, filesystem, partition, newer, kernel, processes, usage, proc, stats, enabled, pid, read_bytes, write_bytes, cancelled_write_bytes,

Latest Articles

  • Nvidia Datacenter Driver Tesla Slow nvidia-smi response and high utilization with 0 usage
  • ffmpeg how to normalize / increase the volume of your audio
  • kdenlive audio blips pops cracks artifacts solution fix
  • haproxy / nginx certbot SSL issues
  • nginx how to see the real IP when behind a CDN
  • Docker how to find real container child process ID
  • Alibaba Aliyun how to reset password solution 'Setup does not meet the requirements, please resetting'
  • RTL88X Series 80Mhz hostapd mode for Linux Debian Kali
  • How To Deploy Your Own Mastodon Server in Docker
  • ffmpeg burning subtitles in non-English errors [Parsed_subtitles_0 @ 0x561d3a0b3b80] Glyph 0x6709 not found, selecting one more font for (Sans, 700, 0)
  • rsyslog in container config
  • Interesting Whisper AI CPU vs GPU Test
  • How to install pytorch with cuda capability for AI acceleration with Nvidia Tesla etc.. GPUs
  • How to Spider the web archive.org to recover your old website/webpage
  • Debian 10 /etc/apt/sources.list
  • Linux Debian Cannot Upgrade Kernel Headers Missing Solution
  • How to install ZFS on Linux Ubuntu Debian Mint
  • How to Add Contrib Packages to Debian/Linux/Mint by editing /etc/apt/sources.list
  • Proxmox How To Purge Ceph
  • VMWare ESXi/VSphere Disable Balloon Segfault in Services Solution