How to Track Down/Find Cause of Linux high disk iowait/wa

top - 09:34:12 up 2 days, 20:57,  2 users,  load average: 1.83, 1.99, 2.03
Tasks:  59 total,   2 running,  57 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.0%sy,  0.0%ni,  0.0%id, 99.7%wa,  0.0%hi,  0.0%si,  0.0%st

That 99.7% wa is iowait, it means the server is waiting for a process to complete an IO operation or in plain English, there is a delay in reading and/or writing data to the hard drive.

Here are some ways to get an idea of what is going on (in RPM based Distros this package is called "sysstat") and contains the very helpful binary called "iostat".  There's another package I like called "dstat" that does a great job too.

Run iostat -d 5 (it refreshes the data every 5 seconds):

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1              2.07         5.27        28.66    1309570    7120384
sda2              0.00         0.03         0.04       7496       8832

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1             27.15         0.00       357.68          0       1792
sda2              0.00         0.00         0.00          0          0

What we're seeing is that the iowait must be caused primarily by blocks being written to /dev/sda1 which is mounted as / (my root filesystem).  So now we know the main cause is by an application constantly writing to the root partition, but we still don't know what process is directly responsible for this.

If you have a newer kernel it's often possible to view processes by io usage, you can test for this by running: cat /proc/self/io

If the file is not found then you don't have io stats enabled in your kernel.  Then you can check a suspected process by running cat /proc/13/io (replace the 13 with the PID you want to check).

read_bytes: 0
write_bytes: 0
cancelled_write_bytes: 0

 


Tags:

linux, disk, iowait, watop, users, tasks, cpu, sy, ni, wa, server, io, rpm, distros, quot, sysstat, contains, binary, iostat, dstat, refreshes, tps, blk_read, blk_wrtn, sda, primarily, dev, mounted, filesystem, partition, newer, kernel, processes, usage, proc, stats, enabled, pid, read_bytes, write_bytes, cancelled_write_bytes,

Latest Articles

  • Grandstream Phone Vulnerability Security Issue Remote Backdoor Connection to 207.246.119.209:3478
  • Linux How to Check Which NIC is Onboard eth0 or eth1 Ubuntu Centos Debian Mint
  • VboxManage VirtualBox NAT Network Issues Managment Troubleshooting
  • Dell PowerEdge Server iDRAC Remote KVM/IP Default Username, Password Reset and Login Information Solution
  • Nvidia Tesla GPUs K40/K80/M40/P40/P100/V100 at home/desktop hacking, cooling, powering, cable solutions Tutorial AIO Solutions
  • Stop ls in Linux Debian Mint CentOS Ubuntu from applying quotes around filenames and directory names
  • Thunderbird Attachment Download Error Corrupt Wrong filesize of 29 or 27 bytes Solution
  • Generic IP Camera LAN Default IP Settings DVR
  • Ubuntu Debian Mint Linux How To Update Initramfs Manually update-initramfs
  • Enable Turbo Mode for CPU Ubuntu Linux Mint Debian Redhat
  • docker / kubernetes breaks Proxmox QEMU KVM Bridge VMs
  • How To Change Storage Location in Docker.io
  • RTL8812BU and RTL8822BU Linux Driver Ubuntu Setup Archer T3U Plus
  • Kazam video blank/high size and not working when recording solution
  • Cisco UC CME How To Enable Licensed Features
  • from pip._internal.cli.main import main File "/usr/local/lib/python3.5/dist-packages/pip/_internal/cli/main.py", line 60 sys.stderr.write(f"ERROR: {exc}") from pip._internal.cli.main import main File "/usr/local/lib/python3.5/dist-packag
  • ModuleNotFoundError: No module named 'pip._internal' solution python
  • grub blank screen how to manually boot kernel and initrd Linux Ubuntu Debian Centos won't boot solution
  • Cisco Switch / Router How To Restore Factory Default Settings
  • Cisco 2900 3900 Router Password Reset How To Reset Enable Password