How to Track Down/Find Cause of Linux high disk iowait/wa -

How to Track Down/Find Cause of Linux high disk iowait/wa

top - 09:34:12 up 2 days, 20:57,  2 users,  load average: 1.83, 1.99, 2.03
Tasks:  59 total,   2 running,  57 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.0%sy,  0.0%ni,  0.0%id, 99.7%wa,  0.0%hi,  0.0%si,  0.0%st

That 99.7% wa is iowait, it means the server is waiting for a process to complete an IO operation or in plain English, there is a delay in reading and/or writing data to the hard drive.

Here are some ways to get an idea of what is going on (in RPM based Distros this package is called "sysstat") and contains the very helpful binary called "iostat".  There's another package I like called "dstat" that does a great job too.

Run iostat -d 5 (it refreshes the data every 5 seconds):

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1              2.07         5.27        28.66    1309570    7120384
sda2              0.00         0.03         0.04       7496       8832

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1             27.15         0.00       357.68          0       1792
sda2              0.00         0.00         0.00          0          0

What we're seeing is that the iowait must be caused primarily by blocks being written to /dev/sda1 which is mounted as / (my root filesystem).  So now we know the main cause is by an application constantly writing to the root partition, but we still don't know what process is directly responsible for this.

If you have a newer kernel it's often possible to view processes by io usage, you can test for this by running: cat /proc/self/io

If the file is not found then you don't have io stats enabled in your kernel.  Then you can check a suspected process by running cat /proc/13/io (replace the 13 with the PID you want to check).

read_bytes: 0
write_bytes: 0
cancelled_write_bytes: 0

 


  • Linux Mint 18 Screen Goes Dark or Black After Screensaver or even when using the Desktop Solution
  • iptables guide and examples and howto
  • Postfix How To Change Sending IP Address To Specific IP Binding or Interface
  • How to qemu-kvm enable bridged networking in Debian Ubuntu Linux Mint on KVM containers
  • VirtualBox Nested Virtual Machine Containers with KVM Not Working no SVM or VMX module in the guest
  • VSphere InternalServerError - Error When Adding Permissions
  • NFS Share Won't Mount Solution - mount: wrong fs type, bad option, bad superblock on 10.10.2.20:/tmp/nfsmount
  • OVF Tool: Error: Task failed on server: This host does not support Intel VT-x. VMWare VCenter install On ESXi ERror
  • Install NFS (Network File System) On Debian Linux Ubuntu Mint Howto
  • Relocating modules and starting up the kernel - VMWare ESXi 6.7 Error and Solution
  • VMWare 6.7 VCSA VSphere ESXi Management SSO Install Guide on Linux using the CLI
  • Hard Drive Serial Number Examples
  • vino server error cannot login
  • OpenVPN auth-user-pass-verify ENV script error
  • OpenVPN error ERROR: Cannot ioctl TUNSETIFF tun: Operation not permitted (errno=1)
  • Howto install ioncube loader to PHP by Zend
  • Cannot create gradle for conversations
  • not allowed to execute '/usr/bin/apt-get install eclipse' as root linux sudo user permisson issue and solution
  • Database Error One or more of the WHMCS database tables appear to be either missing or corrupted. Please check and repair. - WHMCS Solution
  • postfix errors fatal: no SASL authentication mechanisms /usr/libexec/postfix/smtpd: bad command startup -- throttling solution