How to Track Down/Find Cause of Linux high disk iowait/wa -

How to Track Down/Find Cause of Linux high disk iowait/wa

top - 09:34:12 up 2 days, 20:57,  2 users,  load average: 1.83, 1.99, 2.03
Tasks:  59 total,   2 running,  57 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.0%sy,  0.0%ni,  0.0%id, 99.7%wa,  0.0%hi,  0.0%si,  0.0%st

That 99.7% wa is iowait, it means the server is waiting for a process to complete an IO operation or in plain English, there is a delay in reading and/or writing data to the hard drive.

Here are some ways to get an idea of what is going on (in RPM based Distros this package is called "sysstat") and contains the very helpful binary called "iostat".  There's another package I like called "dstat" that does a great job too.

Run iostat -d 5 (it refreshes the data every 5 seconds):

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1              2.07         5.27        28.66    1309570    7120384
sda2              0.00         0.03         0.04       7496       8832

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1             27.15         0.00       357.68          0       1792
sda2              0.00         0.00         0.00          0          0

What we're seeing is that the iowait must be caused primarily by blocks being written to /dev/sda1 which is mounted as / (my root filesystem).  So now we know the main cause is by an application constantly writing to the root partition, but we still don't know what process is directly responsible for this.

If you have a newer kernel it's often possible to view processes by io usage, you can test for this by running: cat /proc/self/io

If the file is not found then you don't have io stats enabled in your kernel.  Then you can check a suspected process by running cat /proc/13/io (replace the 13 with the PID you want to check).

read_bytes: 0
write_bytes: 0
cancelled_write_bytes: 0

 


  • Centos 6.6/6.9 KVM VM Kernel Panic On Boot - Kernel panic - not syncing: Attempted to kill init!
  • Cannot allocate TUN/TAP dev dynamically - FreeBSD/OpenBSD OpenVPN Client error Solution
  • Linux Mint/Ubuntu/Debian apt how to downgrade a package
  • Ubuntu Linux Mint How To Exclude Stop Package from being Upgraded or Installed
  • Intel NUC Lower Power Green Computing Boxes Review/Comparison of J3160 and J3455 Models
  • How to backup entire computer Linux Mint Ubuntu with tar
  • tar how to ignore failed reads and not exit
  • kdenlive titles/text renders as white screen when using .sh script
  • [1035724.274610] [drm:intel_pipe_update_end [i915_bpo]] *ERROR* Atomic update failure on pipe A (start=62076478 end=62076479) time 102 us, min 894, max 899, scanline start 893, end 900 W: Possible missing firmware /lib/firmware/i915/kbl_dmc_ver1.bin
  • ffmpeg how to watermark and concatenate in one command
  • /usr/bin/ld: cannot find -lboost_system-mt-s /usr/bin/ld: cannot find -lboost_filesystem-mt-s /usr/bin/ld: cannot find -lboost_program_options-mt-s /usr/bin/ld: cannot find -lboost_thread-mt-s collect2: error: ld returned 1 exit status make: *** [cag
  • Wine uninstalled broken on Linux Mint
  • ffmpeg trouble concatenating similar but different files
  • ffmpeg Unable to Use Hardware Encoding with Nvidia 3.40 Driver and GT210 card
  • Linux Mint USB Kernel Tainted and Locked Port/Dev File
  • ffmpeg Linux Mint download, compile and install howto
  • OpenVZ error : Container start failed (try to check kernel messages, e.g. "dmesg | tail") Locked by: pid 166638, cmdline vzctl start 888171
  • How to extract view contents of initramfs image gzip'd
  • Linux how to copy GPT partition table with dd
  • Centos 7 How To Change Hostname