How to Track Down/Find Cause of Linux high disk iowait/wa -

How to Track Down/Find Cause of Linux high disk iowait/wa

top - 09:34:12 up 2 days, 20:57,  2 users,  load average: 1.83, 1.99, 2.03
Tasks:  59 total,   2 running,  57 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.0%sy,  0.0%ni,  0.0%id, 99.7%wa,  0.0%hi,  0.0%si,  0.0%st

That 99.7% wa is iowait, it means the server is waiting for a process to complete an IO operation or in plain English, there is a delay in reading and/or writing data to the hard drive.

Here are some ways to get an idea of what is going on (in RPM based Distros this package is called "sysstat") and contains the very helpful binary called "iostat".  There's another package I like called "dstat" that does a great job too.

Run iostat -d 5 (it refreshes the data every 5 seconds):

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1              2.07         5.27        28.66    1309570    7120384
sda2              0.00         0.03         0.04       7496       8832

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda1             27.15         0.00       357.68          0       1792
sda2              0.00         0.00         0.00          0          0

What we're seeing is that the iowait must be caused primarily by blocks being written to /dev/sda1 which is mounted as / (my root filesystem).  So now we know the main cause is by an application constantly writing to the root partition, but we still don't know what process is directly responsible for this.

If you have a newer kernel it's often possible to view processes by io usage, you can test for this by running: cat /proc/self/io

If the file is not found then you don't have io stats enabled in your kernel.  Then you can check a suspected process by running cat /proc/13/io (replace the 13 with the PID you want to check).

read_bytes: 0
write_bytes: 0
cancelled_write_bytes: 0

 


  • Migrate Centos 7 from Single HDD to mdadm RAID 10 array:
  • How to change reserved blocks in Linux partition
  • Reading package lists... Done W: GPG error: http://ppa.launchpad.net trusty InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY D46F45428842CE5E
  • Centos 7 A start job is running for dev-mapper-clx2droot.device (8min 44s / no limit)
  • USB 3.0 External HDD Enclosure Seagate UAS problems - [sdd] tag#1 CDB: Write(16) 8a 00 00 00 00 01 70 04 08 68 00 00 00 08 00 00
  • Centos 7 Cudaminer Nvidia setup guide
  • USB 3.0 PCI x1 Card Review VIA VL805 on Linux Review and Experience
  • rsync run as root sudo without password
  • Why won't my Linux Mint boot after I manually installed a new kernel?
  • cudaminer freezes Linux Mint taints kernel
  • Linux Mint Intel Screen Tearing Horizontal Line Solution Fix
  • process_usershare_file: stat of /var/lib/samba/usershares/dump failed. Permission denied - samba error solution
  • imagemagick pdf convert error solution - **** Warning: can't process font stream, loading font by the name. Can't find CMap Identity-UTF16-H building a CIDDecoding resource. Warning: falling back to Identity ordering **** Error reading a conte
  • php read file into variable
  • USB Device/MP3 Player not working properly and can't browse the device filesystem? Check the cable twice and again!
  • Centos/Linux Bash Script Warning when exiting/logging out of bash/shell/terminal for scripts running in the background
  • Wifi Antenna Extension Cable is called RP-SMA
  • Nvidia Linux Mint/Ubuntu screentearing horizontal line solution
  • Openshot 2.2 is very unstable always crashing in Linux Mint 18.1!
  • Apache SSL very slow response with Firefox Freezes/Loads Very slow when checking self-signed SSL certificate