DRBD Slow Performance - 99.99 % [jbd2/drbd0-8] highiowait solution -

DRBD Slow Performance - 99.99 % [jbd2/drbd0-8] highiowait solution

Tired of checking iotop and seeing that your drbd partition is using 99.99% of io all the time and finding your drbd device performs slow in general?

This is especially an issue in versions of DRBD in the 8.3 tree in particular one documented case is on "8.3.13" but it likely applies to other devices.

The symptoms are that resyncing is fine and normal but any reasonable amount of activity is very slow and lagged and creates a high server load and consequently high io wait.  You may not notice it though until you apply a reasonable load or usage to the server and DRBD device.

In some kernels and some versions you may get the following error in dmesg

block drbd0: [drbd0_worker/1670] sock_sendmsg time expired, ko = 4294967295

*But we have seen many cases where the above error is not present (perhaps an older kernel or module does not recognize it is being blocked).

Here are some quick sysctl.conf kernel tuning tips that have changed a server's load from 10-18, to less than 1.

Add or edit these in sysctl.conf to solve the issue

net.ipv4.tcp_rmem = 131072  131072  10485760
net.ipv4.tcp_wmem = 131072  131072  10485760
vm.dirty_ratio = 10
vm.dirty_background_ratio = 4

 

This is a preferable way rather than hacking or changing DRBD settings by upgrading the userland or kernel manually or while it runs (results can be unpredictable) whereas this kernel tuning has virtually no risk or impact on DRBD except a positive one.


  • zip warning: name not matched: in Linux Solution
  • ls in Linux with full directory path with files howto
  • bash how to check if a symlink exists as a file or directory
  • css .ul and .li how to remove the identation padding/margin from list items?
  • how to extract .xz file in Linux Centos/Debian/Ubuntu
  • Linux how to whois query other gTLDs such as .club solution
  • How to verify SSL SHA-1 Certificate Fingerprnit Signature of your mail/web server to avoid hijacking/man-in-the-middle attacks
  • Linux Ubuntu Mint how to view RAW image files .arw?
  • ecryptfs errors
  • What happens when you unplug 1 or more devices from an mdadm RAID array to simulate a failure in Linux Ubuntu/Centos/Debian?
  • Migrating from Linux Mint 17.2 mdadm RAID array to a new one (because I upgraded to larger drives).
  • Linux how to reformat html file code solution
  • ENOM how to do a 301 redirect to another domain/site
  • ENOM TRANSFER passwords do not match - solution
  • Linux how to check http headers using bash/curl for SEO
  • mdadm: CREATE group disk not found Incrementally started RAID arrays. Incrementally starting RAID arrays...
  • Linux SAMBA does not work with symlinks
  • How to Execute PHP in .html files with Apache in Linux Centos/Debian/Ubuntu etc
  • mdadm how to recover from failed drive
  • yum Packages skipped because of dependency problems: