DRBD Slow Performance - 99.99 % [jbd2/drbd0-8] highiowait solution -

DRBD Slow Performance - 99.99 % [jbd2/drbd0-8] highiowait solution

Tired of checking iotop and seeing that your drbd partition is using 99.99% of io all the time and finding your drbd device performs slow in general?

This is especially an issue in versions of DRBD in the 8.3 tree in particular one documented case is on "8.3.13" but it likely applies to other devices.

The symptoms are that resyncing is fine and normal but any reasonable amount of activity is very slow and lagged and creates a high server load and consequently high io wait.  You may not notice it though until you apply a reasonable load or usage to the server and DRBD device.

In some kernels and some versions you may get the following error in dmesg

block drbd0: [drbd0_worker/1670] sock_sendmsg time expired, ko = 4294967295

*But we have seen many cases where the above error is not present (perhaps an older kernel or module does not recognize it is being blocked).

Here are some quick sysctl.conf kernel tuning tips that have changed a server's load from 10-18, to less than 1.

Add or edit these in sysctl.conf to solve the issue

net.ipv4.tcp_rmem = 131072  131072  10485760
net.ipv4.tcp_wmem = 131072  131072  10485760
vm.dirty_ratio = 10
vm.dirty_background_ratio = 4

 

This is a preferable way rather than hacking or changing DRBD settings by upgrading the userland or kernel manually or while it runs (results can be unpredictable) whereas this kernel tuning has virtually no risk or impact on DRBD except a positive one.


  • MySQL Adding New Field to Existing Database Table
  • MySQL Cannot Update/Write to any database table solution
  • Centos How To Update to Glibc 2.14 Plus
  • php remove last letters of string
  • MySQL Maximum INT Size Truncation Issue/Warning
  • MySQL How To Add New Field Column To Existing Table
  • mysql how to reset passwords with a few commands
  • htaccess apache how deny/allow to block or allow by IP address
  • PHP geoip.so fatal error Solution
  • Syntax error on line 221 of /etc/httpd/conf/httpd.conf: Syntax error on line 6 of /etc/httpd/conf.d/php.conf: Cannot load /etc/httpd/modules/libphp5.so into server: /etc/httpd/modules/libphp5.so: cannot open shared object file: No such file or direct
  • GCC 5 on Centos 6 - How To Install
  • bash find line and replace howto
  • How To Create Apache htpasswd file
  • possible SYN flooding on ctid 42131, port 80. Sending cookies. - Solution
  • Linux last command show login by IP instead of hostname
  • Install Windows From a Linux TFTP Server instead of using WDS Solution
  • How To Secure Samba NMBD/SMBD to bind to a specific IP address
  • tftp: client does not accept options - solution if you are using UEFI PXEBoot disable it!
  • Linux how to view video card make and exact model
  • Authentication refused: bad ownership or modes for directory /home/user SSH Public Key Authentication Failed Solution