DRBD Slow Performance - 99.99 % [jbd2/drbd0-8] highiowait solution -

DRBD Slow Performance - 99.99 % [jbd2/drbd0-8] highiowait solution

Tired of checking iotop and seeing that your drbd partition is using 99.99% of io all the time and finding your drbd device performs slow in general?

This is especially an issue in versions of DRBD in the 8.3 tree in particular one documented case is on "8.3.13" but it likely applies to other devices.

The symptoms are that resyncing is fine and normal but any reasonable amount of activity is very slow and lagged and creates a high server load and consequently high io wait.  You may not notice it though until you apply a reasonable load or usage to the server and DRBD device.

In some kernels and some versions you may get the following error in dmesg

block drbd0: [drbd0_worker/1670] sock_sendmsg time expired, ko = 4294967295

*But we have seen many cases where the above error is not present (perhaps an older kernel or module does not recognize it is being blocked).

Here are some quick sysctl.conf kernel tuning tips that have changed a server's load from 10-18, to less than 1.

Add or edit these in sysctl.conf to solve the issue

net.ipv4.tcp_rmem = 131072  131072  10485760
net.ipv4.tcp_wmem = 131072  131072  10485760
vm.dirty_ratio = 10
vm.dirty_background_ratio = 4

 

This is a preferable way rather than hacking or changing DRBD settings by upgrading the userland or kernel manually or while it runs (results can be unpredictable) whereas this kernel tuning has virtually no risk or impact on DRBD except a positive one.


  • wget how to resume download!
  • strange vi errors in Linux Mint/Ubuntu line 58: E488: Trailing characters: t_Sbet line 63: E171: Missing :endif
  • MySQL Add multiple IPs for remote user including root howto
  • cPanel error Access denied for user 'root'@'localhost' when adding remote MySQL IP address solution
  • Text Editors - Top Linux Ubuntu/Debian/Mint Options
  • Linux shows my drive as being dead is it really? Buffer I/O error on device sdb, logical block 0 sd 3:0:0:0: [sdb] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
  • use ffmpeg to watermark videos if mencoder/bmovl fail
  • mencoder bmovl error vf_bmovl: Unknown command: ''. Ignoring.
  • Linux Mint/Ubuntu/Debian Nvidia driver becomes slow graphics performance issue
  • Linux Mint missing ffmpeg solution to install from Ubuntu PPA
  • phpBB3 slow and internal dummy connection
  • check if Apache uses worker MPM or prefork MPM
  • text to speech for Linux
  • How to convert xlsx/xls Excel file into csv
  • Linux bash script how to generate a random password using /dev/urandom
  • ssh forward multiple ports in the same connection and command even works with NAT!
  • How to create openssl md5 password hash to use in /etc/shadow using bash
  • Avocent DSR8020 KVM/IP - Network Connect Error - Solution
  • PHP Parse error: syntax error, unexpected '[' in phpBB3/vendor/react/promise/src/functions.php on line 15
  • tar extraction changes ownership of /root directory