DRBD Slow Performance - 99.99 % [jbd2/drbd0-8] highiowait solution

Tired of checking iotop and seeing that your drbd partition is using 99.99% of io all the time and finding your drbd device performs slow in general?

This is especially an issue in versions of DRBD in the 8.3 tree in particular one documented case is on "8.3.13" but it likely applies to other devices.

The symptoms are that resyncing is fine and normal but any reasonable amount of activity is very slow and lagged and creates a high server load and consequently high io wait.  You may not notice it though until you apply a reasonable load or usage to the server and DRBD device.

In some kernels and some versions you may get the following error in dmesg

block drbd0: [drbd0_worker/1670] sock_sendmsg time expired, ko = 4294967295

*But we have seen many cases where the above error is not present (perhaps an older kernel or module does not recognize it is being blocked).

Here are some quick sysctl.conf kernel tuning tips that have changed a server's load from 10-18, to less than 1.

Add or edit these in sysctl.conf to solve the issue

net.ipv4.tcp_rmem = 131072  131072  10485760
net.ipv4.tcp_wmem = 131072  131072  10485760
vm.dirty_ratio = 10
vm.dirty_background_ratio = 4

 

This is a preferable way rather than hacking or changing DRBD settings by upgrading the userland or kernel manually or while it runs (results can be unpredictable) whereas this kernel tuning has virtually no risk or impact on DRBD except a positive one.

Latest Articles

  • prosody xmpp server "Failed in all attempts to connect to proxy.eu.jabber.org"
  • VMWare ESXi 6.7 SSH/PowerShell CLI Commands
  • VMWare Vsphere VCSA Graphical Install Creates json
  • error while loading shared libraries: libasound.so.2: cannot open shared object file: No such file or directory
  • ./xmr-stak: error while loading shared libraries: libmicrohttpd.so.10: cannot open shared object file: No such file or directory
  • qemu-img resize howto
  • gmail.com address failing in Postfix
  • VMWare Hardware virtualization is selected and cannot be deselected due to selection of VBS Insufficient resources to satisfy configured failover level for vSphere HA.
  • Debian Mint Ubuntu compiling xmr-stak
  • ./nsgpucnminer: error while loading shared libraries: libOpenCL.so.1w: cannot open shared object file: No such file or directory
  • *** These critical programs are missing or too old: gawk
  • migration/4 migration 4 is using too much CPU
  • convert.im6: Unknown field with tag 317 (0x13d) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/788.
  • Python SyntaxError: Missing parentheses in call to 'print'
  • Linux How To Rename Files
  • OpenVZ container will not stop Child 546246 exited with status 1
  • [warn] VirtualHost 10.2.5.101:443 overlaps with VirtualHost 10.2.5.101:443, the first has precedence, perhaps you need a NameVirtualHost directive
  • Ethereum Client Errors
  • PayPal IPN Failed "result: IPN Handshake Invalid"
  • Linux Centos 7 HowTo Install Yarn