DRBD Slow Performance - 99.99 % [jbd2/drbd0-8] highiowait solution

Tired of checking iotop and seeing that your drbd partition is using 99.99% of io all the time and finding your drbd device performs slow in general?

This is especially an issue in versions of DRBD in the 8.3 tree in particular one documented case is on "8.3.13" but it likely applies to other devices.

The symptoms are that resyncing is fine and normal but any reasonable amount of activity is very slow and lagged and creates a high server load and consequently high io wait.  You may not notice it though until you apply a reasonable load or usage to the server and DRBD device.

In some kernels and some versions you may get the following error in dmesg

block drbd0: [drbd0_worker/1670] sock_sendmsg time expired, ko = 4294967295

*But we have seen many cases where the above error is not present (perhaps an older kernel or module does not recognize it is being blocked).

Here are some quick sysctl.conf kernel tuning tips that have changed a server's load from 10-18, to less than 1.

Add or edit these in sysctl.conf to solve the issue

net.ipv4.tcp_rmem = 131072  131072  10485760
net.ipv4.tcp_wmem = 131072  131072  10485760
vm.dirty_ratio = 10
vm.dirty_background_ratio = 4

 

This is a preferable way rather than hacking or changing DRBD settings by upgrading the userland or kernel manually or while it runs (results can be unpredictable) whereas this kernel tuning has virtually no risk or impact on DRBD except a positive one.


Tags:

drbd, jbd, highiowait, solutiontired, iotop, partition, io, performs, versions, documented, quot, applies, devices, resyncing, lagged, creates, server, consequently, usage, kernels, dmesg, _worker, sock_sendmsg, expired, ko, kernel, module, blocked, sysctl, conf, tuning, edit, ipv, tcp_rmem, tcp_wmem, vm, dirty_ratio, dirty_background_ratio, preferable, hacking, settings, upgrading, userland, manually, unpredictable, whereas, virtually,

Latest Articles

  • Unable to mount location Failed to retrieve share list from server: Connection timed out - Samba/Linux Filesharing Not working Ubuntu Mint Linux Solution
  • How To Resize, Reduce a Video to a Specific Size and Quality Ubuntu Linux using ffmpeg
  • vi how to delete all lines in the file
  • Linux Mint / Ubuntu 20 Intel I219 NIC disconnects
  • Linux can't boot/grub boot loader screen with no options solution
  • EFI PXE grub2 Howto guide for Linux EFI PXE Booting on Debian, Mint, Ubuntu, RHEL
  • Aruba/HP/Dell IAP Wireless Controller Common Default Passwords
  • Debian, Mint Ubuntu how to remove package and associated config files
  • Linux Grub not booting the intended kernel solution in Debian, Mint, Ubuntu how to specify which kernel to boot by default
  • QEMU KVM Keyboard Problems Not Working Right Repeating Characters, Ctrl+C Copy and Paste not working right when using PS2 mouse in guests Solution
  • Linux how to compile binary with static sharedobjects embedded instead of dynamic to use on multi-distributions and avoid glibc compatiblity issues
  • /bin/sh: msgfmt: not found error solution on Linux Compilation Ubuntu Debian Mint Centos
  • Mikrotik RouterOS CHR/ISO Basic and Quick Setup Howto Guide
  • qemu 4 compilation options
  • CentOS 7 8 PXEBoot Netinstall Not Working Solution "Pane is dead "new value non-exisetnt xfs filesystem is not valid as a default fs type"
  • CentOS 6 EOL yum repo won't work Error: Cannot find a valid baseurl for repo: base Solution
  • CentOS 7 8 How To Disable SELinux
  • Wordpress How To Add Featured Image To Post in Hueman Theme
  • kdenlive full reset how to erase all config files
  • CentOS 7 8 yum error Trying other mirror. To address this issue please refer to the below wiki article