DRBD Slow Performance - 99.99 % [jbd2/drbd0-8] highiowait solution

Tired of checking iotop and seeing that your drbd partition is using 99.99% of io all the time and finding your drbd device performs slow in general?

This is especially an issue in versions of DRBD in the 8.3 tree in particular one documented case is on "8.3.13" but it likely applies to other devices.

The symptoms are that resyncing is fine and normal but any reasonable amount of activity is very slow and lagged and creates a high server load and consequently high io wait.  You may not notice it though until you apply a reasonable load or usage to the server and DRBD device.

In some kernels and some versions you may get the following error in dmesg

block drbd0: [drbd0_worker/1670] sock_sendmsg time expired, ko = 4294967295

*But we have seen many cases where the above error is not present (perhaps an older kernel or module does not recognize it is being blocked).

Here are some quick sysctl.conf kernel tuning tips that have changed a server's load from 10-18, to less than 1.

Add or edit these in sysctl.conf to solve the issue

net.ipv4.tcp_rmem = 131072  131072  10485760
net.ipv4.tcp_wmem = 131072  131072  10485760
vm.dirty_ratio = 10
vm.dirty_background_ratio = 4

 

This is a preferable way rather than hacking or changing DRBD settings by upgrading the userland or kernel manually or while it runs (results can be unpredictable) whereas this kernel tuning has virtually no risk or impact on DRBD except a positive one.


Tags:

drbd, jbd, highiowait, solutiontired, iotop, partition, io, performs, versions, documented, quot, applies, devices, resyncing, lagged, creates, server, consequently, usage, kernels, dmesg, _worker, sock_sendmsg, expired, ko, kernel, module, blocked, sysctl, conf, tuning, edit, ipv, tcp_rmem, tcp_wmem, vm, dirty_ratio, dirty_background_ratio, preferable, hacking, settings, upgrading, userland, manually, unpredictable, whereas, virtually,

Latest Articles

  • CentOS 6 impossible to compile a newer libguestfs
  • chroot
  • How To Get Started on Ubuntu with gpt-2 OpenAI Text Prediction
  • Remove cloud-init in your VM
  • QEMU-KVM KVM Command Line Practical Guide
  • Linux How To Change NIC Name to eth0 instead of enps33 or enp0s25
  • virt-resize: error: libguestfs error: could not create appliance through libvirt.
  • Asterisk Does Not Retry When Authentication Fails
  • Linux Debian Ubuntu How To Install PEPPER Faster and Latest Adobe Flash Player in Firefox
  • How To Speed Up Linux Ubuntu and Debian Based Computers By Improving CPU Performance and Changing the CPU Governor
  • Convert data or file to base64 on a single line
  • Linux Mint Ubuntu Debian radeon slow 2D performance issues radeon_dp_aux_transfer_native: 158 callbacks suppressed
  • mdadm: super0.90 cannot open /dev/sdb1: Device or resource busy mdadm: /dev/sdb1 is not suitable for this array.
  • How To Install NextCloud on Centos 7 and Centos 8
  • AH01630: client denied by server configuration:
  • ERROR: Could not find a version that satisfies the requirement PIL (from versions: none) ERROR: No matching distribution found for PIL
  • ZTE Camera Cannot Work unable to connect to camera. Camera has been disabled becaue of security policies or is being used by other apps
  • QEMU KVM how to boot off a physical CD/DVD/BDROM Drive
  • How To Install OpenProject on Centos 7 Step-by-Step Guide
  • Ubuntu Debian Linux Cannot Install Wine Solution - wine1.6 : Depends: wine1.6-i386 (= 1:1.6.2-0ubuntu14.2) but it is not installable wine1.4 : Depends: wine1.6 but it is not going to be installed