DRBD Slow Performance - 99.99 % [jbd2/drbd0-8] highiowait solution

Tired of checking iotop and seeing that your drbd partition is using 99.99% of io all the time and finding your drbd device performs slow in general?

This is especially an issue in versions of DRBD in the 8.3 tree in particular one documented case is on "8.3.13" but it likely applies to other devices.

The symptoms are that resyncing is fine and normal but any reasonable amount of activity is very slow and lagged and creates a high server load and consequently high io wait.  You may not notice it though until you apply a reasonable load or usage to the server and DRBD device.

In some kernels and some versions you may get the following error in dmesg

block drbd0: [drbd0_worker/1670] sock_sendmsg time expired, ko = 4294967295

*But we have seen many cases where the above error is not present (perhaps an older kernel or module does not recognize it is being blocked).

Here are some quick sysctl.conf kernel tuning tips that have changed a server's load from 10-18, to less than 1.

Add or edit these in sysctl.conf to solve the issue

net.ipv4.tcp_rmem = 131072  131072  10485760
net.ipv4.tcp_wmem = 131072  131072  10485760
vm.dirty_ratio = 10
vm.dirty_background_ratio = 4

 

This is a preferable way rather than hacking or changing DRBD settings by upgrading the userland or kernel manually or while it runs (results can be unpredictable) whereas this kernel tuning has virtually no risk or impact on DRBD except a positive one.


Tags:

drbd, jbd, highiowait, solutiontired, iotop, partition, io, performs, versions, documented, quot, applies, devices, resyncing, lagged, creates, server, consequently, usage, kernels, dmesg, _worker, sock_sendmsg, expired, ko, kernel, module, blocked, sysctl, conf, tuning, edit, ipv, tcp_rmem, tcp_wmem, vm, dirty_ratio, dirty_background_ratio, preferable, hacking, settings, upgrading, userland, manually, unpredictable, whereas, virtually,

Latest Articles

  • Cisco Router Setup Guide and Tutorial Howto With Commands and Examples
  • Linux Bash Script To List All Connected IPs and their network name
  • Cisco Switches How To Get Of Port Line Status Console Messages
  • Cisco DHCP Snooping Relay Setup Information
  • Cisco Switch Setup Guide Command List
  • Cisco 2960 Switch Reset To Factory Defaults
  • How To Boot Cisco CUCM UCSInstall 8.6, 10, 11 and 12 on KVM/Proxmox
  • VBOX VirtualBox How To Import Raw .img Disk File
  • Windows Server 2012, 2016, 2019 How To Install and Missing Disabled Telnet Client
  • proxmox vm networking breaks when you restart your network on the hostnode
  • Linux ln symlink how to update existing symbolic link
  • Ubuntu 18.04 / Linux Mint 19.1 Cannot Type or Login - solution
  • LUKS Hard Drive Encryption on Linux Mint Ubuntu Debian etc how to mount encrypted hard drive
  • How to use nmap locate other machines/computers/servers on your network using nmap
  • Linux Mint 18.2 Create Config File To Start Application Upon Login
  • Dell Wyse Thin Client BIOS Access Key
  • sudoers file in /etc warning about comments/includes!
  • Centos 7 Reallocate logical volume space to another
  • lvm how to reduce volume size
  • letsencrypt certbot error "Unable to find a virtual host listening on port 80 which is currently needed for Certbot to prove to the CA that you control your domain. Please add a virtual host for port 80."