How to kill a docker swarm

Assign way more replicas than you have of memory on all nodes and watch the Swarm crash which can easily reproduce in a small VM for testing.

root@Deb11Docker01:~# docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED        STATUS        PORTS      NAMES
1336c9b0275e   redis:5   "docker-entrypoint.s…"   13 hours ago   Up 13 hours   6379/tcp   clever_bouman.7.ue8ff3g3ef2pv4ezu4cy5qc23
f5496728a590   redis:5   "docker-entrypoint.s…"   13 hours ago   Up 13 hours   6379/tcp   clever_bouman.11.kk3ij52i7dq0p4xqe3r05apzc
b0b2a47dc6b6   redis:5   "docker-entrypoint.s…"   13 hours ago   Up 13 hours   6379/tcp   clever_bouman.8.p5woskaqpyc6kod0njfmdq5m9
6dc2c2a0f8f2   redis:5   "docker-entrypoint.s…"   13 hours ago   Up 13 hours   6379/tcp   clever_bouman.23.x27zfpun3mf8j40qofers8mqs
91e264dc4ec9   redis:5   "docker-entrypoint.s…"   13 hours ago   Up 13 hours   6379/tcp   clever_bouman.17.lma6im540lw77uk6nolxg7rej
47830106e26d   redis:5   "docker-entrypoint.s…"   13 hours ago   Up 13 hours   6379/tcp   clever_bouman.18.8y6rm833sgxvbtnj1np4e8q0i
a4246ca036e9   redis:5   "docker-entrypoint.s…"   13 hours ago   Up 13 hours   6379/tcp   clever_bouman.19.z7jj4hiz6sjyux2hmgy6d9449
965754f35798   redis:5   "docker-entrypoint.s…"   13 hours ago   Up 13 hours   6379/tcp   clever_bouman.13.ue6b58bkx4w44n8gkht0bu0r8
d4a9572cae71   redis:5   "docker-entrypoint.s…"   13 hours ago   Up 13 hours   6379/tcp   clever_bouman.6.4sa0179qxpoy4m522d5mc2shy
1ea0cd1e493c   redis:5   "docker-entrypoint.s…"   13 hours ago   Up 13 hours   6379/tcp   clever_bouman.28.nmzn1jd77lybe211thm1f9q8n
root@Deb11Docker01:~# docker service update --replicas=60 clever_bouman
clever_bouman
overall progress: 51 out of 60 tasks
Error response from daemon: rpc error: code = DeadlineExceeded desc = context deadline exceeded
root@Deb11Docker01:~# docker service update --replicas=40 clever_bouman
Error response from daemon: rpc error: code = Unknown desc = The swarm does not have a leader. It's possible that too few managers are online. Make sure more than half of the managers are online.

[137927.550974] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=docker.service,mems_allowed=0,global_oom,task_memcg=/system.slice/docker.service,task=dockerd,pid=55994,uid=0
[137927.551065] Out of memory: Killed process 55994 (dockerd) total-vm:1859264kB, anon-rss:80588kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:664kB oom_score_adj:0



root@Deb11Docker02:~# ps aux|grep container
root        1404  0.0  3.5 1370844 35244 ?       Ssl  Nov30   1:44 /usr/bin/containerd
root       48601  1.3  0.3 1328876 3528 ?        Sl   02:34   0:08 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 8be1aac07725843e36d47934209ee280ec2705df3bd098a9320eb0abd659eaaa -address /run/containerd/containerd.sock
root       49954  0.9  0.4 1329132 4456 ?        Sl   02:35   0:05 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 3445ae9397be1d9cbb80dcf9679c7765c2e6f5093672543b4c16d5b0e7935c71 -address /run/containerd/containerd.sock
root       50004  0.9  1.4 1322088 14560 ?       Sl   02:35   0:05 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 72301e658829c8fc3679b0744a110100cd31423c6f510bf330e8b90842c308cb -address /run/containerd/containerd.sock
root       50037  0.7  0.6 1314148 6360 ?        Sl   02:35   0:04 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 13f22679e833befe833f14073d800d6fa6a04e44bb30e317dc01a282b19a3e1b -address /run/containerd/containerd.sock
root       50045  0.7  0.4 1322088 4560 ?        Sl   02:35   0:04 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 1c0e7f3e65032590ac3078afc2b227c1003680c7b8c48c755193a27b17027aec -address /run/containerd/containerd.sock
root       50061  0.7  0.6 1328940 6472 ?        Sl   02:35   0:04 /usr/bin/containerd-shim-runc-v2 -namespace moby -id b258bac1daebc0eebc75195802f22631991b99effda0c8dd32d916dda53b3ecd -address /run/containerd/containerd.sock
root       50109  0.7  0.4 1322088 4672 ?        Sl   02:35   0:04 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 5d9b13fc6a53e71a9b0efcf9524b4b410843a6351b273020cb38bd49fdeda8d1 -address /run/containerd/containerd.sock
root       50150  1.0  0.6 1320680 6564 ?        Sl   02:35   0:05 /usr/bin/containerd-shim-runc-v2 -namespace moby -id cb18a243b3cc30d4d62d9fec8995c14da81e8c2a3ed6de670a880a4e174c4e55 -address /run/containerd/containerd.sock
root       51985  0.0  0.0   6180   716 pts/0    S+   02:44   0:00 grep container
root@Deb11Docker02:~# killall -9 containerd-shim-runc-v2
root@Deb11Docker02:~# ps aux|grep container
root        1404  0.0  3.5 1379040 35244 ?       Ssl  Nov30   1:44 /usr/bin/containerd
root       52092  0.0  0.0   6180   716 pts/0    S+   02:45   0:00 grep container
root@Deb11Docker02:~#


 docker swarm init --force-new-cluster


Tags:

docker, swarmassign, replicas, nodes, swarm, reproduce, vm, deb, ps, container, ports, redis, quot, entrypoint, hellip, tcp, clever_bouman, ue, ff, ef, pv, ezu, cy, qc, kk, ij, dq, xqe, apzc, dc, woskaqpyc, kod, njfmdq, zfpun, mf, qofers, mqs, ec, lma, im, lw, uk, nolxg, rej, rm, sgxvbtnj, np, ca, z, jj, hiz, sjyux, hmgy, bkx, gkht, bu, cae, sa, qxpoy, mc, ea, nmzn, jd, lybe, thm, update, overall, tasks, daemon, rpc, deadlineexceeded, desc, context, deadline, exceeded, managers, online, oom, constraint, constraint_none, nodemask, null, cpuset, mems_allowed, global_oom, task_memcg, slice, task, dockerd, pid, uid, kb, anon, rss, shmem, pgtables, oom_score_adj, aux, grep, ssl, nov, usr, bin, containerd, sl, shim, runc, namespace, moby, aac, ee, df, bd, eb, abd, eaaa, ae, cbb, dcf, fc, bf, cb, befe, fa, bb, ac, afc, aec, bac, daebc, eebc, effda, dd, dda, ecd, efcf, fdeda, cc, fec, pts, killall, init, cluster,

Latest Articles

  • How high can a Xeon CPU get?
  • bash fix PATH environment variable "command not found" solution
  • Ubuntu Linux Mint Debian Redhat Youtube Cannot Play HD or 4K videos, dropped frames or high CPU usage with Nvidia or AMD Driver
  • hostapd example configuration for high speed AC on 5GHz using WPA2
  • hostapd how to enable and use WPS to connect wireless devices like printers
  • Dell Server Workstation iDRAC Dead after Firmware Update Solution R720, R320, R730
  • Cloned VM/Server/Computer in Linux won't boot and goes to initramfs busybox Solution
  • How To Add Windows 7 8 10 11 to GRUB Boot List Dual Booting
  • How to configure OpenDKIM on Linux with Postfix and setup bind zonefile
  • Debian Ubuntu 10/11/12 Linux how to get tftpd-hpa server setup tutorial
  • efibootmgr: option requires an argument -- 'd' efibootmgr version 15 grub-install.real: error: efibootmgr failed to register the boot entry: Operation not permitted.
  • Apache Error Won't start SSL Cert Issue Solution Unable to configure verify locations for client authentication SSL Library Error: 151441510 error:0906D066:PEM routines:PEM_read_bio:bad end line SSL Library Error: 185090057 error:0B084009:x509 certif
  • Linux Debian Mint Ubuntu Bridge br0 gets random IP
  • redis requirements
  • How to kill a docker swarm
  • docker swarm silly issues
  • isc-dhcp-server dhcpd how to get longer lease
  • nvidia cannot resume from sleep Comm: nvidia-sleep.sh Tainted: Linux Ubuntu Mint Debian
  • zfs and LUKS how to recover in Linux
  • [error] (28)No space left on device: Cannot create SSLMutex Apache Solution Linux CentOS Ubuntu Debian Mint