After looking at several deployments of VMWare, we had one deployment where the company enabled balloon mode which is essentially a way to overcommit memory the hostnode(s) don't have. Let's say you have put your database into memory or use a cached service in Apache or haproxy/nginx. VMWare will start taking some portions of these and at some point it corrupts things and the application running will either crash, malfunction or segfault.
Another factor is that OOMwill be........
Assign way more replicas than you have of memory on all nodes and watch the Swarm crash which can easily reproduce in a small VMfor testing.
root@Deb11Docker01:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAM........
Apr 17 11:55:47 Cluster01 heartbeat: [1426]: WARN: ha_msg_add_nv_depth: line doesn't contain '='
Apr 17 11:55:47 Cluster01 heartbeat: [1426]: info: >>>
Apr 17 11:55:47 Cluster01 heartbeat: [1426]: ERROR: NV failure (msgfromsteam): [>>>
]
I have never been able to reproduce the above but it happened on heartbeat-3.04-2 when Itook a node down for testing.
One thing Ibelieve caused it was by configuring ha.cf........