A lot of times I've seen questions about how this works when you have multiple nodes or a CDN, it can be quite tricky in theory if you have random IPs or several IPs.
The way certbot works at least for non-DNS challenges is that it will hit a random server that it resolves to, you have no control over which one it hits.
If certbot hits node 1 at first to tell it to create the well-known file, then checks node 2 or any other node, you will find auhorization fails.........
heartbeat is stopped for some reason
Anyway hnode2 was active and the services are running fine but I see heartbeat has been stopped somehow.
Here is the last log I see of heartbeat:
[quote:23c84415f5]
Sep 9 17:15:32 hnode2 heartbeat: [16738]: info: MSG stats: 9/1762471 ms age 0 [pid16738/MST_CONTROL]
Sep 9 17:15:32 hnode2 heartbeat: [16738]: info: cl_malloc stats: 716/51784021 152624/74519 [pid16738/MST_CONTROL]
Sep 9 17:15:32........