In short the two drives in the array were /dev/sdd and /dev/sde. The kernel sees they were unplugged and have gone down as you can see below.
mdadm caught the first one being unplugged /dev/sde and disabled the missing drive. However when the final drive that was part of the array is unplugged it didn't notice at all. Instead it complains about an IO error later for drives that the kernel knows do not exist anymore.
[45817.162728] ata4: exception........
heartbeat is stopped for some reason
Anyway hnode2 was active and the services are running fine but I see heartbeat has been stopped somehow.
Here is the last log I see of heartbeat:
[quote:23c84415f5]
Sep 9 17:15:32 hnode2 heartbeat: [16738]: info: MSG stats: 9/1762471 ms age 0 [pid16738/MST_CONTROL]
Sep 9 17:15:32 hnode2 heartbeat: [16738]: info: cl_malloc stats: 716/51784021 152624/74519 [pid16738/MST_CONTROL]
Sep 9 17:15:32........
Centos 4.3 x64 & VMWare Server Beta[code:1:6d0b2c8c2f]
The correct version of one or more libraries needed to run VMware Server may be
missing. This is the output of ldd /usr/bin/vmware:
linux-gate.so.1 => (0xffffe000)
libm.so.6 => /lib/tls/libm.so.6 (0xf7fbd000)
libdl.so.2 => /lib/libdl.so.2 (0xf7fb9000)
libpthread.so.0 => /lib/tls/libpthread.so.0 (0xf7fa7000)
libX11.so.6 => not f........