In short the solution is just to use vgremove for the actual /dev/mapper device:
vgremove /dev/mapper/backups-backuplv
box mnt # mdadm --manage /dev/md8 --stop
mdadm: Cannot get exclusive access to /dev/md8:Perhaps a running process, mounted filesystem or active volume group?
box mnt # lv
lvchange lvconvert lvcreate l........