In short the solution is just to use vgremove for the actual /dev/mapper device:
vgremove /dev/mapper/backups-backuplv
box mnt # mdadm --manage /dev/md8 --stop
mdadm: Cannot get exclusive access to /dev/md8:Perhaps a running process, mounted filesystem or active volume group?
box mnt # lv
lvchange lvconvert lvcreate l........
# first we need a physical volume which we use the pvcreate tool to create
# I create mine on /dev/sdb3
pvcreate /dev/sdb3
dev_is_mpath: failed to get device for 8:19
Physical volume "/dev/sdb3" successfully created
# pvdisplay shows the newly created volume
pvdisplay
"/dev/sdb3" is a new physical volume of "1.35 TiB"
--- NEW Physical volume ---
PV N........