xen: when removing a backend don't remove many of them