power: charger-manager: Fix accessing invalidated power supply after charger unbind