remote: fix use-after-free error detected by glibc in ref_remove_duplicates