remote: Make ref_remove_duplicates faster for large numbers of refs