Remove unnecessary delaying of free_refs(ref_map) in builtin-fetch