replace_object: don't check read_replace_refs twice