Add an unref flag to handle_ref so that the unreferencing is optional.