server: Remove the async from the queue in async_set_result().