Cache the host port like we cache the task port