vfio/container: Move listener to base container