USB: usb-serial: call port_probe and port_remove at the right times