usb-serial: change referencing of port and serial structures