usb-serial: only check speed once at realize time