[IrDA]: Fix Rx/Tx path race.