Adjust rd calculation in choose_tx_size_from_rd