rt2800lib: fix wrong -128dBm when signal is stronger than -12dBm