softfloat: fix floatx80 handling of NaN