target/i386: fix fbstp handling of out-of-range values
commit374ff4d0a3c2cce2bc6e4ba8a77eaba55c165252
authorJoseph Myers <joseph@codesourcery.com>
Wed, 13 May 2020 23:51:42 +0000 (13 23:51 +0000)
committerPaolo Bonzini <pbonzini@redhat.com>
Wed, 10 Jun 2020 16:10:25 +0000 (10 12:10 -0400)
tree47c17438c4186135cb39318e867aeeaf9cd0114a
parent18c53e1e73197a24f9f4b66b1276eb9868db5bf0
target/i386: fix fbstp handling of out-of-range values

The fbstp implementation fails to check for out-of-range and invalid
values, instead just taking the result of conversion to int64_t and
storing its sign and low 18 decimal digits.  Fix this by checking for
an out-of-range result (invalid conversions always result in INT64_MAX
or INT64_MIN from the softfloat code, which are large enough to be
considered as out-of-range by this code) and storing the packed BCD
indefinite encoding in that case.

Signed-off-by: Joseph Myers <joseph@codesourcery.com>
Message-Id: <alpine.DEB.2.21.2005132351110.11687@digraph.polyomino.org.uk>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
target/i386/fpu_helper.c
tests/tcg/i386/test-i386-fbstp.c