target/i386: fix fxam handling of invalid encodings