qemu-kvm: an ugly hack to work around the suberror 1 emulation failures

qemu-kvm: an ugly hack to work around the suberror 1 emulation failures

The suberror 1 indicates that qemu-kvm was asked to emulate an instruction
it didn't know how to emulate. The reason for this emulation is that the
guest tried to access a page that the host hasn't mapped in yet:

kvm`kvm_mmut_page_fault+0x88
kvm`handle_ept_violation+0x111
kvm`vmx_handle_exit+0x16a
kvm`vcpu_enter_guest+0x3ea
kvm`__vcpu_run+0x8b
kvm`kvm_arch_vcpu_ioctl_run+0x112
kvm`kvm_ioctl+0x466
cdev_ioctl+0x39
specfs`spec_ioctl+0x60
fop_ioctl+0x55
ioctl+0x9b
sys_syscall+0x17a

kvm should at this point emulate the instruction and resume execution. But
it doesn't know how to emulate:

0x66 0x0f 0xe7 0x07

or

movntdq %xmm0,(%rdi)

I can't speak for non-Illumos based OSes, but Illumos will happily use SSE
instructions to zero out a page of memory:

hwblkclr+0x37:                  pxor   %xmm0,%xmm0
hwblkclr+0x3b:                  movntdq %xmm0,(%rdi)     <=== HERE
hwblkclr+0x3f:                  movntdq %xmm0,0x10(%rdi)
hwblkclr+0x44:                  movntdq %xmm0,0x20(%rdi)
hwblkclr+0x49:                  movntdq %xmm0,0x30(%rdi)

It doesn't take long before Illumos boot tries to zero out a page - that's
why Illumos guests trigger this bug:

pfnzero
ufs`ufs_getpage_miss+0x35f
ufs`ufs_getpage+0x8a6
fop_getpage+0x7e
segmap_fault+0x141
fbread+0xd1
ufs`blkatoff+0xf4
ufs`ufs_dirlook+0x216
ufs`ufs_lookup+0x191
fop_lookup+0xa2
lookuppnvp+0x1f6
lookuppnatcred+0x15e
lookupnameatcred+0xad
lookupname+0x38
vfs_mountdevices+0x96
vfs_mountroot+0x1b5
main+0x138
_locore_start+0x90

I have an ugly workaround that I'll try to get into Hipster's userland. The
basic idea is to fault in all the pages before qemu starts the virtual cpus.
That way, kvm never gets an mmu page fault that's valid.