CVE-2015-1779: incrementally decode websocket frames

CVE-2015-1779: incrementally decode websocket frames

The logic for decoding websocket frames wants to fully
decode the frame header and payload, before allowing the
VNC server to see any of the payload data. There is no
size limit on websocket payloads, so this allows a
malicious network client to consume 2^64 bytes in memory
in QEMU. It can trigger this denial of service before
the VNC server even performs any authentication.

The fix is to decode the header, and then incrementally
decode the payload data as it is needed. With this fix
the websocket decoder will allow at most 4k of data to
be buffered before decoding and processing payload.

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
[ kraxel: fix frequent spurious disconnects, suggested by Peter Maydell ]

  @@ -361,7 +361,7 @@ int vncws_decode_frame_payload(Buffer *input,
  -        *payload_size = input->offset;
  +        *payload_size = *payload_remain;

[ kraxel: fix 32bit build ]

  @@ -306,7 +306,7 @@ struct VncState
  -    uint64_t ws_payload_remain;
  +    size_t ws_payload_remain;

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>