Avoid an underflow in the SSE mixers if BufferSize is less than 4