Fix variance overflow
commitfea3556e20f63ed1a03daa45804f724e70705aa0
authorJohann <johannkoenig@google.com>
Thu, 9 Feb 2012 20:38:31 +0000 (9 12:38 -0800)
committerJohann <johannkoenig@google.com>
Thu, 9 Feb 2012 20:38:31 +0000 (9 12:38 -0800)
tree6f77544f3b6063f61876dc22bb0436d6affafa8d
parent2e0d55314c28c546d98595e4b3799ca451a5a777
Fix variance overflow

In the variance calculations the difference is summed and later squared.
When the sum exceeds sqrt(2^31) the value is treated as a negative when
it is shifted which gives incorrect results.

To fix this we cast the result of the multiplication as unsigned.

The alternative fix is to shift sum down by 4 before multiplying.
However that will reduce precision.

For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and
change).

PPC change is untested.

Change-Id: I1bad27ea0720067def6d71a6da5f789508cec265
14 files changed:
vp8/encoder/arm/armv6/vp8_variance16x16_armv6.asm
vp8/encoder/arm/armv6/vp8_variance_halfpixvar16x16_h_armv6.asm
vp8/encoder/arm/armv6/vp8_variance_halfpixvar16x16_hv_armv6.asm
vp8/encoder/arm/armv6/vp8_variance_halfpixvar16x16_v_armv6.asm
vp8/encoder/arm/neon/variance_neon.asm
vp8/encoder/arm/neon/vp8_subpixelvariance16x16_neon.asm
vp8/encoder/arm/neon/vp8_subpixelvariance16x16s_neon.asm
vp8/encoder/arm/neon/vp8_subpixelvariance8x8_neon.asm
vp8/encoder/ppc/variance_altivec.asm
vp8/encoder/ppc/variance_subpixel_altivec.asm
vp8/encoder/variance_c.c
vp8/encoder/x86/variance_mmx.c
vp8/encoder/x86/variance_sse2.c
vp8/encoder/x86/variance_ssse3.c