vo_gl: fix crash with ati-hack after 20256a8a64c9679bc9136bb

vo_gl: fix crash with ati-hack after 20256a8a64c9679bc9136bb

Using the ati-hack and force-pbo suboptions (both can be enabled
automatically when particular ATI drivers are detected) could lead to
segfaults due to incorrect length calculation.

The problem is that width*((bpp+7)/8) == width*(bpp+7)/8 doesn't hold
true in general. The old code used bpp/8, because bpp was always
guaranteed to be divisible by 8. When commit 20256a8a64c96 fixed bugs
around 10 bit pixel formats, this assumption was broken, and resulted
in out of bounds memory accesses.

Additionally, the color values that were used to clear image borders
were incorrect and showed up as pink borders on the right/bottom sides
of the video when playing 10 bit content. Make the clear value
approximately correct; the borders are still filled with a single byte
value even though formats with 9-16 bits should be cleared with 2-byte
values with the low byte 0.

This commit also contains some other minor cleanups with no functional
changes.