math: fix bug in bitwise AND