math: handle bitwise AND better (fix upper limit)