math: fix a couple type bugs