avcodec encoder: fix sign comparison issue