binarybuffer: Remove unnecessary cast and fix hidden "bug"