[media] m5mols: Fix set_fmt to return proper pixel format code