build: handle the non-selinux spec build case correctly also