configure: Fix previous change to still build all the binaries on