build: Remove zlib from the cache if we failed to pass all the tests