Don't try to destroy a NULL buffer on failure