xfs: prevent bogus assert when trying to remove non-existent attribute