Remove patches for unmatched spans.