USB: fix regression occurring during device removal