spi: fix refcount-related spidev oops-on-rmmod