spi: Drop owner assignment from spi_drivers