spi/spi-pl022: Call pl022_dma_remove(pl022) only if enable_dma is true