drm/ttm: Optimize delayed buffer destruction