Update after changes to the provision API, make sure temp dir gets cleaned