Make the buffer padding size dependent on the resampler