Fix deleting the same buffer ID multiple times in one call