Don't hold the global mutex while destroying the primary buffer