Fix reloading of references to not cause access to freed memory