Only do the unload_dll server call at the time we actually unload the