If the server went away, don't segfault by attempting to FD_SET -1.