server: Only call gettimeofday once per poll loop.