ubik: ensure correct election timeout for vlserver

ubik: ensure correct election timeout for vlserver

Ubik invariants (see src/ubik/ubik.p.h) document that for proper quorum
elections, the VOTE service requires:

  "#SMALLTIME and #BIGTIME must be larger than
   #RPCTIMEOUT+max(#RPCTIMEOUT, #POLLTIME)"

Solving this for VOTE connection Rx dead timeout yields:

   Rx dead timeout < 30s

The current default rx_connDeadTime is 12s; if this value is in force
when VOTE client connections are created or recreated, ubik elections
will work correctly.

(Apparently, RPCTIMEOUT 20s was once intended to be used for this value,
but this constant has not been referenced by any live code since the
original IBM code import.)

However, since the original IBM code import, vlserver has set
rx_connDeadTime to 50s.  In 1.6.x and older releases, this was only done
after the ubik VOTE and DISK client connections had been established;
therefore the initial VOTE connections had the correct default
rx_connDeadTime of 12s.  However, if a VOTE client connection ever failed
(e.g., due to a down vlserver), it would be reestablished with the 50s
timeout.  Because this violates a ubik invariant, it may lead to random
quorum disturbances if further vlserver outages occur.  This is because
the election results may be delayed for up to 50s until the
multi_VOTE_Beacon to the down vlserver times out.  If the vlserver goes
down just a few seconds before the next election, this delay is just
long enough to cause the syncsite's votes to expire (SMALLTIME 60s),
triggering a temporary loss of quorum even when there are sufficient yes
votes to maintain quorum.

As of commit 3e531db9ce50 (vlserver: rx_SetRxDeadTime before ubik init)
introduced with OpenAFS 1.8.0, rx_connDeadTime is set to 50s for ALL
vlserver connections, including the initial ubik VOTE and DISK client
connections.  However, as explained above, this violates a ubik
invariant for the VOTE service.  This results in a consistent loss of
quorum every time a vlserver goes down within a few seconds of a new
election, even if there are sufficient votes to maintain quorum.

Create and use a new function ubeacon_NewVOTEConnection to create all
VOTE client connections with an rx dead time VOTE_RPCTIMEOUT which
complies with the ubik invariants.  This allows ubik applications like
vlserver to change rx_connDeadTime without affecting ubik election
timing.

Note: vlserver is the only in-tree ubik server affected by this issue.
Even though buserver (src/budb/server.c) also sets a problematic
rx_connDeadTime of 60s at initialization, this is immediately reset to
12s by a subsequent call to rx_InitHost.  (This buserver anomaly will be
addressed in another commit.) The other ubik servers (ptserver and
kaserver) always use the default 12s time.

Change-Id: I63e2c46b6097c9c2c89e6e858e3958846c6cb190
Reviewed-on: https://gerrit.openafs.org/14608
Tested-by: BuildBot <buildbot@rampaginggeek.com>
Reviewed-by: Andrew Deason <adeason@sinenomine.net>
Reviewed-by: Benjamin Kaduk <kaduk@mit.edu>