ctdb-tools: Sanity check changes before processing "reloadnodes"

ctdb-tools: Sanity check changes before processing "reloadnodes"

"ctdb reloadnodes" currently does no sanity checking of the nodes
file.  This can cause chaos if a line is deleted from the nodes file
rather than commented out.  It also repeatedly produces a spurious
warning for each deleted node, even if the node was deleted a long
time ago.

Instead compare the nodemap with the contents of the local nodes file
to sanity check before attempting any reloads.  Note that this is
still imperfect if the nodes files are inconsistent across nodes but
it is better.  Also ensure that any nodes that are to be deleted are
already disconnected.  Avoid trying to talk to deleted nodes.

The current implementation is a bit unfortunate when it comes to
deleting nodes.  The most obvious alternative to the above complexity
would be to reloadnodes on the specified node first, then fetch the
node map (in which newly deleted nodes would be marked as such) and
then handle the remote nodes.  However, the implementation of
reloadnodes is asynchronous and it only actions the reload after 1
second.  This is presumably to avoid the recovery master noticing the
inconsistency between nodemaps and triggering a recovery before all
nodes have had their nodemaps updated.

Note that this recovery can still occur if the check is done at an
inconvenient time.  A better long term approach might be to quiesce
the recovery master checks while reloadnodes is in progress.

Update a unit test to reflect the change.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>