s3:dbwrap_ctdb: fix race condition with concurrent transactions on the same node.

s3:dbwrap_ctdb: fix race condition with concurrent transactions on the same node.

In ctdb_transaction_commit(), when the trans2_commit control fails, there
is a race condition in the 1 second sleep between the local transaction_cancel
and the call to ctdb_replay_transaction(): The database is not locked, and
neither is the transaction_lock record. So another client can start and possibly
complete a new transaction in this gap, but only on the same node: The locking
of the transaction_lock record on a different node which involves migration of
the record to the other node has been disabled by introduction of the
transaction_active flag on the db which closes precisely this gap from the start
of the commit until the call to TRANS2_FINISH or TRANS2_ERROR.
But this mechanism does not cover the case where a process on the same node
tries to start a transaction: There is no obstacle to locking the transaction_lock
record because the record does not need to be migrated.

This commit closes this race condition in ctdb_transaction_fetch_start()
by using the new ctdb_ctrl_transaction_active() call to ask the local
ctdb daemon whether it has a transaction running on the database.
If so, the check is repeated until the running transaction is done.

This does introduce an additional call to the local ctdbd when starting
transactions, but it does close the (hopefully) last race condition.

Michael

Signed-off-by: Michael Adam <obnox@samba.org>
(cherry picked from commit 056c2376ccfdff724701edee6b319a7d6d97822e)