[coop] Add new states and transitions for hybrid coop suspend
The hybrid cooperative suspend mechanism works like this:
- when a thread is executing in GC Unsafe mode, we use cooperative suspend and expect
the thread to periodically checkpoint its execution (as in the ordinary full
coop mode).
- when a thread is executing in GC Safe mode, we use a preemptive signal-based
suspend. This is new - previously in full coop we would allow BLOCKING
threads to continue executing and only suspend them when they wanted to go
from GC Safe to GC Unsafe (via the BLOCKING_SELF_SUSPENDED state).
There are two new states and one updated transition - the transition will
service both running and blocking threads: The idea is that suspend mechanism
is determined by the suspend initiator, and is not embodied in the the state
machine. The state machine just has to return distinctive enough values from
the transitions for the initiator to select a policy.
Resume mechanism is determined by the state that a thread finds itself in when
it is resumed, and the original suspend policy.
The primary differences between suspend mechanisms are: the preemptive
suspend is two phase - the finish_async_suspension transition only does not
apply for cooperative suspend; the cooperative policy uses the poll transition
when the thread is running.
We renamed mono_threads_transition_request_async_suspension to mono_threads_transition_request_suspension.
- It must be called by a suspend initiator on a victim thread that is not
itself.
- There can only be one suspend initiator for a victim at a time - if the
suspend initiator initiates a suspension it must follow through with the
whole protocol.
- If a victim is RUNNING we transition it to ASYNC_SUSPEND_REQUSTED and return
ReqSuspendInitSuspendRunning which signals that the caller must initiate
suspension for a running thread. (Same as the old AsyncSuspendInitSuspend).
- If a victim is BLOCKING we transition it to BLOCKING_SUSPEND_REQUESTED.
Note that this is different from the old request_async_suspension which just
incremented the suspend count and returned AsyncSuspendBlocking. Now we
return ReqSuspendInitSuspendBlocking. The return of
ReqSuspendInitSuspendBlocking means the initiator may signal the victim and
must wait to be notified of the suspension (in preemptive suspend it will be
notified by the signal handler, in cooperative when the blocking thread
attempts to exit from blocking mode).
- In BLOCKING_SUSPEND_REQUESTED we may increment the suspend count. This is
slightly too lax if we're going to be using preemptive suspend on blocking
threads: in that case we're in the middle of a two-phase suspend and since
there is only one suspend initiator, we wouldn't expect another suspend to
be initiated until we have a chance to finish suspending. We leave it to
the suspend initiator to rule out by returning
ReqSuspendAlreadySuspendedBlocking from this state.
- In all the not executing states (SELF_SUSPENDED, ASYNC_SUSPENDED,
BLOCKING_SELF_SUSPENDED, BLOCKING_ASYNC_SUSPENDED) we just increment the
suspend count.
- All other transitions are illegal.
- Note that BLOCKING must always have a suspend count of 0 and
BLOCKING_SUSPEND_REQUESTED must have suspend_count > 0.
We add two new states:
- STATE_BLOCKING_SUSPEND_REQUESTED - this is a new state that indicates that a
suspend initiator wants to suspend this victim thread.
- The thread is still executing in this state.
- We only transition into this state with
mono_threads_transition_request_suspension.
- The only legal transitions out from this state are:
- resume - decrements the suspend_count. If the suspend_count becomes 0 we
go back to BLOCKING, otherwise we stay in BLOCKING_SUSPEND_REQUESTED.
- finish_async_suspension (executed by the suspend signal handler to
finish the two phase preemptive suspend request and notify the suspend initiator),
- done_blocking and abort_blocking. It means that the a
suspend initiator requested a suspend but we got to done (or abort)
blocking before the signal was delivered. We return NotifyAndWait and the
caller is supposed to send a notification to the suspend initiator and
wait for a resume(with suspend count == 0) which will return to
RUNNING (the victim thread goes to STATE_BLOCKING_SELF_SUSPENDED).
- STATE_BLOCKING_ASYNC_SUSPENDED
- The thread is not executing in this state (it is waiting for a resume).
- a thread transitions into this state from STATE_BLOCKING_SUSPEND_REQUESTED
on a finish_async_suspension transition (see above).
- on a resume transition we decrement the suspend count. if it's still
positive we stay in this state. if it's 0 the thread transitions out of
this state. Unlike other resume transitions, in this case the thread goes
back to BLOCKING and resumes executing in GC Safe mode. The resume caller
must notify the resume initiator (ie this is a ResumeInitAsyncResume).
- on a request_suspension we stay in this state and just increment the
suspend count.
- all other transitions from this state are illegal.
We add two new return values for abort_blocking and done_blocking:
- DoneBlockingNotifyAndWait and AbortBlockingNotifyAndWait. Similar to
SelfSuspendNotifyAndWait - there was a race between the second phase of a
preemptive suspend and another operation (in that case a poll, in this case a
done or abort blocking) and the other operation won, so its caller should
notify the suspend initiator. Since the thread is now suspended, the caller
should wait for the resume signal.
We add new results for request_suspension MonoRequestSuspendResult
(formerly MonoRequestAsyncSuspendResult):
- ReqSuspendAlreadySuspended means the thread is not executing and we just
incremented its suspend count and there's nothing else for the suspend
initiator to do. (The old AsyncSuspendAlreadySuspended)
- ReqSuspendAlreadySuspendedBlocking means the thread is in
BLOCKING_SUSPEND_REQUESTED and we initiated another suspend request. This
should only happen with full cooperative suspend - with hybrid suspend we
will use a two phase preemptive suspend on a blocking thread and since only
a single suspend initiator is active, this state should not be visible to
another suspend initiator.
- ReqSuspendInitSuspendRunning - Thread is executing in GC Unsafe mode and the
caller should suspend it. This is the old AsyncSuspendInitSuspend. In
full preemptive suspend this is the only suspend initiation action and the caller
should begin the preemptive suspend procedure. In full coop we expect the
victim thread to reach a safepoint and poll to finish suspending.
- ReqSuspendInitSuspendBlocking - Thread is executing in GC Safe mode and
the caller should initiate a suspend. In full coop the caller has nothing
to do - a thread executing in blocking mode is assumed to be suspended. In
hybrid suspend, the caller has to initiate a preemptive suspend.