Bug 1744358 move MediaEngineDefault-specific MediaEngineSource classes out of header...

[gecko.git] / gfx / docs / Silk.rst

Silk Overview
==========================

.. image:: SilkArchitecture.png

Architecture
------------

Our current architecture is to align three components to hardware vsync
timers:

1. Compositor
2. RefreshDriver / Painting
3. Input Events

The flow of our rendering engine is as follows:

1. Hardware Vsync event occurs on an OS specific *Hardware Vsync Thread*
   on a per monitor basis.
2. The *Hardware Vsync Thread* attached to the monitor notifies the
   ``CompositorVsyncDispatchers`` and ``RefreshTimerVsyncDispatcher``.
3. For every Firefox window on the specific monitor, notify a
   ``CompositorVsyncDispatcher``. The ``CompositorVsyncDispatcher`` is
   specific to one window.
4. The ``CompositorVsyncDispatcher`` notifies a
   ``CompositorWidgetVsyncObserver`` when remote compositing, or a
   ``CompositorVsyncScheduler::Observer`` when compositing in-process.
5. If remote compositing, a vsync notification is sent from the
   ``CompositorWidgetVsyncObserver`` to the ``VsyncBridgeChild`` on the
   UI process, which sends an IPDL message to the ``VsyncBridgeParent``
   on the compositor thread of the GPU process, which then dispatches to
   ``CompositorVsyncScheduler::Observer``.
6. The ``RefreshTimerVsyncDispatcher`` notifies the Chrome
   ``RefreshTimer`` that a vsync has occurred.
7. The ``RefreshTimerVsyncDispatcher`` sends IPC messages to all content
   processes to tick their respective active ``RefreshTimer``.
8. The ``Compositor`` dispatches input events on the *Compositor
   Thread*, then composites. Input events are only dispatched on the
   *Compositor Thread* on b2g.
9. The ``RefreshDriver`` paints on the *Main Thread*.

Hardware Vsync
--------------

Hardware vsync events from (1), occur on a specific ``Display`` Object.
The ``Display`` object is responsible for enabling / disabling vsync on
a per connected display basis. For example, if two monitors are
connected, two ``Display`` objects will be created, each listening to
vsync events for their respective displays. We require one ``Display``
object per monitor as each monitor may have different vsync rates. As a
fallback solution, we have one global ``Display`` object that can
synchronize across all connected displays. The global ``Display`` is
useful if a window is positioned halfway between the two monitors. Each
platform will have to implement a specific ``Display`` object to hook
and listen to vsync events. As of this writing, both Firefox OS and OS X
create their own hardware specific *Hardware Vsync Thread* that executes
after a vsync has occurred. OS X creates one *Hardware Vsync Thread* per
``CVDisplayLinkRef``. We do not currently support multiple displays, so
we use one global ``CVDisplayLinkRef`` that works across all active
displays. On Windows, we have to create a new platform ``thread`` that
waits for DwmFlush(), which works across all active displays. Once the
thread wakes up from DwmFlush(), the actual vsync timestamp is retrieved
from DwmGetCompositionTimingInfo(), which is the timestamp that is
actually passed into the compositor and refresh driver.

When a vsync occurs on a ``Display``, the *Hardware Vsync Thread*
callback fetches all ``CompositorVsyncDispatchers`` associated with the
``Display``. Each ``CompositorVsyncDispatcher`` is notified that a vsync
has occurred with the vsync’s timestamp. It is the responsibility of the
``CompositorVsyncDispatcher`` to notify the ``Compositor`` that is
awaiting vsync notifications. The ``Display`` will then notify the
associated ``RefreshTimerVsyncDispatcher``, which should notify all
active ``RefreshDrivers`` to tick.

All ``Display`` objects are encapsulated in a ``VsyncSource`` object.
The ``VsyncSource`` object lives in ``gfxPlatform`` and is instantiated
only on the parent process when ``gfxPlatform`` is created. The
``VsyncSource`` is destroyed when ``gfxPlatform`` is destroyed. It can
also be destroyed when the layout frame rate pref (or other prefs that
influence frame rate) are changed. This may mean we switch from hardware
to software vsync (or vice versa) at runtime. During the switch, there
may briefly be 2 vsync sources. Otherwise, there is only one
``VsyncSource`` object throughout the entire lifetime of Firefox. Each
platform is expected to implement their own ``VsyncSource`` to manage
vsync events. On OS X, this is through ``CVDisplayLinkRef``. On
Windows, it should be through ``DwmGetCompositionTimingInfo``.

Compositor
----------

When the ``CompositorVsyncDispatcher`` is notified of the vsync event,
the ``CompositorVsyncScheduler::Observer`` associated with the
``CompositorVsyncDispatcher`` begins execution. Since the
``CompositorVsyncDispatcher`` executes on the *Hardware Vsync Thread*
and the ``Compositor`` composites on the ``CompositorThread``, the
``CompositorVsyncScheduler::Observer`` posts a task to the
``CompositorThread``. The ``CompositorBridgeParent`` then composites.
The model where the ``CompositorVsyncDispatcher`` notifies components on
the *Hardware Vsync Thread*, and the component schedules the task on the
appropriate thread is used everywhere.

The ``CompositorVsyncScheduler::Observer`` listens to vsync events as
needed and stops listening to vsync when composites are no longer
scheduled or required. Every ``CompositorBridgeParent`` is associated
and tied to one ``CompositorVsyncScheduler::Observer``, which is
associated with the ``CompositorVsyncDispatcher``. Each
``CompositorBridgeParent`` is associated with one widget and is created
when a new platform window or ``nsBaseWidget`` is created. The
``CompositorBridgeParent``, ``CompositorVsyncDispatcher``,
``CompositorVsyncScheduler::Observer``, and ``nsBaseWidget`` all have
the same lifetimes, which are created and destroyed together.

Out-of-process Compositors
--------------------------

When compositing out-of-process, this model changes slightly. In this
case there are effectively two observers: a UI process observer
(``CompositorWidgetVsyncObserver``), and the
``CompositorVsyncScheduler::Observer`` in the GPU process. There are
also two dispatchers: the widget dispatcher in the UI process
(``CompositorVsyncDispatcher``), and the IPDL-based dispatcher in the
GPU process (``CompositorBridgeParent::NotifyVsync``). The UI process
observer and the GPU process dispatcher are linked via an IPDL protocol
called PVsyncBridge. ``PVsyncBridge`` is a top-level protocol for
sending vsync notifications to the compositor thread in the GPU process.
The compositor controls vsync observation through a separate actor,
``PCompositorWidget``, which (as a subactor for
``CompositorBridgeChild``) links the compositor thread in the GPU
process to the main thread in the UI process.

Out-of-process compositors do not go through
``CompositorVsyncDispatcher`` directly. Instead, the
``CompositorWidgetDelegate`` in the UI process creates one, and gives it
a ``CompositorWidgetVsyncObserver``. This observer forwards
notifications to a Vsync I/O thread, where ``VsyncBridgeChild`` then
forwards the notification again to the compositor thread in the GPU
process. The notification is received by a ``VsyncBridgeParent``. The
GPU process uses the layers ID in the notification to find the correct
compositor to dispatch the notification to.

CompositorVsyncDispatcher
-------------------------

The ``CompositorVsyncDispatcher`` executes on the *Hardware Vsync
Thread*. It contains references to the ``nsBaseWidget`` it is associated
with and has a lifetime equal to the ``nsBaseWidget``. The
``CompositorVsyncDispatcher`` is responsible for notifying the
``CompositorBridgeParent`` that a vsync event has occurred. There can be
multiple ``CompositorVsyncDispatchers`` per ``Display``, one
``CompositorVsyncDispatcher`` per window. The only responsibility of the
``CompositorVsyncDispatcher`` is to notify components when a vsync event
has occurred, and to stop listening to vsync when no components require
vsync events. We require one ``CompositorVsyncDispatcher`` per window so
that we can handle multiple ``Displays``. When compositing in-process,
the ``CompositorVsyncDispatcher`` is attached to the CompositorWidget
for the window. When out-of-process, it is attached to the
CompositorWidgetDelegate, which forwards observer notifications over
IPDL. In the latter case, its lifetime is tied to a CompositorSession
rather than the nsIWidget.

Multiple Displays
-----------------

The ``VsyncSource`` has an API to switch a ``CompositorVsyncDispatcher``
from one ``Display`` to another ``Display``. For example, when one
window either goes into full screen mode or moves from one connected
monitor to another. When one window moves to another monitor, we expect
a platform specific notification to occur. The detection of when a
window enters full screen mode or moves is not covered by Silk itself,
but the framework is built to support this use case. The expected flow
is that the OS notification occurs on ``nsIWidget``, which retrieves the
associated ``CompositorVsyncDispatcher``. The
``CompositorVsyncDispatcher`` then notifies the ``VsyncSource`` to
switch to the correct ``Display`` the ``CompositorVsyncDispatcher`` is
connected to. Because the notification works through the ``nsIWidget``,
the actual switching of the ``CompositorVsyncDispatcher`` to the correct
``Display`` should occur on the *Main Thread*. The current
implementation of Silk does not handle this case and needs to be built
out.

CompositorVsyncScheduler::Observer
----------------------------------

The ``CompositorVsyncScheduler::Observer`` handles the vsync
notifications and interactions with the ``CompositorVsyncDispatcher``.
When the ``Compositor`` requires a scheduled composite, it notifies the
``CompositorVsyncScheduler::Observer`` that it needs to listen to vsync.
The ``CompositorVsyncScheduler::Observer`` then observes / unobserves
vsync as needed from the ``CompositorVsyncDispatcher`` to enable
composites.

GeckoTouchDispatcher
--------------------

The ``GeckoTouchDispatcher`` is a singleton that resamples touch events
to smooth out jank while tracking a user’s finger. Because input and
composite are linked together, the
``CompositorVsyncScheduler::Observer`` has a reference to the
``GeckoTouchDispatcher`` and vice versa.

Input Events
------------

One large goal of Silk is to align touch events with vsync events. On
Firefox OS, touchscreens often have different touch scan rates than the
display refreshes. A Flame device has a touch refresh rate of 75 HZ,
while a Nexus 4 has a touch refresh rate of 100 HZ, while the device’s
display refresh rate is 60HZ. When a vsync event occurs, we resample
touch events, and then dispatch the resampled touch event to APZ. Touch
events on Firefox OS occur on a *Touch Input Thread* whereas they are
processed by APZ on the *APZ Controller Thread*. We use `Google
Android’s touch
resampling <https://web.archive.org/web/20200909082458/http://www.masonchang.com/blog/2014/8/25/androids-touch-resampling-algorithm>`__
algorithm to resample touch events.

Currently, we have a strict ordering between Composites and touch
events. When a touch event occurs on the *Touch Input Thread*, we store
the touch event in a queue. When a vsync event occurs, the
``CompositorVsyncDispatcher`` notifies the ``Compositor`` of a vsync
event, which notifies the ``GeckoTouchDispatcher``. The
``GeckoTouchDispatcher`` processes the touch event first on the *APZ
Controller Thread*, which is the same as the *Compositor Thread* on b2g,
then the ``Compositor`` finishes compositing. We require this strict
ordering because if a vsync notification is dispatched to both the
``Compositor`` and ``GeckoTouchDispatcher`` at the same time, a race
condition occurs between processing the touch event and therefore
position versus compositing. In practice, this creates very janky
scrolling. As of this writing, we have not analyzed input events on
desktop platforms.

One slight quirk is that input events can start a composite, for example
during a scroll and after the ``Compositor`` is no longer listening to
vsync events. In these cases, we notify the ``Compositor`` to observe
vsync so that it dispatches touch events. If touch events were not
dispatched, and since the ``Compositor`` is not listening to vsync
events, the touch events would never be dispatched. The
``GeckoTouchDispatcher`` handles this case by always forcing the
``Compositor`` to listen to vsync events while touch events are
occurring.

Widget, Compositor, CompositorVsyncDispatcher, GeckoTouchDispatcher Shutdown Procedure
--------------------------------------------------------------------------------------

When the `nsBaseWidget shuts
down <https://hg.mozilla.org/mozilla-central/file/0df249a0e4d3/widget/nsBaseWidget.cpp#l182>`__
- It calls nsBaseWidget::DestroyCompositor on the *Gecko Main Thread*.
During nsBaseWidget::DestroyCompositor, it first destroys the
CompositorBridgeChild. CompositorBridgeChild sends a sync IPC call to
CompositorBridgeParent::RecvStop, which calls
`CompositorBridgeParent::Destroy <https://hg.mozilla.org/mozilla-central/file/ab0490972e1e/gfx/layers/ipc/CompositorParent.cpp#l509>`__.
During this time, the *main thread* is blocked on the parent process.
CompositorBridgeParent::RecvStop runs on the *Compositor thread* and
cleans up some resources, including setting the
``CompositorVsyncScheduler::Observer`` to nullptr.
CompositorBridgeParent::RecvStop also explicitly keeps the
CompositorBridgeParent alive and posts another task to run
CompositorBridgeParent::DeferredDestroy on the Compositor loop so that
all ipdl code can finish executing. The
``CompositorVsyncScheduler::Observer`` also unobserves from vsync and
cancels any pending composite tasks. Once
CompositorBridgeParent::RecvStop finishes, the *main thread* in the
parent process continues shutting down the nsBaseWidget.

At the same time, the *Compositor thread* is executing tasks until
CompositorBridgeParent::DeferredDestroy runs, which flushes the
compositor message loop. Now we have two tasks as both the nsBaseWidget
releases a reference to the Compositor on the *main thread* during
destruction and the CompositorBridgeParent::DeferredDestroy releases a
reference to the CompositorBridgeParent on the *Compositor Thread*.
Finally, the CompositorBridgeParent itself is destroyed on the *main
thread* once both references are gone due to explicit `main thread
destruction <https://hg.mozilla.org/mozilla-central/file/50b95032152c/gfx/layers/ipc/CompositorParent.h#l148>`__.

With the ``CompositorVsyncScheduler::Observer``, any accesses to the
widget after nsBaseWidget::DestroyCompositor executes are invalid. Any
accesses to the compositor between the time the
nsBaseWidget::DestroyCompositor runs and the
CompositorVsyncScheduler::Observer’s destructor runs aren’t safe yet a
hardware vsync event could occur between these times. Since any tasks
posted on the Compositor loop after
CompositorBridgeParent::DeferredDestroy is posted are invalid, we make
sure that no vsync tasks can be posted once
CompositorBridgeParent::RecvStop executes and DeferredDestroy is posted
on the Compositor thread. When the sync call to
CompositorBridgeParent::RecvStop executes, we explicitly set the
CompositorVsyncScheduler::Observer to null to prevent vsync
notifications from occurring. If vsync notifications were allowed to
occur, since the ``CompositorVsyncScheduler::Observer``\ ’s vsync
notification executes on the *hardware vsync thread*, it would post a
task to the Compositor loop and may execute after
CompositorBridgeParent::DeferredDestroy. Thus, we explicitly shut down
vsync events in the ``CompositorVsyncDispatcher`` and
``CompositorVsyncScheduler::Observer`` during nsBaseWidget::Shutdown to
prevent any vsync tasks from executing after
CompositorBridgeParent::DeferredDestroy.

The ``CompositorVsyncDispatcher`` may be destroyed on either the *main
thread* or *Compositor Thread*, since both the nsBaseWidget and
``CompositorVsyncScheduler::Observer`` race to destroy on different
threads. nsBaseWidget is destroyed on the *main thread* and releases a
reference to the ``CompositorVsyncDispatcher`` during destruction. The
``CompositorVsyncScheduler::Observer`` has a race to be destroyed either
during CompositorBridgeParent shutdown or from the
``GeckoTouchDispatcher`` which is destroyed on the main thread with
`ClearOnShutdown <https://hg.mozilla.org/mozilla-central/file/21567e9a6e40/xpcom/base/ClearOnShutdown.h#l15>`__.
Whichever object, the CompositorBridgeParent or the
``GeckoTouchDispatcher`` is destroyed last will hold the last reference
to the ``CompositorVsyncDispatcher``, which destroys the object.

Refresh Driver
--------------

The Refresh Driver is ticked from a `single active
timer <https://hg.mozilla.org/mozilla-central/file/ab0490972e1e/layout/base/nsRefreshDriver.cpp#l11>`__.
The assumption is that there are multiple ``RefreshDrivers`` connected
to a single ``RefreshTimer``. There are two ``RefreshTimers``: an active
and an inactive ``RefreshTimer``. Each Tab has its own
``RefreshDriver``, which connects to one of the global
``RefreshTimers``. The ``RefreshTimers`` execute on the *Main Thread*
and tick their connected ``RefreshDrivers``. We do not want to break
this model of multiple ``RefreshDrivers`` per a set of two global
``RefreshTimers``. Each ``RefreshDriver`` switches between the active
and inactive ``RefreshTimer``.

Instead, we create a new ``RefreshTimer``, the ``VsyncRefreshTimer``
which ticks based on vsync messages. We replace the current active timer
with a ``VsyncRefreshTimer``. All tabs will then tick based on this new
active timer. Since the ``RefreshTimer`` has a lifetime of the process,
we only need to create a single ``RefreshTimerVsyncDispatcher`` per
``Display`` when Firefox starts. Even if we do not have any content
processes, the Chrome process will still need a ``VsyncRefreshTimer``,
thus we can associate the ``RefreshTimerVsyncDispatcher`` with each
``Display``.

When Firefox starts, we initially create a new ``VsyncRefreshTimer`` in
the Chrome process. The ``VsyncRefreshTimer`` will listen to vsync
notifications from ``RefreshTimerVsyncDispatcher`` on the global
``Display``. When nsRefreshDriver::Shutdown executes, it will delete the
``VsyncRefreshTimer``. This creates a problem as all the
``RefreshTimers`` are currently manually memory managed whereas
``VsyncObservers`` are ref counted. To work around this problem, we
create a new ``RefreshDriverVsyncObserver`` as an inner class to
``VsyncRefreshTimer``, which actually receives vsync notifications. It
then ticks the ``RefreshDrivers`` inside ``VsyncRefreshTimer``.

With Content processes, the start up process is more complicated. We
send vsync IPC messages via the use of the PBackground thread on the
parent process, which allows us to send messages from the Parent
process’ without waiting on the *main thread*. This sends messages from
the Parent::\ *PBackground Thread* to the Child::\ *Main Thread*. The
*main thread* receiving IPC messages on the content process is
acceptable because ``RefreshDrivers`` must execute on the *main thread*.
However, there is some amount of time required to setup the IPC
connection upon process creation and during this time, the
``RefreshDrivers`` must tick to set up the process. To get around this,
we initially use software ``RefreshTimers`` that already exist during
content process startup and swap in the ``VsyncRefreshTimer`` once the
IPC connection is created.

During nsRefreshDriver::ChooseTimer, we create an async PBackground IPC
open request to create a ``VsyncParent`` and ``VsyncChild``. At the same
time, we create a software ``RefreshTimer`` and tick the
``RefreshDrivers`` as normal. Once the PBackground callback is executed
and an IPC connection exists, we swap all ``RefreshDrivers`` currently
associated with the active ``RefreshTimer`` and swap the
``RefreshDrivers`` to use the ``VsyncRefreshTimer``. Since all
interactions on the content process occur on the main thread, there are
no need for locks. The ``VsyncParent`` listens to vsync events through
the ``VsyncRefreshTimerDispatcher`` on the parent side and sends vsync
IPC messages to the ``VsyncChild``. The ``VsyncChild`` notifies the
``VsyncRefreshTimer`` on the content process.

During the shutdown process of the content process, ActorDestroy is
called on the ``VsyncChild`` and ``VsyncParent`` due to the normal
PBackground shutdown process. Once ActorDestroy is called, no IPC
messages should be sent across the channel. After ActorDestroy is
called, the IPDL machinery will delete the **VsyncParent/Child** pair.
The ``VsyncParent``, due to being a ``VsyncObserver``, is ref counted.
After ``VsyncParent::ActorDestroy`` is called, it unregisters itself
from the ``RefreshTimerVsyncDispatcher``, which holds the last reference
to the ``VsyncParent``, and the object will be deleted.

Thus the overall flow during normal execution is:

1. VsyncSource::Display::RefreshTimerVsyncDispatcher receives a Vsync
   notification from the OS in the parent process.
2. RefreshTimerVsyncDispatcher notifies
   VsyncRefreshTimer::RefreshDriverVsyncObserver that a vsync occurred on
   the parent process on the hardware vsync thread.
3. RefreshTimerVsyncDispatcher notifies the VsyncParent on the hardware
   vsync thread that a vsync occurred.
4. The VsyncRefreshTimer::RefreshDriverVsyncObserver in the parent
   process posts a task to the main thread that ticks the refresh
   drivers.
5. VsyncParent posts a task to the PBackground thread to send a vsync
   IPC message to VsyncChild.
6. VsyncChild receive a vsync notification on the content process on the
   main thread and ticks their respective RefreshDrivers.

Compressing Vsync Messages
--------------------------

Vsync messages occur quite often and the *main thread* can be busy for
long periods of time due to JavaScript. Consistently sending vsync
messages to the refresh driver timer can flood the *main thread* with
refresh driver ticks, causing even more delays. To avoid this problem,
we compress vsync messages on both the parent and child processes.

On the parent process, newer vsync messages update a vsync timestamp but
do not actually queue any tasks on the *main thread*. Once the parent
process’ *main thread* executes the refresh driver tick, it uses the
most updated vsync timestamp to tick the refresh driver. After the
refresh driver has ticked, one single vsync message is queued for
another refresh driver tick task. On the content process, the IPDL
``compress`` keyword automatically compresses IPC messages.

Multiple Monitors
-----------------

In order to have multiple monitor support for the ``RefreshDrivers``, we
have multiple active ``RefreshTimers``. Each ``RefreshTimer`` is
associated with a specific ``Display`` via an id and tick when it’s
respective ``Display`` vsync occurs. We have **N RefreshTimers**, where
N is the number of connected displays. Each ``RefreshTimer`` still has
multiple ``RefreshDrivers``.

When a tab or window changes monitors, the ``nsIWidget`` receives a
display changed notification. Based on which display the window is on,
the window switches to the correct ``RefreshTimerVsyncDispatcher`` and
``CompositorVsyncDispatcher`` on the parent process based on the display
id. Each ``TabParent`` should also send a notification to their child.
Each ``TabChild``, given the display ID, switches to the correct
``RefreshTimer`` associated with the display ID. When each display vsync
occurs, it sends one IPC message to notify vsync. The vsync message
contains a display ID, to tick the appropriate ``RefreshTimer`` on the
content process. There is still only one **VsyncParent/VsyncChild**
pair, just each vsync notification will include a display ID, which maps
to the correct ``RefreshTimer``.

Object Lifetime
---------------

1. CompositorVsyncDispatcher - Lives as long as the nsBaseWidget
   associated with the VsyncDispatcher
2. CompositorVsyncScheduler::Observer - Lives and dies the same time as
   the CompositorBridgeParent.
3. RefreshTimerVsyncDispatcher - As long as the associated display
   object, which is the lifetime of Firefox.
4. VsyncSource - Lives as long as the gfxPlatform on the chrome process,
   which is the lifetime of Firefox.
5. VsyncParent/VsyncChild - Lives as long as the content process
6. RefreshTimer - Lives as long as the process

Threads
-------

All ``VsyncObservers`` are notified on the *Hardware Vsync Thread*. It
is the responsibility of the ``VsyncObservers`` to post tasks to their
respective correct thread. For example, the
``CompositorVsyncScheduler::Observer`` will be notified on the *Hardware
Vsync Thread*, and post a task to the *Compositor Thread* to do the
actual composition.

1. Compositor Thread - Nothing changes
2. Main Thread - PVsyncChild receives IPC messages on the main thread.
   We also enable/disable vsync on the main thread.
3. PBackground Thread - Creates a connection from the PBackground thread
   on the parent process to the main thread in the content process.
4. Hardware Vsync Thread - Every platform is different, but we always
   have the concept of a hardware vsync thread. Sometimes this is
   actually created by the host OS. On Windows, we have to create a
   separate platform thread that blocks on DwmFlush().