Bug 1764201 Part 3: Remove screen info stuff from gfxPlatform. r=jgilbert,geckoview...

[gecko.git] / gfx / docs / RenderingOverview.rst

Rendering Overview
==================

This document is an overview of the steps to render a webpage, and how HTML
gets transformed and broken down, step by step, into commands that can execute
on the GPU.

If you're coming into the graphics team with not a lot of background
in browsers, start here :)

.. contents::

High level overview
-------------------

.. image:: RenderingOverviewSimple.png
   :width: 100%

Layout
~~~~~~
Starting at the left in the above image, we have a document
represented by a DOM - a Document Object Model.  A Javascript engine
will execute JS code, either to make changes to the DOM, or to respond to
events generated by the DOM (or do both).

The DOM is a high level description and we don't know what to draw or
where until it is combined with a Cascading Style Sheet (CSS).
Combining these two and figuring out what, where and how to draw
things is the responsibility of the Layout team.  The
DOM is converted into a hierarchical Frame Tree, which nests visual
elements (boxes).  Each element points to some node in a Style Tree
that describes what it should look like -- color, transparency, etc.
The result is that now we know exactly what to render where, what goes
on top of what (layering and blending) and at what pixel coordinate.
This is the Display List.

The Display List is a light-weight data structure because it's shallow
-- it mostly points back to the Frame Tree.  There are two problems
with this.  First, we want to cross process boundaries at this point.
Everything up until now happens in a Content Process (of which there are
several).  Actual GPU rendering happens in a GPU Process (on some
platforms).  Second, everything up until now was written in C++; but
WebRender is written in Rust.  Thus the shallow Display List needs to
be serialized in a completely self-contained binary blob that will
survive Interprocess Communication (IPC) and a language switch (C++ to
Rust).  The result is the WebRender Display List.

WebRender
~~~~~~~~~

The GPU process receives the WebRender Display List blob and
de-serializes it into a Scene.  This Scene contains more than the
strictly visible elements; for example, to anticipate scrolling, we
might have several paragraphs of text extending past the visible page.

For a given viewport, the Scene gets culled and stripped down to a
Frame.  This is also where we start preparing data structures for GPU
rendering, for example getting some font glyphs into an atlas for
rasterizing text.

The final step takes the Frame and submits commands to the GPU to
actually render it.  The GPU will execute the commands and composite
the final page.

Software
~~~~~~~~

The above is the new WebRender-enabled way to do things.  But in the
schematic you'll note a second branch towards the bottom: this is the
legacy code path which does not use WebRender (nor Rust).  In this
case, the Display List is converted into a Layer Tree. The purpose of
this Tree is to try and avoid having to re-render absolutely
everything when the page needs to be refreshed. For example, when
scrolling we should be able to redraw the page by mostly shifting
things around. However that requires those 'things' to still be around
from last time we drew the page.  In other words, visual elements that
are likely to be static and reusable need to be drawn into their own
private "page" (a cache).  Then we can recombine (composite) all of
these when redrawing the actual page.

Figuring out which elements would be good candidates for this, and
striking a balance between good performance versus excessive memory
use, is the purpose of the Layer Tree.  Each 'layer' is a cached image
of some element(s).  This logic also takes occlusion into account, eg.
don't allocate and render a layer for elements that are known to be
completely obscured by something in front of them.

Redrawing the page by combining the Layer Tree with any newly
rasterized elements is the job of the Compositor.


Even when a layer cannot be reused in its entirety, it is likely
that only a small part of it was invalidated.  Thus there is an
elaborate system for tracking dirty rectangles, starting an update by
copying the area that can be salvaged, and then redrawing only what
cannot.

In fact, this idea can be extended to delta-tracking of display lists
themselves. Traversing the layout tree and building a display list is
also not cheap, so the code tries to partially invalidate and rebuild
the display list incrementally when possible.
This optimization is used both for non-WebRender and WebRender in
fact.


Asynchronous Panning And Zooming
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Earlier we mentioned that a Scene might contain more elements than are
strictly necessary for rendering what's visible (the Frame).  The
reason for that is Asynchronous Panning and Zooming, or APZ for short.
The browser will feel much more responsive if scrolling & zooming can
short-circuit all of these data transformations and IPC boundaries,
and instead directly update an offset of some layer and recomposite.
(Think of late-latching in a VR context)

This simple idea introduces a lot of complexity: how much extra do you
rasterize, and in which direction?  How much memory can we afford?
What about Javascript that responds to scroll events and perhaps does
something 'interesting' with the page in return?  What about nested
frames or nested scrollbars?  What if we scroll so much that we go
past the boundaries of the Scene that we know about?

See AsyncPanZoom.rst for all that and more.

A Few More Details
~~~~~~~~~~~~~~~~~~

Here's another schematic which basically repeats the previous one, but
showing a little bit more detail.  Note that the direction is reversed
-- the data flow starts at the right.  Sorry about that :)

.. image:: RenderingOverviewDetail.png
   :width: 100%

Some things to note:

- there are multiple content processes, currently 4 of them.  This is
  for security reasons (sandboxing), stability (isolate crashes) and
  performance (multi-core machines);
- ideally each "webpage" would run in its own process for security;
  this is being developed under the term 'fission';
- there is only a single GPU process, if there is one at all;
  some platforms have it as part of the Parent;
- not shown here is the Extension process that isolates WebExtensions;
- for non-WebRender, rasterization happens in the Content Process, and
  we send entire Layers to the GPU/Compositor process (via shared
  memory, only using actual IPC for its metadata like width & height);
- if the GPU process crashes (a bug or a driver issue) we can simply
  restart it, resend the display list, and the browser itself doesn't crash;
- the browser UI is just another set of DOM+JS, albeit one that runs
  with elevated privileges. That is, its JS can do things that
  normal JS cannot.  It lives in the Parent Process, which then uses
  IPC to get it rendered, same as regular Content. (the IPC arrow also
  goes to WebRender Display List but is omitted to reduce clutter);
- UI events get routed to APZ first, to minimize latency. By running
  inside the GPU process, we may have access to data such
  as rasterized clipping masks that enables finer grained hit testing;
- the GPU process talks back to the content process; in particular,
  when APZ scrolls out of bounds, it asks Content to enlarge/shift the
  Scene with a new "display port";
- we still use the GPU when we can for compositing even in the
  non-WebRender case;


WebRender In Detail
-------------------

Converting a display list into GPU commands is broken down into a
number of steps and intermediate data structures.


.. image:: RenderingOverviewTrees.png
   :width: 75%
   :align: center

..

    *Each element in the picture tree points to exactly one node in the spatial
    tree. Only a few of these links are shown for clarity (the dashed lines).*

The Picture Tree
~~~~~~~~~~~~~~~~

The incoming display list uses "stacking contexts".  For example, to
render some text with a drop shadow, a display list will contain three
items:

- "enable shadow" with some parameters such as shadow color, blur size, and offset;
- the text item;
- "pop all shadows" to deactivate shadows;

WebRender will break this down into two distinct elements, or
"pictures".  The first represents the shadow, so it contains a copy of the
text item, but modified to use the shadow's color, and to shift the
text by the shadow's offset.  The second picture contains the original text
to draw on top of the shadow.

The fact that the first picture, the shadow, needs to be blurred, is a
"compositing" property of the picture which we'll deal with later.

Thus, the stack-based display list gets converted into a list of pictures
-- or more generally, a hierarchy of pictures, since items are nested
as per the original HTML.

Example visual elements are a TextRun, a LineDecoration, or an Image
(like a .png file).

Compared to 3D rendering, the picture tree is similar to a scenegraph: it's a
parent/child hierarchy of all the drawable elements that make up the "scene", in
this case the webpage.  One important difference is that the transformations are
stored in a separate tree, the spatial tree.

The Spatial Tree
~~~~~~~~~~~~~~~~

The nodes in the spatial tree represent coordinate transforms.  Every time the
DOM hierarchy needs child elements to be transformed relative to their parent,
we add a new Spatial Node to the tree. All those child elements will then point
to this node as their "local space" reference (aka coordinate frame).  In
traditional 3D terms, it's a scenegraph but only containing transform nodes.

The nodes are called frames, as in "coordinate frame":

- a Reference Frame corresponds to a ``<div>``;
- a Scrolling Frame corresponds to a scrollable part of the page;
- a Sticky Frame corresponds to some fixed position CSS style.

Each element in the picture tree then points to a spatial node inside this tree,
so by walking up and down the tree we can find the absolute position of where
each element should render (traversing down) and how large each element needs to
be (traversing up).  Originally the transform information was part of the
picture tree, as in a traditional scenegraph, but visual elements and their
transforms were split apart for technical reasons.

Some of these nodes are dynamic.  A scroll-frame can obviously scroll, but a
Reference Frame might also use a property binding to enable a live link with
JavaScript, for dynamic updates of (currently) the transform and opacity.

Axis-aligned transformations (scales and translations) are considered "simple",
and are conceptually combined into a single "CoordinateSystem".  When we
encounter a non-axis-aligned transform, we start a new CoordinateSystem.  We
start in CoordinateSystem 0 at the root, and would bump this to CoordinateSystem
1 when we encounter a Reference Frame with a rotation or 3D transform, for
example.  This would then be the CoordinateSystem index for all its children,
until we run into another (nested) non-simple transform, and so on.  Roughly
speaking, as long as we're in the same CoordinateSystem, the transform stack is
simple enough that we have a reasonable chance of being able to flatten it. That
lets us directly rasterize text at its final scale for example, optimizing
away some of the intermediate pictures (offscreen textures).

The layout code positions elements relative to their parent.  Thus to position
the element on the actual page, we need to walk the Spatial Tree all the way to
the root and apply each transform; the result is a ``LayoutToWorldTransform``.

One final step transforms from World to Device coordinates, which deals with
DPI scaling and such.

.. csv-table::
    :header: "WebRender term", "Rough analogy"

      Spatial Tree, Scenegraph -- transforms only
      Picture Tree, Scenegraph -- drawables only (grouping)
      Spatial Tree Rootnode, World Space
      Layout space, Local/Object Space
      Picture, RenderTarget (sort of; see RenderTask below)
      Layout-To-World transform, Local-To-World transform
      World-To-Device transform, World-To-Clipspace transform


The Clip Tree
~~~~~~~~~~~~~

Finally, we also have a Clip Tree, which contains Clip Shapes. For
example, a rounded corner div will produce a clip shape, and since
divs can be nested, you end up with another tree.  By pointing at a Clip Shape,
visual elements will be clipped against this shape plus all parent shapes above it
in the Clip Tree.

As with CoordinateSystems, a chain of simple 2D clip shapes can be collapsed
into something that can be handled in the vertex shader, at very little extra
cost.  More complex clips must be rasterized into a mask first, which we then
sample from to ``discard`` in the pixel shader as needed.

In summary, at the end of scene building the display list turned into
a picture tree, plus a spatial tree that tells us what goes where
relative to what, plus a clip tree.

RenderTask Tree
~~~~~~~~~~~~~~~

Now in a perfect world we could simply traverse the picture tree and start
drawing things: one drawcall per picture to render its contents, plus one
drawcall to draw the picture into its parent.  However, recall that the first
picture in our example is a "text shadow" that needs to be blurred.  We can't
just rasterize blurry text directly, so we need a number of steps or "render
passes" to get the intended effect:

.. image:: RenderingOverviewBlurTask.png
   :align: right
   :height: 400px

- rasterize the text into an offscreen rendertarget;
- apply one or more downscaling passes until the blur radius is reasonable;
- apply a horizontal Gaussian blur;
- apply a vertical Gaussian blur;
- use the result as an input for whatever comes next, or blit it to
  its final position on the page (or more generally, on the containing
  parent surface/picture).

In the general case, which passes we need and how many of them depends
on how the picture is supposed to be composited (CSS filters, SVG
filters, effects) and its parameters (very large vs. small blur
radius, say).

Thus, we walk the picture tree and build a render task tree: each high
level abstraction like "blur me" gets broken down into the necessary
render passes to get the effect.  The result is again a tree because a
render pass can have multiple input dependencies (eg. blending).

(Cfr. games, this has echoes of the Frostbite Framegraph in that it
dynamically builds up a renderpass DAG and dynamically allocates storage
for the outputs).

If there are complicated clip shapes that need to be rasterized first,
so their output can be sampled as a texture for clip/discard
operations, that would also end up in this tree as a dependency... (I think?).

Once we have the entire tree of dependencies, we analyze it to see
which tasks can be combined into a single pass for efficiency.  We
ping-pong rendertargets when we can, but sometimes the dependencies
cut across more than one level of the rendertask tree, and some
copying is necessary.

Once we've figured out the passes and allocated storage for anything
we wish to persist in the texture cache, we finally start rendering.

When rasterizing the elements into the Picture's offscreen texture, we'd
position them by walking the transform hierarchy as far up as the picture's
transform node, resulting in a ``Layout To Picture`` transform.  The picture
would then go onto the page using a ``Picture To World`` coordinate transform.

Caching
```````

Just as with layers in the software rasterizer, it is not always necessary to
redraw absolutely everything when parts of a document change.  The webrender
equivalent of layers is Slices -- a grouping of pictures that are expected to
render and update together.  Slices are automatically created based on
heuristics and layout hints/flags.

Implementation wise, slices reuse a lot of the existing machinery for Pictures;
in fact they're implemented as a "Virtual picture" of sorts.  The similarities
make sense: both need to allocate offscreen textures in a cache, both will
position and render all their children into it, and both then draw themselves
into their parent as part of the parent's draw.

If a slice isn't expected to change much, we give it a TileCacheInstance. It is
itself made up of Tiles, where each tile will track what's in it, what's
changing, and if it needs to be invalidated and redrawn or not as a result.
Thus the "damage" from changes can be localized to single tiles, while we
salvage the rest of the cache.  If tiles keep seeing a lot of invalidations,
they will recursively divide themselves in a quad-tree like structure to try and
localize the invalidations.  (And conversely, they'll recombine children if
nothing is invalidating them "for a while").

Interning
`````````

To spot invalidated tiles, we need a fast way to compare its contents from the
previous frame with the current frame.  To speed this up, we use interning;
similar to string-interning, this means that each ``TextRun``, ``Decoration``,
``Image`` and so on is registered in a repository (a ``DataStore``) and
consequently referred to by its unique ID. Cache contents can then be encoded as a
list of IDs (one such list per internable element type).  Diffing is then just a
fast list comparison.


Callbacks
`````````
GPU text rendering assumes that the individual font-glyphs are already
available in a texture atlas.  Likewise SVG is not being rendered on
the GPU.  Both inputs are prepared during scene building; glyph
rasterization via a thread pool from within Rust itself, and SVG via
opaque callbacks (back to C++) that produce blobs.