1 Off Main Thread Painting
2 ========================
4 OMTP, or ‘off main thread painting’, is the component of Gecko that
5 allows us to perform painting of web content off of the main thread.
6 This gives us more time on the main thread for javascript, layout,
7 display list building, and other tasks which allows us to increase our
10 Take a look at this `blog
11 post <https://mozillagfx.wordpress.com/2017/12/05/off-main-thread-painting/>`__
17 Painting (or rasterization) is the last operation that happens in a
18 layer transaction before we forward it to the compositor. At this point,
19 all display items have been assigned to a layer and invalid regions have
20 been calculated and assigned to each layer.
22 The painted layer uses a content client to acquire a buffer for
23 painting. The main purpose of the content client is to allow us to
24 retain already painted content when we are scrolling a layer. We have
25 two main strategies for this, rotated buffer and tiling.
27 This is implemented with two class hierarchies. ``ContentClient`` for
28 rotated buffer and ``TiledContentClient`` for tiling. Additionally we
29 have two different painted layer implementations, ``ClientPaintedLayer``
30 and ``ClientTiledPaintedLayer``.
32 The main distinction between rotated buffer and tiling is the amount of
33 graphics surfaces required. Rotated buffer utilizes just a single buffer
34 for a frame but potentially requires painting into it multiple times.
35 Tiling uses multiple buffers but doesn’t require painting into the
36 buffers multiple times.
38 Once the painted layer has a surface (or surfaces with tiling) to paint
39 into, they are wrapped in a ``DrawTarget`` of some form and a callback
40 to ``FrameLayerBuilder`` is called. This callback uses the assigned
41 display items and invalid regions to trigger rasterization. Each
42 ``nsDisplayItem`` has their ``Paint`` method called with the provided
43 ``DrawTarget`` that represents the surface, and they paint into it.
48 The key abstraction that allows us to paint off of the main thread is
49 ``DrawTargetCapture`` [1]_. ``DrawTargetCapture`` is a special
50 ``DrawTarget`` which records all draw commands for replaying to another
51 draw target in the local process. This is similar to
52 ``DrawTargetRecording``, but only holds a reference to resources instead
53 of copying them into the command stream. This allows the command stream
54 to be much more lightweight than ``DrawTargetRecording``.
56 OMTP works by instrumenting the content clients to use a capture target
57 for all painting [2]_ [3]_ [4]_ [5]_. This capture draw target records all
58 the operations that would normally be performed directly on the
59 surface’s draw target. Once we have all of the commands, we send the
60 capture and surface draw target to the ``PaintThread`` [6]_ where the
61 commands are replayed onto the surface. Once the rasterization is done,
62 we forward the layer transaction to the compositor.
64 Tiling and parallel painting
65 ----------------------------
67 We can make one additional improvement if we are using tiling as our
68 content client backend.
70 When we are tiling, the screen is subdivided into a grid of equally
71 sized surfaces and draw commands are performed on the tiles they affect.
72 Each tile is independent of the others, so we’re able to parallelize
73 painting by using a worker thread pool and dispatching a task for each
76 This is commonly referred to as P-OMTP or parallel painting.
78 Main thread rasterization
79 -------------------------
81 Even with OMTP it’s still possible for the main thread to perform
82 rasterization. A common pattern for painting code is to create a
83 temporary draw target, perform drawing with it, take a snapshot, and
84 then draw the snapshot onto the main draw target. This is done for
85 blurs, box shadows, text shadows, and with the basic layer manager
88 If the temporary draw target is not a draw target capture, then this
89 will perform rasterization on the main thread. This can be bad as it
90 lowers our parallelism and can cause contention with content backends,
91 like Direct2D, that use locking around shared resources.
93 To work around this, we changed the main thread painting code to use a
94 draw target capture for these operations and added a source surface
95 capture [7]_ which only resolves the painting of the draw commands when
96 needed on the paint thread.
98 There are still possible cases we can perform main thread rasterization,
99 but we try and address them when they come up.
104 The web is very complex, and so we can sometimes have a large amount of
105 draw commands for a content paint. We’ve observed OOM errors for capture
106 command lists that have grown to be 200MiB large.
108 We initially tried to mitigate this by lowering the overhead of capture
109 command lists. We do this by filtering commands that don’t actually
110 change the draw target state and folding consecutive transform changes,
111 but that was not always enough. So we added the ability for our draw
112 target capture’s to flush their command lists to the surface draw target
113 while we are capturing on the main thread [8]_.
115 This is triggered by a configurable memory limit. Because this
116 introduces a new source of main thread rasterization we try to balance
117 setting this too low and suffering poor performance, or setting this too
118 high and suffering crashes.
123 OMTP is conceptually simple, but in practice it relies on subtle code to
124 ensure thread safety. This was the most arguably the most difficult part
127 There are roughly four areas that are critical.
129 1. Compositor message ordering
131 Immediately after we queue the async paints to be asynchronously
132 completed, we have a problem. We need to forward the layer
133 transaction at some point, but the compositor cannot process the
134 transaction until all async paints have finished. If it did, it could
135 access unfinished painted content.
137 We obviously can’t block on the async paints completing as that would
138 beat the whole point of OMTP. We also can’t hold off on sending the
139 layer transaction to ``IPDL``, as we’d trigger race conditions for
140 messages sent after the layer transaction is built but before it is
141 forwarded. Reftest and other code assumes that messages sent after a
142 layer transaction to the compositor are processed after that layer
143 transaction is processed.
145 The solution is to forward the layer transaction to the compositor
146 over ``IPDL``, but flag the message channel to start postponing
147 messages [9]_. Then once all async paints have completed, we unflag
148 the message channel and all postponed messages are sent [10]_. This
149 allows us to keep our message ordering guarantees and not have to
150 worry about scheduling a runnable in the future.
154 The backing store for content surfaces is managed by texture client.
155 While async paints are executing, it’s possible for shutdown or any
156 number of things to happen that could cause layer manager, all
157 layers, all content clients, and therefore all texture clients to be
158 destroyed. Therefore it’s important that we keep these texture
159 clients alive throughout async painting. Texture clients also manage
160 IPC resources and must be destroyed on the main thread, so we are
161 careful to do that [11]_.
165 We currently double buffer our content painting - our content clients
166 only ever have zero or one texture that is available to be painted
169 This implies that we cannot start async painting a layer tree while
170 previous async paints are still active as this would lead to awful
171 races. We also don’t support multiple nested sets of postponed IPC
172 messages to allow sending the first layer transaction to the
173 compositor, but not the second.
175 To prevent issues with this, we flush all active async paints before
176 we begin to paint a new layer transaction [12]_.
178 There was some initial debate about implementing triple buffering for
179 content painting, but we have not seen evidence it would help us
182 4. Moz2D thread safety
184 Finally, most Moz2D objects were not thread safe. We had to insert
185 special locking into draw target and source surface as they have a
186 special copy on write relationship that must be consistent even if
187 they are on different threads.
189 Some platform specific resources like fonts needed locking added in
190 order to be thread safe. We also did some work to make filter nodes
191 work with multiple threads executing them at the same time.
196 Currently only content processes are able to use OMTP.
198 This restriction was added because of concern about message ordering
199 between ``APZ`` and OMTP. It might be able to lifted in the future.
204 1. `OMTP Meta <https://bugzilla.mozilla.org/show_bug.cgi?id=omtp>`__
206 Windows <https://bugzilla.mozilla.org/show_bug.cgi?id=1403935>`__
208 OSX <https://bugzilla.mozilla.org/show_bug.cgi?id=1422392>`__
210 Linux <https://bugzilla.mozilla.org/show_bug.cgi?id=1432531>`__
212 painting <https://bugzilla.mozilla.org/show_bug.cgi?id=1425056>`__
217 .. [1] `DrawTargetCapture <https://searchfox.org/mozilla-central/rev/dd965445ec47fbf3cee566eff93b301666bda0e1/gfx/2d/DrawTargetCapture.h#22>`__
218 .. [2] `Creating DrawTargetCapture for rotated
219 buffer <https://searchfox.org/mozilla-central/rev/dd965445ec47fbf3cee566eff93b301666bda0e1/gfx/layers/client/ContentClient.cpp#185>`__
220 .. [3] `Dispatch DrawTargetCapture for rotated
221 buffer <https://searchfox.org/mozilla-central/rev/dd965445ec47fbf3cee566eff93b301666bda0e1/gfx/layers/client/ClientPaintedLayer.cpp#99>`__
222 .. [4] `Creating DrawTargetCapture for
223 tiling <https://searchfox.org/mozilla-central/rev/dd965445ec47fbf3cee566eff93b301666bda0e1/gfx/layers/client/TiledContentClient.cpp#714>`__
224 .. [5] `Dispatch DrawTargetCapture for
225 tiling <https://searchfox.org/mozilla-central/rev/dd965445ec47fbf3cee566eff93b301666bda0e1/gfx/layers/client/MultiTiledContentClient.cpp#288>`__
226 .. [6] `PaintThread <https://searchfox.org/mozilla-central/rev/dd965445ec47fbf3cee566eff93b301666bda0e1/gfx/layers/PaintThread.h#53>`__
227 .. [7] `SourceSurfaceCapture <https://searchfox.org/mozilla-central/rev/dd965445ec47fbf3cee566eff93b301666bda0e1/gfx/2d/SourceSurfaceCapture.h#19>`__
228 .. [8] `Sync flushing draw
229 commands <https://searchfox.org/mozilla-central/rev/dd965445ec47fbf3cee566eff93b301666bda0e1/gfx/2d/DrawTargetCapture.h#165>`__
230 .. [9] `Postponing messages for
231 PCompositorBridge <https://searchfox.org/mozilla-central/rev/dd965445ec47fbf3cee566eff93b301666bda0e1/gfx/layers/ipc/CompositorBridgeChild.cpp#1319>`__
232 .. [10] `Releasing messages for
233 PCompositorBridge <https://searchfox.org/mozilla-central/rev/dd965445ec47fbf3cee566eff93b301666bda0e1/gfx/layers/ipc/CompositorBridgeChild.cpp#1303>`__
234 .. [11] `Releasing texture clients on main
235 thread <https://searchfox.org/mozilla-central/rev/dd965445ec47fbf3cee566eff93b301666bda0e1/gfx/layers/ipc/CompositorBridgeChild.cpp#1170>`__
236 .. [12] `Flushing async
237 paints <https://searchfox.org/mozilla-central/rev/dd965445ec47fbf3cee566eff93b301666bda0e1/gfx/layers/client/ClientLayerManager.cpp#289>`__