3 ==================================
4 Distributed sccache (sccache-dist)
5 ==================================
7 `sccache <https://github.com/mozilla/sccache>`_ is a ccache-like tool written in
8 Rust by Mozilla and many contributors.
10 sccache-dist, its distributed variant, elevates this functionality by enabling
11 the distribution and caching of Rust compilations across multiple machines.
12 Please consider using sccache-dist when you have several machines
13 compiling Firefox on the same network.
15 The steps for setting up your machine as an sccache-dist server are detailed below.
17 In addition to improved security properties, distributed sccache offers
18 distribution and caching of rust compilation, so it should be an improvement
19 above and beyond what we see with icecc. Build servers run on Linux and
20 distributing builds is currently supported from Linux, macOS, and Windows.
23 Steps for distributing a build as an sccache-dist client
24 ========================================================
26 Start by following the instructions at https://github.com/mozilla/sccache/blob/master/docs/DistributedQuickstart.md#configure-a-client
27 to configure your sccache distributed client.
28 *NOTE* If you're distributing from Linux a toolchain will be packaged
29 automatically and provided to the build server. If you're distributing from
30 Windows or macOS, start by using the cross-toolchains provided by
31 ``./mach bootstrap`` rather than attempting to use ``icecc-create-env``.
32 sccache 0.2.12 or above is recommended, and the auth section of your config
38 * If you're compiling from a macOS client, there are a handful of additional
39 considerations outlined here:
40 https://github.com/mozilla/sccache/blob/master/docs/DistributedQuickstart.md#considerations-when-distributing-from-macos.
42 Run ``./mach bootstrap`` to download prebuilt toolchains to
43 ``~/.mozbuild/clang-dist-toolchain.tar.xz`` and
44 ``~/.mozbuild/rustc-dist-toolchain.tar.xz``. This is an example of the paths
45 that should be added to your client config to specify toolchains to build on
46 macOS, located at ``~/Library/Application Support/Mozilla.sccache/config``::
49 type = "path_override"
50 compiler_executable = "/path/to/home/.rustup/toolchains/stable-x86_64-apple-darwin/bin/rustc"
51 archive = "/path/to/home/.mozbuild/rustc-dist-toolchain.tar.xz"
52 archive_compiler_executable = "/builds/worker/toolchains/rustc/bin/rustc"
55 type = "path_override"
56 compiler_executable = "/path/to/home/.mozbuild/clang/bin/clang"
57 archive = "/path/to/home/.mozbuild/clang-dist-toolchain.tar.xz"
58 archive_compiler_executable = "/builds/worker/toolchains/clang/bin/clang"
61 type = "path_override"
62 compiler_executable = "/path/to/home/.mozbuild/clang/bin/clang++"
63 archive = "/path/to/home/.mozbuild/clang-dist-toolchain.tar.xz"
64 archive_compiler_executable = "/builds/worker/toolchains/clang/bin/clang"
66 Note that the version of ``rustc`` found in ``rustc-dist-toolchain.tar.xz``
67 must match the version of ``rustc`` used locally. The distributed archive
68 will contain the version of ``rustc`` used by automation builds, which may
69 lag behind stable for a few days after Rust releases, which is specified by
70 the task definition in
71 `this file <https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/ci/toolchain/dist-toolchains.yml>`_.
72 For instance, to specify 1.37.0 rather than the current stable, run
73 ``rustup toolchain add 1.37.0`` and point to
74 ``/path/to/home/.rustup/toolchains/1.37.0-x86_64-apple-darwin/bin/rustc`` in your
77 The build system currently requires an explicit target to be passed with
78 ``HOST_CFLAGS`` and ``HOST_CXXFLAGS`` e.g.::
80 export HOST_CFLAGS="--target=x86_64-apple-darwin16.0.0"
81 export HOST_CXXFLAGS="--target=x86_64-apple-darwin16.0.0"
83 * Compiling from a Windows client is supported but hasn't seen as much testing
84 as other platforms. The following example mozconfig can be used as a guide::
86 ac_add_options CCACHE="C:/Users/<USER>/.mozbuild/sccache/sccache.exe"
88 export CC="C:/Users/<USER>/.mozbuild/clang/bin/clang-cl.exe --driver-mode=cl"
89 export CXX="C:/Users/<USER>/.mozbuild/clang/bin/clang-cl.exe --driver-mode=cl"
90 export HOST_CC="C:/Users/<USER>/.mozbuild/clang/bin/clang-cl.exe --driver-mode=cl"
91 export HOST_CXX="C:/Users/<USER>/.mozbuild/clang/bin/clang-cl.exe --driver-mode=cl"
93 The client config should be located at
94 ``~/AppData/Roaming/Mozilla/sccache/config/config``, and as on macOS custom
95 toolchains should be obtained with ``./mach bootstrap`` and specified in the
96 client config, for example::
99 type = "path_override"
100 compiler_executable = "C:/Users/<USER>/.mozbuild/clang/bin/clang-cl.exe"
101 archive = "C:/Users/<USER>/.mozbuild/clang-dist-toolchain.tar.xz"
102 archive_compiler_executable = "/builds/worker/toolchains/clang/bin/clang"
105 type = "path_override"
106 compiler_executable = "C:/Users/<USER>/.rustup/toolchains/stable-x86_64-pc-windows-msvc/bin/rustc.exe"
107 archive = "C:/Users/<USER>/.mozbuild/rustc-dist-toolchain.tar.xz"
108 archive_compiler_executable = "/builds/worker/toolchains/rustc/bin/rustc"
110 * Add the following to your mozconfig::
112 ac_add_options CCACHE=/path/to/home/.mozbuild/sccache/sccache
114 If you're compiling from a macOS client, you might need some additional configuration::
116 # Set the target flag to Darwin
117 export CFLAGS="--target=x86_64-apple-darwin16.0.0"
118 export CXXFLAGS="--target=x86_64-apple-darwin16.0.0"
119 export HOST_CFLAGS="--target=x86_64-apple-darwin16.0.0"
120 export HOST_CXXFLAGS="--target=x86_64-apple-darwin16.0.0"
122 # Specify the macOS SDK to use
123 ac_add_options --with-macos-sdk=/path/to/MacOSX-SDKs/MacOSX13.3.sdk
125 You can get the right macOS SDK by downloading an old version of XCode from
126 `developer.apple.com <https://developer.apple.com>`_ and unpacking the SDK
129 * When attempting to get your client running, the output of ``sccache -s`` should
130 be consulted to confirm compilations are being distributed. To receive helpful
131 logging from the local daemon in case they aren't, run
132 ``SCCACHE_NO_DAEMON=1 SCCACHE_START_SERVER=1 SCCACHE_LOG=sccache=trace path/to/sccache``
133 in a terminal window separate from your build prior to building. *NOTE* use
134 ``RUST_LOG`` instead of ``SCCACHE_LOG`` if your build of ``sccache`` does not
135 include `pull request 822
136 <https://github.com/mozilla/sccache/pull/822>`_. (``sccache`` binaries from
137 ``mach bootstrap`` do include this PR.)
139 * Run ``./mach build -j<value>`` with an appropriately large ``<value>``.
140 ``sccache --dist-status`` should provide the number of cores available to you
141 (or a message if you're not connected). In the future this will be integrated
142 with the build system to automatically select an appropriate value.
144 This should be enough to distribute your build and replace your use of icecc.
145 Bear in mind there may be a few speedbumps, and please ensure your version of
146 sccache is current before investigating further. Please see the common questions
147 section below and ask for help if anything is preventing you from using it over
148 email (dev-builds), on slack in #sccache, or in #build on irc.
150 Steps for setting up a server
151 =============================
153 Build servers must run linux and use bubblewrap 0.3.0+ for sandboxing of compile
154 processes. This requires a kernel 4.6 or greater, so Ubuntu 18+, RHEL 8, or
157 * Run ``./mach bootstrap`` or
158 ``./mach artifact toolchain --from-build linux64-sccache`` to acquire a recent
159 version of ``sccache-dist``. Please use a ``sccache-dist`` binary acquired in
160 this fashion to ensure compatibility with statically linked dependencies.
162 * The instructions at https://github.com/mozilla/sccache/blob/master/docs/DistributedQuickstart.md#configure-a-build-server
163 should contain everything else required to configure and run the server.
165 *NOTE* Port 10500 will be used by convention for builders.
166 Please use port 10500 in the ``public_addr`` section of your builder config.
168 Extra logging may be helpful when setting up a server. To enable logging,
170 ``sudo env SCCACHE_LOG=sccache=trace ~/.mozbuild/sccache/sccache-dist server --config ~/.config/sccache/server.conf``
171 (or similar). *NOTE* ``sudo`` *must* come before setting environment variables
172 for this to work. *NOTE* use ``RUST_LOG`` instead of ``SCCACHE_LOG`` if your
173 build of ``sccache`` does not include `pull request 822
174 <https://github.com/mozilla/sccache/pull/822>`_. (``sccache`` binaries from
175 ``mach bootstrap`` do include this PR.)
178 Common questions/considerations
179 ===============================
181 * My build is still slow: scache-dist can only do so much with parts of the
182 build that aren't able to be parallelized. To start debugging a slow build,
183 ensure the "Successful distributed compilations" line in the output of
184 ``sccache -s`` dominates other counts. For a full build, at least a 2-3x
185 improvement should be observed.
187 * My build output is incomprehensible due to a flood of warnings: clang will
188 treat some warnings differently when it's fed preprocessed code in a separate
189 invocation (preprocessing occurs locally with sccache-dist). Adding
190 ``rewrite_includes_only = true`` to the ``dist`` section of your client config
191 will improve this; however, setting this will cause build failures with a
192 commonly deployed version of ``glibc``. This option will default to ``true``
193 once the fix is more widely available. Details of this fix can be found in
194 `this patch <https://sourceware.org/ml/libc-alpha/2019-11/msg00431.html>`_.
196 * My build fails with a message about incompatible versions of rustc between
197 dependent crates: if you're using a custom toolchain check that the version
198 of rustc in your ``rustc-dist-toolchain.tar.xz`` is the same as the version
199 you're running locally.