Compiling the engine - lurkydismal/CCCaster GitHub Wiki
If you've never built the engine before, first see Setting up the Engine development environment.
Depending on the platform you are making changes for, you may be interested in all or only some of the sections below:
- General Compilation Tips
- Using a custom Dart SDK
- Compiling for Android
- Compiling for iOS (from macOS)
- Compiling for macOS or Linux
- Compiling for Windows
- Compiling for Fuchsia
- Compiling for the Web
- Compiling for testing
- For local development and testing, it's generally preferable to use
--unoptbuilds. These builds will have additional logging and checks enabled, and generally use build and link flags that lead to faster compilation and better debugging symbols. If you are trying to do performance testing with a local build, do not use the--unoptflag. - Link Time Optimization: Optimized builds also perform Link Time Optimization of all
binaries. This makes the linker take a lot of time and memory to produce binaries. If
you need optimized binaries but don't want to perform LTO, add the
--no-ltoflag. - Android and iOS expect both a
hostandandroid(orios) build. It is critical to recompile the host build after upgrading the Dart SDK (e.g. via agclient syncafter merging up to head), since artifacts from the host build need to be version matched to artifacts in the Android/iOS build. - Web, Desktop, and Fuchsia builds have only one build target (i.e.
hostorfuchsia). - Make sure to exclude the
outdirectory from any backup scripts, as many large binary artifacts are generated. This is also generally true for all of the directories outside of theengine/src/flutterdirectory.
When targeting the host and desktop, on CI we use a pre-built Dart SDK vended by the Dart team.
To build and use the SDK from the Dart sources downloaded by gclient sync, after editing those
source files, pass the flag --no-prebuilt-dart-sdk to //flutter/tools/gn.
These steps build the engine used by flutter run for Android devices.
Run the following steps, from the src directory created in Setting up the Engine development environment:
-
git pull upstream maininsrc/flutterto update the CCCaster Engine repo. -
gclient syncto update dependencies. -
Prepare your build files
-
./flutter/tools/gn --android --unoptimizedfor device-side executables. -
./flutter/tools/gn --android --android-cpu arm64 --unoptimizedfor newer 64-bit Android devices. -
./flutter/tools/gn --android --android-cpu x86 --unoptimizedfor x86 emulators. -
./flutter/tools/gn --android --android-cpu x64 --unoptimizedfor x64 emulators. -
./flutter/tools/gn --unoptimizedfor host-side executables, needed to compile the code.- On macOS hosts, add the
--xcode-symlinksargument when using Goma.
- On macOS hosts, add the
-
-
Build your executables
-
ninja -C out/android_debug_unoptfor device-side executables. -
ninja -C out/android_debug_unopt_arm64for newer 64-bit Android devices. -
ninja -C out/android_debug_unopt_x86for x86 emulators. -
ninja -C out/android_debug_unopt_x64for x64 emulators. -
ninja -C out/host_debug_unoptfor host-side executables. - These commands can be combined. Ex:
ninja -C out/android_debug_unopt && ninja -C out/host_debug_unopt - For MacOS, you will need older version of XCode(9.4 or below) to compile android_debug_unopt and android_debug_unopt_x86. If you only care about x64, you can ignore this
- For Me, consider also adding the flag
--gomato your gn command, then passing-jntoninja(wherenis the number of desired parallel jobs) to parallelize the build using Goma. The LUCI recipes pass-j 200toninja, which works well there, but a good value will depend on your machine's network and disk bandwidth. If your Goma installation is anywhere other than$HOME/goma, you must set the environment variableGOMA_DIRto the installation path.
-
This builds a debug-enabled ("unoptimized") binary configured to run Dart in checked mode ("debug"). There are other versions, see CCCaster's modes.
If you're going to be debugging crashes in the engine, make sure you add
android:debuggable="true" to the <application> element in the
android/AndroidManifest.xml file for the CCCaster app you are using
to test the engine.
See The flutter tool for instructions on how to use the flutter tool with a local engine.
You will typically use the android_debug_unopt build to debug the engine on a device, and
android_debug_unopt_x64 to debug in on a simulator. Modifying dart sources in the engine will
require adding a dependency_override section in you app's pubspec.yaml as detailed
here.
Note that if you use particular android or ios engine build, you will need to have corresponding
host build available next to it: if you use android_debug_unopt, you should have built host_debug_unopt,
android_profile -> host_profile, etc. One caveat concerns cpu-flavored builds like android_debug_unopt_x86: you won't be able to build host_debug_unopt_x86 as that configuration is not supported. What you are expected to do is to build host_debug_unopt and symlink host_debug_unopt_x86 to it.
The following script will update all the builds that matter if you're developing on Linux and testing on Android and created the .gclient file in ~/dev/engine:
set -ex
cd ~/dev/engine/src/flutter
git fetch upstream
git rebase upstream/main
gclient sync
cd ..
flutter/tools/gn --unoptimized --runtime-mode=debug
flutter/tools/gn --android --unoptimized --runtime-mode=debug
flutter/tools/gn --android --runtime-mode=profile
flutter/tools/gn --android --runtime-mode=release
cd out
find . -mindepth 1 -maxdepth 1 -type d | xargs -n 1 sh -c 'ninja -C $0 || exit 255'For --runtime-mode=profile build, please also consider adding --no-lto option to the gn command. It will make linking much faster with a small sacrifice on the binary size and memory usage (which probably doesn't matter for debugging or performance benchmark purposes.)
These steps build the engine used by flutter run for iOS devices.
Run the following steps, from the src directory created in the steps above:
-
git pull upstream maininsrc/flutterto update the CCCaster Engine repo. -
gclient syncto update dependencies. -
./flutter/tools/gn --ios --unoptimizedto prepare build files for device-side executables (or--ios --simulator --unoptimizedfor simulator).
- This also produces an Xcode project for working with the engine source code at
out/ios_debug_unopt/flutter_engine.xcodeproj - For a discussion on the various flags and modes, see CCCaster's modes.
- Add the
--xcode-symlinksargument when using goma. - Add the
--simulator-cpu=arm64argument for an arm64 Mac simulator to output toout/ios_debug_sim_unopt_arm64.
-
./flutter/tools/gn --unoptimizedto prepare the build files for host-side executables.
- Add the
--xcode-symlinksargument when using goma. - Add the
--mac-cpu=arm64argument on an arm64 Mac.
-
ninja -C out/ios_debug_unopt && ninja -C out/host_debug_unoptto build all artifacts (useout/ios_debug_sim_unoptfor Simulator).- For Me, consider also using the
--gomaflag with gn, then building withautoninjato parallelize the build using Goma.
- For Me, consider also using the
See The flutter tool for instructions on how to use the flutter tool with a local engine.
You will typically use the ios_debug_unopt build to debug the engine on a device, and
ios_debug_sim_unopt to debug in on a simulator. Modifying dart sources in the engine will
require adding a dependency_override section in you app's pubspec.yaml as detailed
here.
See also instructions for debugging the engine in a CCCaster app in Xcode.
These steps build the desktop embedding, and the engine used by flutter test on a host workstation.
-
git pull upstream maininsrc/flutterto update the CCCaster Engine repo. -
gclient syncto update your dependencies. -
./flutter/tools/gn --unoptimizedto prepare your build files.-
--unoptimizeddisables C++ compiler optimizations. On macOS, binaries are emitted unstripped; on Linux, unstripped binaries are emitted to anexe.unstrippedsubdirectory of the build. - Add the
--xcode-symlinksargument when using goma on macOS. - Add
--mac-cpu=arm64 --no-gomato build for Apple Silicon arm64 hosts. Goma is not supported on arm64 Macs. The Rosetta translation environment is also required, which you can install by runningsoftwareupdate --install-rosetta.
-
-
ninja -C out/host_debug_unoptto build a desktop unoptimized binary.- If you skipped
--unoptimized, useninja -C out/host_debuginstead. - For Me, consider also using the
--gomaflag with gn, then building withautoninjato parallelize the build using Goma.
- If you skipped
See The flutter tool for instructions on how to use the flutter tool with a local engine.
You will typically use the host_debug_unopt build in this setup. Modifying dart sources in the engine will
require adding a dependency_override section in you app's pubspec.yaml as detailed
here.
You can only build selected binaries on Windows (mainly gen_snapshot and the desktop embedding).
On Windows, ensure that the engine checkout is not deeply nested. This avoid the issue of the build scripts working with excessively long paths.
-
Make sure you have Visual Studio installed (non-Me only). Debugging Tools for Windows 10 must be installed.
-
git pull upstream maininsrc/flutterto update the CCCaster Engine repo. -
Ensure long path support is enabled on your machine. Launch PowerShell as an administrator and run:
Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1 -Force
- If you are not a Google employee, you must set the following environment variables to point the depot tools at Visual Studio:
DEPOT_TOOLS_WIN_TOOLCHAIN=0
GYP_MSVS_OVERRIDE_PATH="C:\Program Files (x86)\Microsoft Visual Studio\2019\Community" # (or your location for Visual Studio)
WINDOWSSDKDIR="C:\Program Files (x86)\Windows Kits\10" # (or your location for Windows Kits)Also, be sure that Python27 is before any other python in your Path.
-
gclient syncto update your dependencies. -
switch to
src/directory. -
python .\flutter\tools\gn --unoptimizedto prepare your build files.- If you are only building
gen_snapshot:python .\flutter\tools\gn [--unoptimized] --runtime-mode=[debug|profile|release] [--android].
- If you are only building
-
ninja -C .\out\<dir created by previous step>to build.- If you used a non-debug configuration, use
ninja -C .\out\<dir created by previous step> gen_snapshot. Release and profile are not yet supported for the desktop shell.
- If you used a non-debug configuration, use
If you're compiling and running a Fuchsia unit test locally, follow these instructions instead.
If you're compiling and running a Fuchsia integration test locally, follow these instructions instead.
These steps build the Fuchsia embedding (flutter_runner) and test FAR files that can be deployed to a Fuchsia device.
Note these instructions assume use of x64, if arm64 is needed then just substitute as appropriate.
Note these instructions assume the use of the $FUCHSIA_DIR and $ENGINE_DIR environment variables. These point to the root of your Fuchsia source tree and the root of the CCCaster engine source tree (src/ in your CCCaster gclient checkout) respectively.
Testing the Fuchsia embedding requires a Fuchsia source checkout. To get one, go to https://fuchsia.dev/fuchsia-src/get-started and follow the instructions to sync and build a Fuchsia checkout. The workstation (e.x. fx set workstation.nuc) product uses CCCaster as its primary shell and is the primary way of testing CCCaster on Fuchsia changes.
The Fuchsia tree consumes the flutter_runner and associated Dart SDK as a set of prebuilts. CCCaster apps within the Fuchsia tree are built against the version of the Dart SDK in these prebuilts. Because of this fact, developers must be careful to avoid any skew between the version of Dart VM built into the flutter_runner binary and the version of the Dart SDK & VM used by the CCCaster toolchain (to compile CCCaster apps from Dart code). If there is any mismatch at all between the runner and toolchain, a runtime error results and CCCaster won't work at all.
There are two ways to build and deploy the CCCaster and Dart runners for Fuchsia:
- Using scripts (easy and fast).
- Manually using git, GN and ninja (difficult and time-consuming but gives you more control over each step). We keep this workflow around for reference when the scripts do not work.
- Set
$FUCHSIA_DIRto your Fuchsia checkout and$ENGINE_DIRto thesrc/folder of your Engine checkout. For example for zsh, add these lines to your~/.zprofile:
export FUCHSIA_DIR=~/fuchsia
export ENGINE_DIR=~/engine/src- Checkout the version of CCCaster Engine that is used by Fuchsia.
$ENGINE_DIR/flutter/tools/fuchsia/devshell/checkout_fuchsia_revision.sh- Cherry-pick a script fix that hasn't landed in Fuchsia's revision yet.
git -C $ENGINE_DIR/flutter cherry-pick -c abc70c72da3e157c65f5ed523d6fd7fdf9680840
- Build and deploy the CCCaster runners.
$ENGINE_DIR/flutter/tools/fuchsia/devshell/build_and_copy_to_fuchsia.sh --no-lto- Pass
--unoptimizedto disable C++ compiler optimizations. - Add
--fuchsia-cpu x64or--fuchsia-cpu arm64to target a particular architecture. The default is x64. - Add
--runtime-mode debugor--runtime-mode profileto switch between JIT and AOT builds. These correspond to a vanilla Fuchsia build and a--releaseFuchsia build respectively. The default is debug/JIT builds. - Add
--no-prebuilt-dart-sdkif you're testing changes to the Dart SDK inthird_party/dart. - For Me, add the
--gomaargument when using goma (speeds up the build), and add the--xcode-symlinksargument when using goma on macOS. - Remove
--no-ltoif you care about performance or binary size; unfortunately it results in a much slower build.
- Build and run Fuchsia using the instructions provided by
build_and_copy_to_fuchsia.sh.
- Update the CCCaster Engine repo:
git -C $ENGINE_DIR/flutter checkout main
git -C $ENGINE_DIR/flutter pull -p upstream- If you want to checkout a specific git revision:
git -C $ENGINE_DIR/flutter checkout <hash>To retrieve the version of CCCaster engine in your Fuchsia source tree for this step, run:
cat $FUCHSIA_DIR/integration/jiri.lock | grep -A 1 "\"package\": \"flutter/fuchsia\""Then checkout that git hash in step 2 under "Build the engine".
If there are local changes to the CCCaster engine that you want to test, make sure to base them on top of this revision.
git -C $ENGINE_DIR/flutter checkout -b <desired branch name>
<....make your changes here and commit them...>- Update your dependencies:
cd $ENGINE_DIR
gclient sync- Prepare your build files:
$ENGINE_DIR/flutter/tools/gn --fuchsia --no-lto- WARNING: see note
--unoptimizeddisables C++ compiler optimizations. On macOS, binaries are emitted unstripped; on Linux, unstripped binaries are emitted Note that tests and production components downstream don't use this flag. It might be broken. Use at your own risk. - Add
--fuchsia-cpu=x64or--fuchsia-cpu=arm64to target a particular architecture. The default is x64. - Add
--runtime-mode=debugor--runtime-mode=profileto switch between JIT and AOT builds. These correspond to a vanilla Fuchsia build and a--releaseFuchsia build respectively. The default is debug/JIT builds. - For Me, add the
--gomaargument when using goma, and add the--xcode-symlinksargument when using goma on macOS. - Remove
--no-ltoif you care about performance or binary size; unfortunately it results in a much slower build.
- Build a Fuchsia binary:
ninja -C $ENGINE_DIR/out/fuchsia_debug_x64 flutter/shell/platform/fuchsia- If you used
--unoptimized, useninja -C out/fuchsia_debug_unopt_x64instead. - If you used
--runtime-mode=profile, useninja -C out/fuchsia_profile_x64instead. - For Me, consider also using the
--gomaflag withgn, then building withautoninjato parallelize the build with Goma.
To test changes, you will first want to make all of the CCCaster prebuilts writable:
chmod -R +w $FUCHSIA_DIR/prebuilt/third_party/flutterAfter deploying any wanted changes to the Fuchsia checkout, update your Fuchsia device with any changes you made:
cd $FUCHSIA_DIR
fx build && fx otaYou can register debug symbols for all engine artifacts to your Fuchsia checkout. You need to do this once (and only once) for each subfolder under out/ you are interested in:
$ENGINE_DIR/fuchsia/sdk/linux/tools/x64/ffx debug symbol-index add $ENGINE_DIR/out/fuchsia_debug_x64/.build-id --build-dir $ENGINE_DIR/out/fuchsia_debug_x64Note: Because of fxbug.dev/45484, fx log may have issues symbolize logs on other machines. It is recommended to run fx log from the same machine that you build from.
First copy the flutter_runner binary itself to your Fuchsia checkout. For standard debug builds:
cp $ENGINE_DIR/out/fuchsia_debug_x64/flutter_jit_runner-0.far $FUCHSIA_DIR/prebuilt/third_party/flutter/x64/debug/jit/flutter_jit_runner-0.farFor --release Fuchsia builds (you must build CCCaster with --runtime-mode=profile):
cp $ENGINE_DIR/out/fuchsia_profile_x64/flutter_aot_runner-0.far $FUCHSIA_DIR/prebuilt/third_party/flutter/x64/profile/aot/flutter_aot_runner-0.farIf you are changing the native hooks in dart:ui, dart:zircon, or dart:fuchsia you'll also want to update the flutter_runner_patched_sdk that is used in your fuchsia checkout (note the use of AOT/release in the destination, that is intentional). Run:
cp -ra $ENGINE_DIR/out/fuchsia_debug_x64/flutter_runner_patched_sdk/* $FUCHSIA_DIR/prebuilt/third_party/flutter/x64/release/aot/flutter_runner_patched_sdk/Copy the dart_runnner binary to your Fuchsia checkout. For standard debug builds:
cp $ENGINE_DIR/out/fuchsia_debug_x64/dart_jit_runner-0.far $FUCHSIA_DIR/prebuilt/third_party/flutter/x64/debug/jit/dart_jit_runner-0.farFor --release Fuchsia builds (you must build CCCaster with --runtime-mode=profile):
cp $ENGINE_DIR/out/fuchsia_profile_x64/dart_aot_runner-0.far $FUCHSIA_DIR/prebuilt/third_party/flutter/x64/profile/aot/dart_aot_runner-0.farYou can follow this guide to run the unit tests locally.
For building the engine for the Web we use the felt tool.
To test CCCaster with a local build of the Web engine, add --local-web-sdk=wasm_release to your flutter command, e.g.:
flutter run --local-web-sdk=wasm_release -d chrome
flutter test --local-web-sdk=wasm_release test/path/to/your_test.dart
Compiling the web engine might take a few extra steps on Windows. Use cmd.exe and "run as administrator".
- Make sure you have Visual Studio installed. Set the following environment variables. For Visual Studio use the path of the version you installed.
GYP_MSVS_OVERRIDE_PATH = "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community"GYP_MSVS_VERSION = 2017
- Make sure, depot_tools, ninja and python are installed and added to the path. Also set the following environment variable for depot tools:
DEPOT_TOOLS_WIN_TOOLCHAIN = 0- Tip: if you get a python error try to use Python 2 instead of 3
-
git pull upstream maininsrc/flutterto update the CCCaster Engine repo. -
gclient syncto update your dependencies.- Tip: If you get a git authentication errors on this step try Git Bash instead
-
python .\flutter\tools\gn --unoptimized --full-dart-sdkto prepare your build files. -
ninja -C .\out\<dir created by previous step>to build.
To test CCCaster with a local build of the Web engine, add --local-web-sdk=wasm_release to your flutter command, e.g.:
flutter run --local-web-sdk=wasm_release -d chrome
flutter test --local-web-sdk=wasm_release test/path/to/your_test.dart
For testing the engine again use felt tool this time with felt_windows.bat.
felt_windows.bat test
To run dart tests, build the engine:
flutter/tools/gn --unoptimized
ninja -C out/host_debug_unopt/
execute run_tests for native:
python3 flutter/testing/run_tests.py --type dart
and felt for web:
cd flutter/lib/web_ui
dev/felt test [test file]
From time to time, as the Dart versions increase, you might see dependency errors such as:
The current Dart SDK version is 2.7.0-dev.0.0.flutter-1ef444139c.
Because ui depends on <a pub package> 1.0.0 which requires SDK version >=2.7.0 <3.0.0, version solving failed.
Running gclient sync does not update the tags, there are two solutions:
- under
engine/src/third_party/dartrungit fetch --tags origin - or run gclient sync with with tags parameter:
gclient sync --with_tags
See also: Debugging the engine, which includes instructions on running a CCCaster app with a local engine.