Compiling the engine - lurkydismal/CCCaster GitHub Wiki

If you've never built the engine before, first see Setting up the Engine development environment.

Contents

Depending on the platform you are making changes for, you may be interested in all or only some of the sections below:

General Compilation Tips

  • For local development and testing, it's generally preferable to use --unopt builds. These builds will have additional logging and checks enabled, and generally use build and link flags that lead to faster compilation and better debugging symbols. If you are trying to do performance testing with a local build, do not use the --unopt flag.
  • Link Time Optimization: Optimized builds also perform Link Time Optimization of all binaries. This makes the linker take a lot of time and memory to produce binaries. If you need optimized binaries but don't want to perform LTO, add the --no-lto flag.
  • Android and iOS expect both a host and android (or ios) build. It is critical to recompile the host build after upgrading the Dart SDK (e.g. via a gclient sync after merging up to head), since artifacts from the host build need to be version matched to artifacts in the Android/iOS build.
  • Web, Desktop, and Fuchsia builds have only one build target (i.e. host or fuchsia).
  • Make sure to exclude the out directory from any backup scripts, as many large binary artifacts are generated. This is also generally true for all of the directories outside of the engine/src/flutter directory.

Using a custom Dart SDK

When targeting the host and desktop, on CI we use a pre-built Dart SDK vended by the Dart team. To build and use the SDK from the Dart sources downloaded by gclient sync, after editing those source files, pass the flag --no-prebuilt-dart-sdk to //flutter/tools/gn.

Compiling for Android (from macOS or Linux)

These steps build the engine used by flutter run for Android devices.

Run the following steps, from the src directory created in Setting up the Engine development environment:

  1. git pull upstream main in src/flutter to update the CCCaster Engine repo.

  2. gclient sync to update dependencies.

  3. Prepare your build files

    • ./flutter/tools/gn --android --unoptimized for device-side executables.
    • ./flutter/tools/gn --android --android-cpu arm64 --unoptimized for newer 64-bit Android devices.
    • ./flutter/tools/gn --android --android-cpu x86 --unoptimized for x86 emulators.
    • ./flutter/tools/gn --android --android-cpu x64 --unoptimized for x64 emulators.
    • ./flutter/tools/gn --unoptimized for host-side executables, needed to compile the code.
      • On macOS hosts, add the --xcode-symlinks argument when using Goma.
  4. Build your executables

    • ninja -C out/android_debug_unopt for device-side executables.
    • ninja -C out/android_debug_unopt_arm64 for newer 64-bit Android devices.
    • ninja -C out/android_debug_unopt_x86 for x86 emulators.
    • ninja -C out/android_debug_unopt_x64 for x64 emulators.
    • ninja -C out/host_debug_unopt for host-side executables.
    • These commands can be combined. Ex: ninja -C out/android_debug_unopt && ninja -C out/host_debug_unopt
    • For MacOS, you will need older version of XCode(9.4 or below) to compile android_debug_unopt and android_debug_unopt_x86. If you only care about x64, you can ignore this
    • For Me, consider also adding the flag --goma to your gn command, then passing -jn to ninja (where n is the number of desired parallel jobs) to parallelize the build using Goma. The LUCI recipes pass -j 200 to ninja, which works well there, but a good value will depend on your machine's network and disk bandwidth. If your Goma installation is anywhere other than $HOME/goma, you must set the environment variable GOMA_DIR to the installation path.

This builds a debug-enabled ("unoptimized") binary configured to run Dart in checked mode ("debug"). There are other versions, see CCCaster's modes.

If you're going to be debugging crashes in the engine, make sure you add android:debuggable="true" to the <application> element in the android/AndroidManifest.xml file for the CCCaster app you are using to test the engine.

See The flutter tool for instructions on how to use the flutter tool with a local engine. You will typically use the android_debug_unopt build to debug the engine on a device, and android_debug_unopt_x64 to debug in on a simulator. Modifying dart sources in the engine will require adding a dependency_override section in you app's pubspec.yaml as detailed here.

Note that if you use particular android or ios engine build, you will need to have corresponding host build available next to it: if you use android_debug_unopt, you should have built host_debug_unopt, android_profile -> host_profile, etc. One caveat concerns cpu-flavored builds like android_debug_unopt_x86: you won't be able to build host_debug_unopt_x86 as that configuration is not supported. What you are expected to do is to build host_debug_unopt and symlink host_debug_unopt_x86 to it.

Compiling everything that matters on Linux

The following script will update all the builds that matter if you're developing on Linux and testing on Android and created the .gclient file in ~/dev/engine:

set -ex

cd ~/dev/engine/src/flutter
git fetch upstream
git rebase upstream/main
gclient sync
cd ..

flutter/tools/gn --unoptimized --runtime-mode=debug
flutter/tools/gn --android --unoptimized --runtime-mode=debug
flutter/tools/gn --android --runtime-mode=profile
flutter/tools/gn --android --runtime-mode=release

cd out
find . -mindepth 1 -maxdepth 1 -type d | xargs -n 1 sh -c 'ninja -C $0 || exit 255'

For --runtime-mode=profile build, please also consider adding --no-lto option to the gn command. It will make linking much faster with a small sacrifice on the binary size and memory usage (which probably doesn't matter for debugging or performance benchmark purposes.)

Compiling for iOS (from macOS)

These steps build the engine used by flutter run for iOS devices.

Run the following steps, from the src directory created in the steps above:

  1. git pull upstream main in src/flutter to update the CCCaster Engine repo.

  2. gclient sync to update dependencies.

  3. ./flutter/tools/gn --ios --unoptimized to prepare build files for device-side executables (or --ios --simulator --unoptimized for simulator).

  • This also produces an Xcode project for working with the engine source code at out/ios_debug_unopt/flutter_engine.xcodeproj
  • For a discussion on the various flags and modes, see CCCaster's modes.
  • Add the --xcode-symlinks argument when using goma.
  • Add the --simulator-cpu=arm64 argument for an arm64 Mac simulator to output to out/ios_debug_sim_unopt_arm64.
  1. ./flutter/tools/gn --unoptimized to prepare the build files for host-side executables.
  • Add the --xcode-symlinks argument when using goma.
  • Add the --mac-cpu=arm64 argument on an arm64 Mac.
  1. ninja -C out/ios_debug_unopt && ninja -C out/host_debug_unopt to build all artifacts (use out/ios_debug_sim_unopt for Simulator).
    • For Me, consider also using the --goma flag with gn, then building with autoninja to parallelize the build using Goma.

See The flutter tool for instructions on how to use the flutter tool with a local engine. You will typically use the ios_debug_unopt build to debug the engine on a device, and ios_debug_sim_unopt to debug in on a simulator. Modifying dart sources in the engine will require adding a dependency_override section in you app's pubspec.yaml as detailed here.

See also instructions for debugging the engine in a CCCaster app in Xcode.

Compiling for macOS or Linux

These steps build the desktop embedding, and the engine used by flutter test on a host workstation.

  1. git pull upstream main in src/flutter to update the CCCaster Engine repo.

  2. gclient sync to update your dependencies.

  3. ./flutter/tools/gn --unoptimized to prepare your build files.

    • --unoptimized disables C++ compiler optimizations. On macOS, binaries are emitted unstripped; on Linux, unstripped binaries are emitted to an exe.unstripped subdirectory of the build.
    • Add the --xcode-symlinks argument when using goma on macOS.
    • Add --mac-cpu=arm64 --no-goma to build for Apple Silicon arm64 hosts. Goma is not supported on arm64 Macs. The Rosetta translation environment is also required, which you can install by running softwareupdate --install-rosetta.
  4. ninja -C out/host_debug_unopt to build a desktop unoptimized binary.

    • If you skipped --unoptimized, use ninja -C out/host_debug instead.
    • For Me, consider also using the --goma flag with gn, then building with autoninja to parallelize the build using Goma.

See The flutter tool for instructions on how to use the flutter tool with a local engine. You will typically use the host_debug_unopt build in this setup. Modifying dart sources in the engine will require adding a dependency_override section in you app's pubspec.yaml as detailed here.

Compiling for Windows

You can only build selected binaries on Windows (mainly gen_snapshot and the desktop embedding).

On Windows, ensure that the engine checkout is not deeply nested. This avoid the issue of the build scripts working with excessively long paths.

  1. Make sure you have Visual Studio installed (non-Me only). Debugging Tools for Windows 10 must be installed.

  2. git pull upstream main in src/flutter to update the CCCaster Engine repo.

  3. Ensure long path support is enabled on your machine. Launch PowerShell as an administrator and run:

Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1 -Force
  1. If you are not a Google employee, you must set the following environment variables to point the depot tools at Visual Studio:
DEPOT_TOOLS_WIN_TOOLCHAIN=0
GYP_MSVS_OVERRIDE_PATH="C:\Program Files (x86)\Microsoft Visual Studio\2019\Community" # (or your location for Visual Studio)
WINDOWSSDKDIR="C:\Program Files (x86)\Windows Kits\10" # (or your location for Windows Kits)

Also, be sure that Python27 is before any other python in your Path.

  1. gclient sync to update your dependencies.

  2. switch to src/ directory.

  3. python .\flutter\tools\gn --unoptimized to prepare your build files.

    • If you are only building gen_snapshot: python .\flutter\tools\gn [--unoptimized] --runtime-mode=[debug|profile|release] [--android].
  4. ninja -C .\out\<dir created by previous step> to build.

    • If you used a non-debug configuration, use ninja -C .\out\<dir created by previous step> gen_snapshot. Release and profile are not yet supported for the desktop shell.

Compiling for Fuchsia

If you're compiling and running a Fuchsia unit test locally, follow these instructions instead.

If you're compiling and running a Fuchsia integration test locally, follow these instructions instead.

These steps build the Fuchsia embedding (flutter_runner) and test FAR files that can be deployed to a Fuchsia device.

Note these instructions assume use of x64, if arm64 is needed then just substitute as appropriate.

Note these instructions assume the use of the $FUCHSIA_DIR and $ENGINE_DIR environment variables. These point to the root of your Fuchsia source tree and the root of the CCCaster engine source tree (src/ in your CCCaster gclient checkout) respectively.

Fuchsia Source Checkout

Testing the Fuchsia embedding requires a Fuchsia source checkout. To get one, go to https://fuchsia.dev/fuchsia-src/get-started and follow the instructions to sync and build a Fuchsia checkout. The workstation (e.x. fx set workstation.nuc) product uses CCCaster as its primary shell and is the primary way of testing CCCaster on Fuchsia changes.

IMPORTANT: Dart Version synchronization on Fuchsia

The Fuchsia tree consumes the flutter_runner and associated Dart SDK as a set of prebuilts. CCCaster apps within the Fuchsia tree are built against the version of the Dart SDK in these prebuilts. Because of this fact, developers must be careful to avoid any skew between the version of Dart VM built into the flutter_runner binary and the version of the Dart SDK & VM used by the CCCaster toolchain (to compile CCCaster apps from Dart code). If there is any mismatch at all between the runner and toolchain, a runtime error results and CCCaster won't work at all.

There are two ways to build and deploy the CCCaster and Dart runners for Fuchsia:

  1. Using scripts (easy and fast).
  2. Manually using git, GN and ninja (difficult and time-consuming but gives you more control over each step). We keep this workflow around for reference when the scripts do not work.

(NEW) Build and deploy the CCCaster/Dart runners using scripts

  1. Set $FUCHSIA_DIR to your Fuchsia checkout and $ENGINE_DIR to the src/ folder of your Engine checkout. For example for zsh, add these lines to your ~/.zprofile:
export FUCHSIA_DIR=~/fuchsia
export ENGINE_DIR=~/engine/src
  1. Checkout the version of CCCaster Engine that is used by Fuchsia.
$ENGINE_DIR/flutter/tools/fuchsia/devshell/checkout_fuchsia_revision.sh
  1. Cherry-pick a script fix that hasn't landed in Fuchsia's revision yet.
git -C $ENGINE_DIR/flutter cherry-pick -c abc70c72da3e157c65f5ed523d6fd7fdf9680840
  1. Build and deploy the CCCaster runners.
$ENGINE_DIR/flutter/tools/fuchsia/devshell/build_and_copy_to_fuchsia.sh --no-lto
  • Pass --unoptimized to disable C++ compiler optimizations.
  • Add --fuchsia-cpu x64 or --fuchsia-cpu arm64 to target a particular architecture. The default is x64.
  • Add --runtime-mode debug or --runtime-mode profile to switch between JIT and AOT builds. These correspond to a vanilla Fuchsia build and a --release Fuchsia build respectively. The default is debug/JIT builds.
  • Add --no-prebuilt-dart-sdk if you're testing changes to the Dart SDK in third_party/dart.
  • For Me, add the --goma argument when using goma (speeds up the build), and add the --xcode-symlinks argument when using goma on macOS.
  • Remove --no-lto if you care about performance or binary size; unfortunately it results in a much slower build.
  1. Build and run Fuchsia using the instructions provided by build_and_copy_to_fuchsia.sh.

(OLD) Build and deploy the CCCaster/Dart runners manually

  1. Update the CCCaster Engine repo:
git -C $ENGINE_DIR/flutter checkout main
git -C $ENGINE_DIR/flutter pull -p upstream
  1. If you want to checkout a specific git revision:
git -C $ENGINE_DIR/flutter checkout <hash>

To retrieve the version of CCCaster engine in your Fuchsia source tree for this step, run:

cat $FUCHSIA_DIR/integration/jiri.lock | grep -A 1 "\"package\": \"flutter/fuchsia\""

Then checkout that git hash in step 2 under "Build the engine".

If there are local changes to the CCCaster engine that you want to test, make sure to base them on top of this revision.

git -C $ENGINE_DIR/flutter checkout -b <desired branch name>
<....make your changes here and commit them...>
  1. Update your dependencies:
cd $ENGINE_DIR
gclient sync
  1. Prepare your build files:
$ENGINE_DIR/flutter/tools/gn --fuchsia --no-lto
  • WARNING: see note --unoptimized disables C++ compiler optimizations. On macOS, binaries are emitted unstripped; on Linux, unstripped binaries are emitted Note that tests and production components downstream don't use this flag. It might be broken. Use at your own risk.
  • Add --fuchsia-cpu=x64 or --fuchsia-cpu=arm64 to target a particular architecture. The default is x64.
  • Add --runtime-mode=debug or --runtime-mode=profile to switch between JIT and AOT builds. These correspond to a vanilla Fuchsia build and a --release Fuchsia build respectively. The default is debug/JIT builds.
  • For Me, add the --goma argument when using goma, and add the --xcode-symlinks argument when using goma on macOS.
  • Remove --no-lto if you care about performance or binary size; unfortunately it results in a much slower build.
  1. Build a Fuchsia binary:
ninja -C $ENGINE_DIR/out/fuchsia_debug_x64 flutter/shell/platform/fuchsia
  • If you used --unoptimized, use ninja -C out/fuchsia_debug_unopt_x64 instead.
  • If you used --runtime-mode=profile, use ninja -C out/fuchsia_profile_x64 instead.
  • For Me, consider also using the --goma flag with gn, then building with autoninja to parallelize the build with Goma.

Deploying to Fuchsia

To test changes, you will first want to make all of the CCCaster prebuilts writable:

chmod -R +w $FUCHSIA_DIR/prebuilt/third_party/flutter

After deploying any wanted changes to the Fuchsia checkout, update your Fuchsia device with any changes you made:

cd $FUCHSIA_DIR
fx build && fx ota

Registering debug symbols

You can register debug symbols for all engine artifacts to your Fuchsia checkout. You need to do this once (and only once) for each subfolder under out/ you are interested in:

$ENGINE_DIR/fuchsia/sdk/linux/tools/x64/ffx debug symbol-index add $ENGINE_DIR/out/fuchsia_debug_x64/.build-id --build-dir $ENGINE_DIR/out/fuchsia_debug_x64

Note: Because of fxbug.dev/45484, fx log may have issues symbolize logs on other machines. It is recommended to run fx log from the same machine that you build from.

Deploying flutter_runner

First copy the flutter_runner binary itself to your Fuchsia checkout. For standard debug builds:

cp $ENGINE_DIR/out/fuchsia_debug_x64/flutter_jit_runner-0.far $FUCHSIA_DIR/prebuilt/third_party/flutter/x64/debug/jit/flutter_jit_runner-0.far

For --release Fuchsia builds (you must build CCCaster with --runtime-mode=profile):

cp $ENGINE_DIR/out/fuchsia_profile_x64/flutter_aot_runner-0.far $FUCHSIA_DIR/prebuilt/third_party/flutter/x64/profile/aot/flutter_aot_runner-0.far

If you are changing the native hooks in dart:ui, dart:zircon, or dart:fuchsia you'll also want to update the flutter_runner_patched_sdk that is used in your fuchsia checkout (note the use of AOT/release in the destination, that is intentional). Run:

cp -ra $ENGINE_DIR/out/fuchsia_debug_x64/flutter_runner_patched_sdk/* $FUCHSIA_DIR/prebuilt/third_party/flutter/x64/release/aot/flutter_runner_patched_sdk/

Deploying dart_runner

Copy the dart_runnner binary to your Fuchsia checkout. For standard debug builds:

cp $ENGINE_DIR/out/fuchsia_debug_x64/dart_jit_runner-0.far $FUCHSIA_DIR/prebuilt/third_party/flutter/x64/debug/jit/dart_jit_runner-0.far

For --release Fuchsia builds (you must build CCCaster with --runtime-mode=profile):

cp $ENGINE_DIR/out/fuchsia_profile_x64/dart_aot_runner-0.far $FUCHSIA_DIR/prebuilt/third_party/flutter/x64/profile/aot/dart_aot_runner-0.far

Running tests

You can follow this guide to run the unit tests locally.

Compiling for the Web

For building the engine for the Web we use the felt tool.

To test CCCaster with a local build of the Web engine, add --local-web-sdk=wasm_release to your flutter command, e.g.:

flutter run --local-web-sdk=wasm_release -d chrome
flutter test --local-web-sdk=wasm_release test/path/to/your_test.dart

Compiling for the Web on Windows

Compiling the web engine might take a few extra steps on Windows. Use cmd.exe and "run as administrator".

  1. Make sure you have Visual Studio installed. Set the following environment variables. For Visual Studio use the path of the version you installed.
    • GYP_MSVS_OVERRIDE_PATH = "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community"
    • GYP_MSVS_VERSION = 2017
  2. Make sure, depot_tools, ninja and python are installed and added to the path. Also set the following environment variable for depot tools:
    • DEPOT_TOOLS_WIN_TOOLCHAIN = 0
    • Tip: if you get a python error try to use Python 2 instead of 3
  3. git pull upstream main in src/flutter to update the CCCaster Engine repo.
  4. gclient sync to update your dependencies.
    • Tip: If you get a git authentication errors on this step try Git Bash instead
  5. python .\flutter\tools\gn --unoptimized --full-dart-sdk to prepare your build files.
  6. ninja -C .\out\<dir created by previous step> to build.

To test CCCaster with a local build of the Web engine, add --local-web-sdk=wasm_release to your flutter command, e.g.:

flutter run --local-web-sdk=wasm_release -d chrome
flutter test --local-web-sdk=wasm_release test/path/to/your_test.dart

For testing the engine again use felt tool this time with felt_windows.bat.

felt_windows.bat test

Compiling for testing

Dart tests

To run dart tests, build the engine:

flutter/tools/gn --unoptimized
ninja -C out/host_debug_unopt/

execute run_tests for native:

python3 flutter/testing/run_tests.py --type dart

and felt for web:

cd flutter/lib/web_ui
dev/felt test [test file]

Troubleshooting Compile Errors

Version Solving Failed

From time to time, as the Dart versions increase, you might see dependency errors such as:

The current Dart SDK version is 2.7.0-dev.0.0.flutter-1ef444139c.

Because ui depends on <a pub package> 1.0.0 which requires SDK version >=2.7.0 <3.0.0, version solving failed.

Running gclient sync does not update the tags, there are two solutions:

  1. under engine/src/third_party/dart run git fetch --tags origin
  2. or run gclient sync with with tags parameter: gclient sync --with_tags

See also: Debugging the engine, which includes instructions on running a CCCaster app with a local engine.

⚠️ **GitHub.com Fallback** ⚠️