Bitrise General Info - my-swift-lab/firefox-ios GitHub Wiki

Bitrise is a CI and CD for mobile apps. It is easy to set up and comes with nice features for testing purposes.

So, we are giving it a try setting up the testing environment to see the new features that could be useful for our environment. It allows running tests on both simulator and real devices. The real device testing can be done using Firebase test lab or your own device. For simulators and Firebase test lab devices it is possible to configure the tests to run in parallel.

In general everything is well documented in Bitrise doc site. As a high level summary, the way Bitrise is configured is by defining Workflows and Steps into each Workflow to execute the desired tasks. One app may have more than one Workflow, there is the possibility to define triggers to launch each Workflow. Also the Steps for each Workflow can be launched or not depending on certain conditions defined. There are some Steps already defined, easy to use just by adding them from the Workflow editor. But, you can always add a step as a script, and so you can configure it as you want. Predefined steps can be modified too.

As you start creating a Workflow and adding Steps to it, a bitrise.yml is created and it is updated with the Steps you add or create. It’s there where you can modify the steps, add conditions and all the customization you may need. It may look like this:

---
format_version: '6'
default_step_lib_source: https://github.com/bitrise-io/bitrise-steplib.git
project_type: ios
trigger_map:
workflows:
  primary:
    steps:
    - [email protected]:
        run_if: 'condition'
    - [email protected]: {}
    - [email protected]: {}
    - [email protected]:
        title: NPM install and build
…
---

In particular for us, we had to change a few settings in our project in this PR. Once we got the build successfully working on Bitrise, we created a workflow and here below you can see the Steps configured for our tests purposes:

[email protected]: {} - Clone the repo

[email protected] - Run npm and install the build

[email protected] - Run carthage and install the dependencies

[email protected] - Install all the certificates

git::https://github.com/bitrise-steplib/steps-xcode-build-for-test.git@export-fix - Generate the test bundle

[email protected] - Run tests in virtual devices

[email protected] - Run tests in real devices

[email protected] - Run tests in parallel in real devices

• In this case 4 scripts were needed since the way Bitrise allows this feature is by using a different tool called Flank These are the Steps needed:
  • [email protected] - Save Google Service Account JSON: Configure the Firebase account/Project ID
  • [email protected] - Generate Flank config: Define the device, iOS version, shards, tests cases to run
  • [email protected] - Run Flank: Run the tests
  • [email protected] - Copy Results: Copy the Firebase results into Bitrise dashboards.

Note that not all of them are active at the same time, depending on what(schema) and where(device or simulator) you want to run the tests you need to add conditions.

Tests results can be seen under Artifacts or Devices tab depending on where the tests run. If regular tests run on simulator on a real device, they run as part of Bitrise and tests are shown under Devices.

If tests run in parallel on simulator or real devices the results are shown as an html file under App&Artifacts tab.

As said before, in order to be able to run tests in real devices in parallel, Bitrise uses a tool called Flank. So Flank is configured in Bitrise, adding its yml file into the bitrise yml, and Flank runs the tests on Firebase Test Lab. So these tests results are also shown as a html file under Artifacts tab. Also note that the Device results are in beta and so the results of passed/failed or test run does not match yet

It was not a straight-forward set up, there were some issues to fix, in order to have an idea about what you can face and help in future set ups, here are some of those issues:

• When building the app some paths needed to be modified. For example:

  Original: $(SRCROOT)/Extensions/Entitlements/FennecEnterprise.entitlements

  New one: $(inherited)Extensions/Entitlements/FennecEnterprise.entitlements

• In order to run on devices all test bundles need to have the Hosted App correctly set with the target to test and the check Allow Testing Host Application APIs has to be selected

• All test target have to be using their corresponding and updated Provisioning Profiles

• In order to use Firebase Test Lab we had to create the project and set the proper obtained keys in Bitrise. Documentation followed for that here (https://gist.github.com/viktorbenei/555a23293ee0ff9478e583a34d2a45a4) The secrets vars are stored in the secrets tab

Once the set up finished, builds are available in the dashboard. Some examples:

• Build running only the Unit tests
• Build running the Smoketest on simulator
• Build running the Smoketest on real device

Current Configuration

Currently Bitrise is configured to run the Smoketest suite on simulator for a new commit on master or new PR, commits on PRs. This configuration is using PuhsInMasterSmokeTest Workflow.

The first intention was to run tests also for every PR but there is a limitation about the availability of Certificates and Provisioning Profiles for contributors PRs in open source projects. There are workarounds to have this working but since it would require some extra work having the checks on master and for people in the team is enough at the moment.

There are more workflows defined:

• TestingWorkflow: used to experiment how to add new steps and so
• TestingWorkflowAllTestsScenarios: Where all the steps needed for running the different tests in the different scenarios are defined so that it is easy to go there and pick the needed ones.
• Primary: is the workflow created by default which currently has same steps as the PushInMaster used but with less configurations in terms on environmental variables.
• L10nBuild: build that manages the L10nScreenshotsTest workflows
• L10nScreenshotsTest: this workflow runs the screenshots test to get the screenshots for each locale. The builds that run this workflows are triggered and managed by Taskcluster. Thanks to it the work to run the screenshots is split into different builds. Each build runs the tests for 6 locales and report them back to Taskcluster.

Tests executions can be seen going to the general dashboard or visiting the slack channel #firefox-ios-alerts where everytime a build finishes there is a notification with the status.