Setting up CI CD for Android test infrastructure - dtoinagn/flyingbird.github.io GitHub Wiki

This page capture the learnings while I research how to set up CI/CD for Android test infrastructure

The first step is to implement the autotests. One option is to use Allure TestOps using Kaspresso.

  1. Test Management System: Test results sharing and test runs and controls are very important features, e.g. selective reruns of tests, specifically initiating a Jenkins launch with the specific tests.
  2. Ease integration with CI/CD
  3. Kaspresso is similar as Espresso? Espresso is a Java automated UI testing framework that is installed with the app's source code. It is "grey box" testing framework. With Espresso, you can write user interface tests (black box) but it has the same access to the internals of the application that any other unit/integration tests (white box) have, hence why it is called "grey-box".

Espresso Framework

Benefits of Espresso

  • Better performance
  • Supports mocking which allows for better ability to cover a wider array of use-cases and allows you to mock out slow operations (REST API calls, database calls, etc...)
  • Some tests can be run in a simulated environment so instead of having to run tests on an Emulator or Real Android Device, tests can be run on a JVM

Limitations of Espresso

  • Having access to the internals has many advantages, but the risk is that because you have the ability to alter the app, you're changing the app in a way that doesn't accurately represent how users will actually use an app
  • Requires the source code to write tests. Which is fine if Android developers are writing tests. Might be a problem for QA engineers.
  • Requires knowledge of Android development to write tests
  • Can only write tests in Android supported languages (Java, Kotlin)

Kaspresso

Kaspresso is a flexible and convenient framework for UI automation. The framework provides a wrapper for Espresso called UI Automator. We choose this framework because of the following advantages:

  • Good readability of the test code
  • Solution for flaky tests
  • Advanced logging
  • Working with AdbServer.
  • Ability to record video and take screenshots Kaspresso also has a variety of additional features, such as disabling Wi-Fi and data networks to test the application offline, emulating calls and SMS, and much more. The full list is here.

CI/CD implementation

GitLab CI is a pipeline tool to run test builds. It need to be integrated with the TestOps tool, which stores test data and results.

Glossary

  • ADB - Android Debug Bridge (adb) is a versatile command-line tool that lets you communicate with a device. The adb command facilitates a variety of device actions, such as installing and debugging apps. Adb provides access to a Unix shell that you can use to run a variety of commands on a device. It is a client-server program that includes three components: A client, which sends commands. The client runs on your development machine. A daemon (adbd), which runs command on a device. The daemon runs as a background process on each device. A server, which manages communication between the client and the daemon. The server runs as a background process on your development machine.

  • Reference - We use the image jangrewe/gitlab-ci-android because it already contains the Android SDK and other libraries for building Android apps.

  • Reference - Setting up CI/CD for Android test infrastructure

Technical Foundation - Quality Engineering - Building the technical skills from your background knowledge

What do I need to learn to be able to be self-sufficient at Espresso for Android UI testing?

  • Kotlin-essential-training on Linkedin
  • https://kotlinlang.org/docs/reference/basic-syntax.html
  • Espresso basics:
    • Components:
    • How to build a test example in Android: https//developer.android.com/training/testing/ui-testing/espresso-testing#build
  • Espresso Robot pattern in Kotlin
  • Screenshot testing - see below

Screenshot Testing - How does it work

What is Screenshot Testing: Screenshot testing is the process of validating the visual aspects of an application's user interface (UI). It focuses on validating the Layout correctness and Appearance correctness. Layout correctness is about having each visual element of the UI properly positioned and it does not overlap or hide other visual elements on the screen. And Appearance correctness means that the visual elements are of the correct font, color or image.

Screenshot testing framework works mainly on 2 modes: Recording mode, where a bitmap of screens are captured at various points of a test and served as the baseline screens; and Assertion mode, which is used to compare the current screens against the baseline screens; and during assertion the screen's pixels are compared to the baseline bitmap. Currently we have been heavily relying on manual testing to validate visual aspects of the mobile apps. With more and more partner themes around we need a reliable screenshot testing framework across both iOS and Android platform so that we can run screenshot regression tests in the pipeline automatically every time when introducing a new UI feature or partner theme to save significant amount of manual testing efforts.

Screenshot Testing Workflow IMG_0693

  1. Add screenshot tests to feature branch (regular Espresso test with an extra method call to take the screenshot). Some tips to write Screenshot tests:
  • Screenshot tests should be the last thing to go into the feature branch
  • No need to add any functional validation steps, which are covered in UI tests
  • Add a check/assert method before taking a screenshot without a miss, to make sure app launched on the right screen
  • Once we write a method to capture final screenshot in that test we do not need any further steps
  • Provide right names for screenshots which follows the camel casing format
  1. Run the screenshot generator Jenkins job to create a new base line, automatically generate PR to the screenshot repository.

A Jenkins job with parameter can be used. The parameters can be 1) targetBranch 2) running mode being one of "GenerateAll", "GenerateDiff", "Assertion"

  1. Review and merge automatic pull requests generated to screenshot repository
  2. Update latest baselined version in artifactory (iOS/Android App repository).

Update the latest base line version in both iOS and Android Jenkins projects and push the changes to feature branch so that every time a feature branch gets merged into "develop" screenshot tests would run against latest baseline version in assertion mode.

  1. Merge feature branch in develop, and screenshot tests run in pipeline