Behavior Driven Test Automation - PeterWangIntel/crosswalk-website GitHub Wiki

Overview

This solution implements Behavior-driven Development (abbreviated BDD) principles to allow users to use tests written in a ubiquitous language style to validate the web/native applications and web/native platform and offers a convenient and efficient way to develop, manage, execute the test cases, and easy to combine with high level test framework

Background

Application Test Automation Challenges

  • Hard to automate System level operations such as opening the Settings menu or the app launcher in a test device
  • Hard to implement cross-app functional UI automation testing across system and installed apps
  • Hard to apply image recognition or platform native UI automation solution to web application
  • Multi-formity of web/native UI layout and elements requires a more convenient test design and development method
  • Existing browser test automation techniques and tools have gap on support on web applications or web runtime

WebDriver

  • WebDriver is a W3C standard: The WebDriver API is defined by a wire protocol and a set of interfaces to discover and manipulate DOM elements on a page, and to control the behavior of the containing browser from a separate controlling process
  • Most popular instance of WebDriver specification is selenium2.0
  • Enable WebDriver on Crosswalk

UI Automator

  • The UI Automator testing framework provides a set of APIs to build UI tests that perform interactions on user apps and system apps. The UI Automator APIs allows you to perform operations such as opening the Settings menu or the app launcher in a test device.
  • uiautomator is the most popular Python wrapper of Android UI Automator testing framework. It works on Android 4.1+ simply with Android device attached via adb, no need to install anything on Android device.

BDD

  • An agile software development technique that encourages collaboration between developers, QA and non-technical or business participants in a software project.
  • Our solution implements BDD principles to allow users to use tests written in a ubiquitous language style to validate the web applications and web platform and offers a convenient and efficient way to develop, manage, execute the test cases, and easy to combine with high level test framework

Solution

BDD-arch

"behave"

"behave is behaviour-driven development, Python style", and the page of "comparisons with other tools" shows why we select "behave" as the default BDD tool.

ATIP - a new "behave" binding library

ATIP(Application Test in Python), a “behave” binding library as the bridge between application and BDD too “behave”, and use WebDriver and platform interfaces to implement detailed BDD steps for application. ATIP resources host on Crosswalk Test Suite Project

Features Status

Features Done

  • Basic WebDriver based steps
  • Basic Android UI Automator based steps
  • Environment initialization from config file
  • Environment initialization from test tools/system environment variables
  • Support Crosswalk on Tizen and Android
  • Crosswalk application launching

Features in Plan

  • Add more WebDriver based UI automation steps
  • Add more platform native application UI automation support
  • Provide APIs level interfaces from ATIP so that generic script can import ATIP too
  • More platforms support

Tests Development

Before below sections, you'd better already pretty familiar with the tests developing with "behave": "behave" docs. A typical "behave" based tests source layout as below:

 tests/
  ├── environment.py
  ├── steps
  │   └── steps.py
  └── test.feature

"test.feature"

Feature file, one tests folder can contains more than one feature file which has a natural language format describing a feature or part of a feature with representative examples of expected outcomes, e.g.

 Feature: api tests
      Scenario: api test 001
           When launch "haha"
            And I go to "http://www.google.com"
            And I wait for 1 seconds
           Then I should see "Google"

"steps.py"

Python steps implementations file. In ATIP usage, this file will import the steps from ATIP. Of course, you can write your own steps to this "steps.py" file too. The build-in ATIP steps as following:

"web" steps(Webdriver API based steps)

  • @step(u'I launch "{app_name}" with "{apk_pkg_name}" and "{apk_activity_name}"')
  • @step(u'switch to "{app_name}"')
  • @step(u'I go to "{url}"')
  • @step(u'I reload')
  • @step(u'I go back')
  • @step(u'I go forward')
  • @step(u'The current URL should be "{text}"')
  • @step(u'I should see title "{text}"')
  • @step(u'I should see "{text}"')
  • @step(u'I should not see "{text}"')
  • @step(u'I should see "{text}" in {timeout:d} seconds')
  • @step(u'I should not see "{text}" in {timeout:d} seconds')
  • @step(u'I should see "{text}" in "{key}" area')
  • @step(u'I press "{key}"')
  • @step(u'press "{key_c}" in "{key_p}"')
  • @step(u'I click "{key}"')
  • @step(u'click "{key_c}" in "{key_p}"')
  • @step(u'I click coords {x:d} and {y:d} of "{key}"')
  • @step(u'I fill in "{key}" with "{text}"')
  • @step(u'I check "{key}"')
  • @step(u'I uncheck "{key}"')
  • @step(u'I should see an alert')
  • @step(u'I should not see an alert')
  • @step(u'I accept the alert')
  • @step(u'I should see an alert with text "{text}"')

"android" steps(Android platform specific steps which is implemented by uiautomator)

  • @step(u'I launch "{app_name}" with "{apk_pkg_name}" and "{apk_activity_name}" on android')
  • @step(u'I scroll to end')
  • @step(u'I fling "{orientation}" goto "{direction}"')
  • @step(u'I swipe view "{params_kw}" to "{orientation}"')
  • @step(u'I swipe saved object "{key}" to "{orientation}"')
  • @step(u'I force to run all watchers')
  • @step(u'I remove all watchers')
  • @step(u'I register watcher "{watcher_name}" when "{when_text}" click "{click_text}"')
  • @step(u'I register watcher2 "{watcher_name}" when "{when_text1}" and "{when_text2}" click "{click_text}"')
  • @step(u'I should see view "{params_kw}"')
  • @step(u'I should see relative view "{params_kw1}" on the "{position}" side of view "{params_kw2}"')
  • @step(u'I should not see view "{params_kw}"')
  • @step(u'I should not see relative view "{params_kw1}" on the "{position}" side of view "{params_kw2}"')
  • @step(u'I click view "{params_kw}"')
  • @step(u'I click saved object "{key}"')
  • @step(u'I edit view "{params_kw}" to input "{text}"')
  • @step(u'I edit index {n:d} EditText to input "{text}"')
  • @step(u'I compare view "{params_kw}" info "{info_name}" with "{except_result}"')
  • @step(u'I compare saved object "{key1}" equal "{key2}" on info "{info_name}"')
  • @step(u'I compare saved object "{key1}" unequal "{key2}" on info "{info_name}"')
  • @step(u'I save view "{params_kw}" to object by "{key}"')
  • @step(u'I save relative view "{params_kw1}" on the "{position}" side of view "{params_kw2}" to object "{key}"')
  • @step(u'I save process of finding view "{params_kw1}" on the "{position}" side of view "{params_kw2}"')
  • @step(u'I reload above process and save result to object "{key}"')
  • @step(u'I wait saved object "{key}" exist in "{time_out}" seconds')
  • @step(u'I wait saved object "{key}" gone in "{time_out}" seconds')

"common" steps(common operations cross different platforms)

  • @step(u'I wait for {timeout:d} seconds')
  • @step(u'call "{js}" scripts') - TBD
  • @step(u'call PIL to handle "{image_file}"') - TBD
  • @step(u'launch "{app_name}"')
  • @step(u'I turn on device')
  • @step(u'I turn off device')
  • @step(u'I set orientation "{orientation}"')
  • @step(u'I take screenshot as "{name}"')
  • @step(u'I open notification')
  • @step(u'I open quick settings')
  • @step(u'I press "{key}" hardware key')
  • @step(u'I open wifi')
  • @step(u'I close wifi')
  • @step(u'I open airplane mode')
  • @step(u'I close airplane mode')
  • @step(u'I execute command "{command_line}"')

"environment.py"

Environmental controls. In ATIP usage, you can write the environment initial processes on this file. ATIP provides two templates of "environment.py" for tests developers. One is for WebDriver backend, the other is for uiautomator backend. These templates support running tests independently by "behave" tool.

In WebDriver testing, the template need to know the some test variables, e.g. which device be tested? Which test platform? Some WebDriver variables. These variables can be got by following ways:

  • By environment variables:
  1. TEST_PLATFORM
  2. DEVICE_ID
  3. CONNECT_TYPE
  4. WEBDRIVER_VARS

In uiautomator testing, only two environment variables are required, "TEST_PLATFORM" and "DEVICE_ID". This allows running uiautomator tests independently without WebDriver.

ATIP source provides a script "tools/set_env.sh" which can help you to setup those environment variables. Especially, the environment.py template can get those environment variables from Testkit-lite tool automatically:

  • A JSON config file which named "bdd.json for environment variables sharing, a template provided for reference: "atip/tools/bdd.*.json"

##Example BDD-example

Run Tests

BDD-following

  • Testkit-lite: Please check testkit-lite project for details
  • Behave:

  1. Setup test ENVs by set_env.sh or bdd.json

  2. Run "behave" as:

    $cd path-to/tests

    $behave