Development Continuous Integration tests - TESTARtool/TESTAR_dev GitHub Wiki

TESTAR Continuous Integration Workflow

The TESTAR project uses the GitHub Actions Workflow feature together with a serie of Gradle tasks to test that the tool starts, connects and correctly detects desktop and web applications widgets.

The workflow file .github/workflows/gradle.yml contains three jobs to build TESTAR inside a Windows, Linux and MacOS environments using Java 1.8.
Windows job also contains multiple steps that run gradle tasks specifically defined to test different TESTAR functionalities.

Gradle tasks

testar/build.gradle file contains all these different test tasks grouped in group = 'test_testar_workflow'.

These tasks run TESTAR using Windows command prompt, read the messages from the TESTAR output buffer, and finally determines if the tasks have completed successfully based on the output buffer information.

Example of a job step:

name: Run desktop_generic protocol with TESTAR using COMMAND_LINE
run: ./gradlew runTestDesktopGenericCommandLineOk

that executes a gradle task:

// Default COMMAND_LINE execution to run Notepad
// Test TagsForSuspiciousOracle feature using the UIATags.UIAAutomationId
// Force TESTAR to find a Suspicious Title (MenuBar widget)
// And verify it reading output command line
task runTestDesktopGenericCommandLineSuspiciousTitle(type: Exec, dependsOn:'installDist') {
    // Save TESTAR output buffer information
    standardOutput = new ByteArrayOutputStream()
    // Assign this task to the workflow group
    group = 'test_testar_workflow'
    description ='runTestDesktopGenericCommandLineSuspiciousTitle'
    // Indicate the TESTAR distributed folder directory
    workingDir 'target/install/testar/bin'
    // Prepare a TESTAR command line execution, with the desired protocol and the required settings
    commandLine 'cmd', '/c', 'testar sse=test_gradle_workflow_desktop_generic AlwaysCompile=true ApplicationName="notepad_command_and_suspicious" TagsForSuspiciousOracle="NotExist;UIAAutomationId" SuspiciousTitles=".*MenuBar.*" ShowVisualSettingsDialogOnStartup=false Mode=Generate SUTConnector=COMMAND_LINE SUTConnectorValue="C:\\\\Windows\\\\System32\\\\notepad.exe" Sequences=1 SequenceLength=2'

    // After TESTAR execution
    doLast {
        // Read TESTAR output buffer information to
        String output = standardOutput.toString()
        // Verify MenuBar UIAAutomationId has been detected
        if(output.readLines().any{line->line.contains("Discovered suspicious widget 'UIAAutomationId' : 'MenuBar")}) {
            println "\n${output} \nTESTAR has successfully detected MenuBar Suspicious Title using TagsForSuspiciousOracle UIAAutomationId! "
        } else {
            throw new GradleException("\n${output} \nERROR: TESTAR didnt detect MenuBar Suspicious Title using TagsForSuspiciousOracle UIAAutomationId")
        }
    }
}

• Test different TESTAR features will require multiple tasks with different protocols and settings.
• Some TESTAR features have additional software dependecies, such as Selenium WebDriver or OrientDB. The use of these software should be defined as additional and dependent tasks.

TESTAR protocols and settings

TESTAR tool makes use of a Java and test.settings protocol files to define the behaviour of the interaction with the System Under Test (SUT), something that we will need to configure if we want to test specific TESTAR features.

To allow the customization of these protocols, the folder testar/resources/workflow/settings has been created. These protocols are not copied with the default TESTAR tool compilation, it is necessary to initialize the copy of these protocols with the existing gradle task init_workflow_test:

1. Prepare TESTAR protocols: gradle init_workflow_test
2. Build TESTAR: gradle build
3. Prepare installed distribution of TESTAR: gradle installDist
4. Run desktop_generic protocol with TESTAR using COMMAND_LINE: gradle runTestDesktopGenericCommandLineOk

- name: Prepare TESTAR protocols
  run: ./gradlew init_workflow_test
- name: Build TESTAR with Gradle
  run: ./gradlew build
- name: Prepare installed distribution of TESTAR with Gradle
  run: ./gradlew installDist
- name: Run desktop_generic protocol with TESTAR using COMMAND_LINE
  run: ./gradlew runTestDesktopGenericCommandLineOk

TESTAR software dependencies

All jobs are executed inside GitHub virtual environments on which it is necessary to download and prepare some software such as Selenium WebDriver executable and an OrientDB testar database.

• For Selenium WebDriver it is important to maintain updated the Chrome browser and web driver executable versions updated.
  GitHub staff updates the version of the browser and the executable weekly.
  The location of the different web drivers is indicated here: https://github.com/actions/virtual-environments/blob/main/images/win/Windows2019-Readme.md

To execute TESTAR with ChromeDriver we need to create a webdriver_protocol that uses this location in the settings file:

SUTConnector = WEB_DRIVER  
SUTConnectorValue = "C:\\SeleniumWebDrivers\\ChromeDriver\\chromedriver.exe" "1920x900+0+0" "https://www.ou.nl"

• To be able to test OrientDB, we have created additional Gradle tasks that download and create an OrientDB database.
  State Model OrientDB tasks download version 3.0.34 and creating a new testar database with the use of the dependency dependsOn: createDatabaseOrientDB

// Download OrientDB 3.0.34 from the official wep page
task downloadOrientDB(type: Download, dependsOn: installDist) {
    // This orientdb3 URL contains the specific version we need
    src 'https://s3.us-east-2.amazonaws.com/orientdb3/releases/3.0.34/orientdb-3.0.34.zip'
    dest 'target/install/testar/bin'
    overwrite false
}

// Verify that we downloaded the correct OrientDB file
task verifyOrientDB(type: Verify, dependsOn: downloadOrientDB) {
    // Because we always use the same orientdb3 URL we can verify this is the desired file
    src new File(downloadOrientDB.dest, 'orientdb-3.0.34.zip')
    algorithm 'MD5'
    checksum 'a0619e5522e7d849d4f1067bf08e3857'
}

// Extract OrientDB binary files
task downloadAndUnzipOrientDB(type: Copy, dependsOn: verifyOrientDB) {
    group = 'test_testar_workflow'
    description ='downloadAndUnzipOrientDB'

    // Extract all OrientDB files from the zip
    from zipTree(new File(downloadOrientDB.dest, 'orientdb-3.0.34.zip'))
    into downloadOrientDB.dest
}

// Execute a command line to create a testar OrientDB database with admin admin credentials
task createDatabaseOrientDB(type: Exec, dependsOn: downloadAndUnzipOrientDB){
    group = 'test_testar_workflow'
    description ='createDatabaseOrientDB'

    // Save output and errpr buffer information
    standardOutput = new ByteArrayOutputStream()
    errorOutput = new ByteArrayOutputStream()
    workingDir 'target/install/testar/bin/orientdb-3.0.34/bin'
    commandLine 'cmd', '/c', 'console.bat CREATE DATABASE plocal:../databases/testar admin admin'
    // Ignore errors creating the database because we check the output buffer message
    ignoreExitValue true
    doLast {
        String output = standardOutput.toString()
        
        // If error creating database because already exists is ok
        if (execResult.getExitValue()==1 && errorOutput.toString().contains("Cannot create new database 'testar' because it already exists")) {
            println "\n${output} \ntestar OrientDB database already exists"
        }
        // Check if testar database created sucessfully
        else if(output.readLines().any{line->line.contains("Database created successfully.")}) {
            println "\n${output} \ntestar OrientDB database created successfully"
        } else {
            throw new GradleException("\n${output} \nERROR: Creating testar OrientDB database")
        }
    }
}

TESTAR artifact result

GitHub Actions Workflow feature also allow us to define and upload a file or folder as a result of a job step. Basically we want to indicate that the artifact of the test step is the TESTAR output folder, that contains the HTML, screenshots, logs, etc...

name: Save runTestDesktopGenericCommandLineSuspiciousTitle HTML report artifact
uses: actions/upload-artifact@v2
with:
  name: runTestDesktopGenericCommandLineSuspiciousTitle-artifact
  path: D:/a/TESTAR_dev/TESTAR_dev/testar/target/install/testar/bin/notepad_command_and_suspicious

Because it is necessary to indicate the naming of the specific path to be uploaded as an artifact and because the name of the TESTAR output folder is based on the dynamic timestamp, we are doing a copy of the output folder name based on the ApplicationName setting value.

// Example: testar/resources/workflow/settings/test_gradle_workflow_desktop_generic/Protocol_test_gradle_workflow_desktop_generic.java
/**
* This method is called after the last sequence, to allow for example handling the reporting of the session
*/
@Override
protected void closeTestSession() {
   super.closeTestSession();
   try {
      // Copy original TESTAR folder result testar/bin/output/timestamp
      File originalFolder = new File(OutputStructure.outerLoopOutputDir).getCanonicalFile();
      // To the desired Artifact testar/bin/ApplicationName
      File artifactFolder = new File(Main.testarDir + settings.get(ConfigTags.ApplicationName,""));
      FileUtils.copyDirectory(originalFolder, artifactFolder);
   } catch(Exception e) {System.out.println("ERROR: Creating Artifact Folder");}
}

This means that is better to use different ApplicationName setting values in the gradle test tasks to have different TESTAR artifacts for the test tasks.

• gradle task runTestDesktopGenericCommandLineSuspiciousTitle

uses the ApplicationName="notepad_command_and_suspicious

• Windows job step name: Save runTestDesktopGenericCommandLineSuspiciousTitle HTML report artifact

saves the artifact from path: D:/a/TESTAR_dev/TESTAR_dev/testar/target/install/testar/bin/notepad_command_and_suspicious

How to reproduce the Gradle CI tasks on your local computer

To be able to run and replicate the Gradle tasks in your computer (e.g., to verify or fix some functionality), you need to:

1. Mimic the ChromeDriver folder that is being used in the GitHub environment with a chromedriver executable version that works with your Chrome Browser version (C:\\SeleniumWebDrivers\\ChromeDriver\\chromedriver.exe)
2. Prepare a clean clone of the project: gradlew clean
3. Prepare the TESTAR protocols: gradlew init_workflow_test
4. Build the TESTAR project and execute JUnit tests: gradlew build
5. Prepare an installed distribution of TESTAR: gradlew installDist
6. Run the desired Gradle task. For example: gradlew runTestDesktopGenericCommandLineOk

TESTAR will be launched automatically, and the Gradle test results will be created in the output folder.
NOTE: Use gradlew Gradle Wrapper commands if you are running the project outside an IDE. Or select and run gradle tasks if you are running the project from an IDE.
WARNING: When running these Gradle tests, use a screen resolution of 100% :)