How metrics are gathered - alandiaz08/e2e-web-framework GitHub Wiki

Stage 1: JUNIT

  • JUnit is a framework which was initially used by many Java applications as a Unit test framework.
  • By default, JUnit tests generate a simple report in XML format for every test execution.
  • These XML files can be used to generate custom reports as per the requirement.
  • On analyzing the XML files, useful information about the tests will be added to our database.

Example of an xml file: XML file

For each class (e.g. customerTests, searchTests, bookingTests, etc.) an xml file is generated with a structure similar to the one shown in the image above.

  • Each test is represented as a testcase “child” node whose attributes describe characteristics of the executed test, such as the execution time, name of the test and class to which it belongs.
  • When the test is skipped or marked as failed, a new sub-node is added in which there is information related to the error captured by the test.

Stage 2: TEST

2.1 PIPELINE & STAGES:

PIPELINE

A Pipeline is a user-defined model of a CD pipeline.A Pipeline’s code defines your entire build process, which typically includes stage blocks comprising of conceptually distinct subset of tasks performed through the entire Pipeline (e.g. "Build", "Test" and "Deploy" stages) which is used by many plugins to visualize or present Jenkins Pipeline status/progress.

In a Jenkins pipeline, the stages are generally executed sequentially (unless parallel executions are defined). Normally if a stage fails, the build fails and subsequent stages are skipped so if we want to continue with the subsequent steps, we need to make some changes on the Jenkins script.

2.2 catchError and statusCode:

The tests are executed through a shell command that will return a status code according to the results of the tests (0 for when there are no failures). Normally, a script which exits with a non-zero status code will cause the step to fail with an exception. If this option is checked, the return value of the step will instead be the status code.

When the tests fail (and the status returned by the shell command is different from 0) we need the subsequent stages to be executed and for that it is necessary to "catch" the error by surrounding the action executed with a catchError step. The status code is used at the end of the script to mark the build as failed (comparing it with an expected value)

For more information: https://www.jenkins.io/doc/pipeline/steps/workflow-basic-steps/ and https://www.jenkins.io/doc/pipeline/steps/workflow-durable-task-step/

Stage 3: METRICS - Groovy script

The groovy script contains all the logic for parsing and analysis of the xml files and their subsequent insertion into the DB tables. The script is defined by the following functions:

  • connectToPostgresql(String dbUser, String dbpassword)-> returns a SQL instance
  • insertJenkinsBuildInfoToJenkinsJobTable(Sql sql) -> returns key: jenkinsJob
  • parseJunitXmlFilesAndInsertIntoTestExecutionTable(String path, int jenkinsJob, Sql sql)
  • postgresqlCloseConnection(Sql sql) -> close the postgresql connection

METRICS

  1. The first step in the Metrics stage is to create the DB connection (connectToPostgresql) to be able to share it with the other functions and thus be able to execute the necessary queries.

  2. Then the job execution info is obtained from the Jenkins Environment Variable and processed by insertJenkinsBuildInfoToJenkinsJobTable function. The JEV is a global variable exposed through the env variable and is used anywhere in the Jenkins file. Any value stored in the env variable is stored as a type of string so it is necessary to process them to comply with the formats of the table columns.

  3. Then parseJunitXmlFilesAndInsertIntoTestExecutionTable takes care of processing and analyzing the information contained on each xml file. This function receives as arguments the SQL instance and, the xml file path and an integer that is used as the foreign key. Jenkins_job.id = PRIMARY KEY -> test_execution.jenkins_job = FOREIGN KEY

  4. The last step in the Metrics stage is to close the DB connection (postgresqlCloseConnection)

Stage 4: DATABASE:

SCHEMA DIAGRAM

jenkins_job table: Jenkins environment variables are used to obtain the values all the values like the job parameters (gradle_arguments, browser, environment, included_groups, excluded_groups) and additional information such as when it was executed and the duration of the build.

test_execution table: After parsing the xml files the different values are obtained according to the test result