Working with TestListeners - Quintity/TestFramework GitHub Wiki

What is a TestListener?

A TestListener provides the mean of extending or integrating additional functionality during the course of a test run. For instance, that extended functionality may include writing test results to third hosted applications (e.g., TestRail, TargetProcess, etc.), writing results to a database or file system or sending test run notification emails. Multiple TestListeners can be attached or registered for concurrent execution. The TestListener capabilities are well suited for a CI/CD architecture where test results can be captured and recorded during unattended execution.

How does work?

The TestListener architecture is based on the Observer design pattern where the Quintity runtime (the provider) notifies the observers (the TestListener mechanism) of test events during runtime execution. Once notified, each custom listener (a .NET library assembly) processes the provided information accordingly.

Implementing a Testlistener

Implementing a TestListener requires three steps: Deploying the TestListener's service

  1. Installing the TestListenersService.
  2. Creating the TestListener (*.dll)
  3. Registering the TestListener

Installing the TestListenersService

The TestListenersService (Quintity.TestFramework.TestListenersService) is an actual NetTcp bound WCF service installed by default as Windows service. The WCF service can also be run within a console application (Quintity.TestFramework.TestListenersService.Host.exe) - be sure to stop the Windows service instance prior to launching. Once a TestListener has been created, it will be registered with this service in order to be called during runtime execution.

Creating a TestListener

  1. Add a .NET Framework class library project to the Visual Studio solution.
  2. Add the Quintity.TestFramework.Core and Quintity.TestFramework.Runtime Nuget packages to the project.
  3. Add a new Class object to the project.
  4. Add using statement references to the two Quintity assemblies.
  5. Make the class public and derive the class from runtime library "TestListener" abstract base class.
  6. Using Intellisense, implement the inherited abstract class members.
  7. Build out TestListener, adding code to the appropriate event handlers to support the goal of the listener. Some guidelines:
    1. Not all abstract methods may be used to support the listener implementation. It is recommended that the "NoOperationAttribute" be added to the unused methods (particularly if the default methods still throw the "NotImplementedException"). The TestListenerService will bypass calls to methods marked accordingly.
    2. A TestListener supports two constructors, the default constructor and a constructor with a single dictionary argument: public SampleTestListener(Dictionary<string, string> args)

Example TestListener class:

using System.Threading;
using System.Diagnostics;
using System.Reflection;
using Quintity.TestFramework.Core;
using Quintity.TestFramework.Runtime;
using System.Collections.Generic;
using System;

namespace Quintity.TestFramework.SampleListeners
{
    public class SampleTestListener : TestListener
    {
        public SampleTestListener(Dictionary<string, string> args)
        { }

        public override void OnTestExecutionBegin(TestExecutor testExecutor, TestExecutionBeginArgs args)
        {
            Debug.WriteLine("SampleTestListener method:  " + MethodInfo.GetCurrentMethod().Name);
        }

        public override void OnTestExecutionComplete(TestExecutor testExecutor, TestExecutionCompleteArgs args)
        {
            Debug.WriteLine("SampleTestListener method:  " + MethodInfo.GetCurrentMethod().Name);
            Thread.Sleep(5000);
            Debug.WriteLine("Test execution sleep complete");
        }

        public override void OnTestSuiteExecutionBegin(TestSuite testSuite, TestSuiteBeginExecutionArgs args)
        {
            Debug.WriteLine("SampleTestListener method:  " + MethodInfo.GetCurrentMethod().Name);
        }

        public override void OnTestPreprocessorBegin(TestSuite testSuite, TestProcessorBeginExecutionArgs args)
        {
            Debug.WriteLine("SampleTestListener method:  " + MethodInfo.GetCurrentMethod().Name);
        }

        public override void OnTestPreprocessorComplete(TestSuite testSuite, TestProcessorResult testProcessorResult)
        {
            Debug.WriteLine("SampleTestListener method:  " + MethodInfo.GetCurrentMethod().Name);
        }

        public override void OnTestPostprocessorBegin(TestSuite testSuite, TestProcessorBeginExecutionArgs args)
        {
            Debug.WriteLine("SampleTestListener method:  " + MethodInfo.GetCurrentMethod().Name);
        }

        public override void OnTestPostprocessorComplete(TestSuite testSuite, TestProcessorResult testProcessorResult)
        {
            Debug.WriteLine("SampleTestListener method:  " + MethodInfo.GetCurrentMethod().Name);
        }

        public override void OnTestSuiteExecutionComplete(TestSuite testSuite, TestSuiteResult testSuiteResult)
        {
            Debug.WriteLine("SampleTestListener method:  " + MethodInfo.GetCurrentMethod().Name);
        }

        public override void OnTestCaseExecutionBegin(TestCase testCase, TestCaseBeginExecutionArgs args)
        {
            Debug.WriteLine("SampleTestListener method:  " + MethodInfo.GetCurrentMethod().Name);
        }

        public override void OnTestCaseExecutionComplete(TestCase testCase, TestCaseResult testCaseResult)
        {
            Debug.WriteLine("SampleTestListener method:  " + MethodInfo.GetCurrentMethod().Name);
            Thread.Sleep(5000);
            Debug.WriteLine("Test case sleep complete");
        }

        public override void OnTestStepExecutionBegin(TestStep testStep, TestStepBeginExecutionArgs args)
        {
            Debug.WriteLine("SampleTestListener method:  " + MethodInfo.GetCurrentMethod().Name);
        }

        public override void OnTestStepExecutionComplete(TestStep testStep, TestStepResult testStepResult)
        {
            Debug.WriteLine("SampleTestListener method:  " + MethodInfo.GetCurrentMethod().Name);
        }

        public override void OnTestCheck(TestCheck testCheck)
        {
            Debug.WriteLine("SampleTestListener method:  " + MethodInfo.GetCurrentMethod().Name);
        }

        public override void OnTestWarning(TestWarning testWarning)
        {
            Debug.WriteLine("SampleTestListener method:  " + MethodInfo.GetCurrentMethod().Name);
        }

        public override void OnTestTrace(string virtualString, string traceMessage)
        {
            Debug.WriteLine("SampleTestListener method:  " + MethodInfo.GetCurrentMethod().Name);
        }

        [NoOperation]
        public override void OnTestMetric(string virtualUser, TestMetricEventArgs args)
        {
            throw new NotImplementedException();
        }
    }
}

Adding TestListeners to a test run

The final step is to register the TestListener with the TestListenerService. This is done by adding the listener to a test listener configuration file and passing that file to the runtime engine. As previously cited, the TestListenerService can support multiple TestListeners concurrently. To add and configure a TestListener, do the following:

  1. Launch the TestEngineer.
  2. Select "Tools" main menu item and "Test Listeners Editor..." from the dropdown. The editor will appear."
  3. Using the editor, open or create a configuration file (typically configuration files are placed in the TestConfigs folder.
  4. Add or modify the listener using the editor.
  5. Add the switch "/l" or "/L" pointing to the TestListener config file to the TestEngineer or TestRunner executable command line, for example " /l="[TestConfigs]\MyTestListeners.config" "

Sample TestRail listener configuration:

<?xml version="1.0" encoding="utf-8"?>
<TestListeners xmlns:i="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.datacontract.org/2004/07/Quintity.TestFramework.Runtime">
  <TestListener>
    <Name>TestRailListener</Name>
    <Description>Updates hosted TestRail application with results.</Description>
    <Status>Active</Status>
    <OnFailure>Stop</OnFailure>
    <Assembly>[TestListeners]\Quintity.TestFramework.TestListeners.TestRail.dll</Assembly>
    <Type>Quintity.TestFramework.TestListeners.TestRail.TestRailListener</Type>
    <Parameters xmlns:d3p1="http://schemas.microsoft.com/2003/10/Serialization/Arrays">
      <d3p1:KeyValueOfstringstring>
        <d3p1:Key>TestRailUrl</d3p1:Key>
        <d3p1:Value>[TestRailUrl]</d3p1:Value>
      </d3p1:KeyValueOfstringstring>
      <d3p1:KeyValueOfstringstring>
        <d3p1:Key>TestRailUser</d3p1:Key>
        <d3p1:Value>[TestRailUser]</d3p1:Value>
      </d3p1:KeyValueOfstringstring>
      <d3p1:KeyValueOfstringstring>
        <d3p1:Key>TestRailPassword</d3p1:Key>
        <d3p1:Value>[TestRailPassword]</d3p1:Value>
      </d3p1:KeyValueOfstringstring>
      <d3p1:KeyValueOfstringstring>
        <d3p1:Key>TestRailProject</d3p1:Key>
        <d3p1:Value>[TestRailProject]</d3p1:Value>
      </d3p1:KeyValueOfstringstring>
    </Parameters>
  </TestListener>
</TestListeners>

Quintity TestFramework

⚠️ **GitHub.com Fallback** ⚠️