Fixie v2 Documentation - fixie/fixie GitHub Wiki

Quick Start

Fixie is a .NET test framework similar to NUnit and xUnit, but with an emphasis on low-ceremony defaults and flexible customization.

Add a test project to your solution. Target net452 or higher, netcoreapp2.1, netcoreapp3.1, or a combination. Reference the Fixie and Fixie.Console NuGet packages. Install a third-party assertion library such as Shouldly. Unlike MSTest and xUnit, you do not need to reference Microsoft.NET.Test.Sdk yourself.

<!-- src/Directory.build.props -->
<Project>
    <PropertyGroup>
        <FixieVersion>2.2.1</FixieVersion>
        <ShouldlyVersion>3.0.2</ShouldlyVersion>
    </PropertyGroup>
</Project>
<!-- src/Example.Tests/Example.Tests.csproj -->
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>netcoreapp3.1;net471</TargetFrameworks>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Fixie" Version="$(FixieVersion)" />
    <PackageReference Include="Shouldly" Version="$(ShouldlyVersion)" />
    <DotNetCliToolReference Include="Fixie.Console" Version="$(FixieVersion)" />
  </ItemGroup>
</Project>

Note you will probably have to add the Fixie.Console reference by editing the project file. Visual Studio NuGet Manager disallows referencing executables as mentioned here.

Add a test class to the project. By default, a test class is any class whose name ends with Tests, and public methods in that class will be treated as tests. The test class will be instantiated once per test method, like xUnit does. These defaults can be customized, but provide a simple starting point:

using Shouldly;

public class CalculatorTests
{
    public void ShouldAdd()
    {
        var calculator = new Calculator();
        calculator.Add(2, 3).ShouldBe(5);
    }

    public void ShouldSubtract()
    {
        var calculator = new Calculator();
        calculator.Subtract(5, 3).ShouldBe(2);
    }
}

To run your tests from the command line, navigate into the test project folder and execute dotnet fixie:

$ cd Example.Tests
$ dotnet fixie

Running Example.Tests (netcoreapp3.1)

2 passed, took 0.02 seconds

Running Example.Tests (net471)

2 passed, took 0.01 seconds

To run your tests from inside Visual Studio, open Test Explorer (Test \ Windows \ Test Explorer). Rebuild your solution and click Run All to get started.

Customizing the Test Assembly Lifecycle

.NET test frameworks have two phases: discovery and execution.

In the discovery phase, the framework inspects your test project and finds all the test classes and test methods. For instance, NUnit scans for attributes like [Test], xUnit scans for [Fact], and Fixie scans for the test class naming convention seen above. The discovery phase may happen all on its own, such as when Visual Studio needs to find out the full list of available tests to display within Test Explorer.

In the execution phase, the discovered test classes are instantiated, the discovered test methods are invoked, and their results are tallied and reported.

Each discovered test method name is a Test. When such a test method is parameterized, that Test may be run multiple times. Each specific run of a Test is called a Case, and each Case can individually pass, fail, or be skipped.

Customizing the Discovery Phase

To customize the test discovery phase, place a subclass of Discovery in your test project. From the constructor, you can customize the discovery phase's behavior for discovering test classes, test methods in those classes, and input parameters for parameterized test methods.

For example, say you wish to mark test classes with a [TestClass] attribute instead of the default "*Tests" naming convention, and you wish to mark test methods with a [Test] attribute instead of the default inclusion of all public methods. Additionally, you'd like to allow for parameterized tests by placing [Input(1, "B", 3.0)] attributes on a test method. Begin by defining those attributes:

[AttributeUsage(AttributeTargets.Class)]
public class TestClass : Attribute { }

[AttributeUsage(AttributeTargets.Method)]
public class Test : Attribute { }

[AttributeUsage(AttributeTargets.Method, AllowMultiple = true)]
public class Input : Attribute
{
    public Input(params object[] parameters)
    {
        Parameters = parameters;
    }

    public object[] Parameters { get; }
}

Next, customize the discovery phase:

using Fixie;

public class TestingConvention : Discovery
{
    public TestingConvention()
    {
        Classes
            .Where(x => x.Has<TestClass>());

        Methods
            .Where(x => x.Has<Test>());

        Parameters
            .Add<InputParameterSource>();
}

public class InputParameterSource : ParameterSource
{
    public IEnumerable<object[]> GetParameters(MethodInfo method)
    {
        var inputAttributes = method.GetCustomAttributes<Input>().ToArray();

        foreach (var input in inputAttributes)
            yield return input.Parameters;
    }
}

Lastly, write test classes which use your new testing convention:

[TestClass]
public class CalculatorTester
{
    public void NotATest()
    {
        //This is no longer treated as a test, because it lacks the [Test] attribute.
    }

    [Test]
    public void ShouldAdd()
    {
        var calculator = new Calculator();
        calculator.Add(2, 3).ShouldBe(5);
    }

    [Test]
    [Input(5, 3, 2)]
    [Input(8, 5, 30)]
    public void ShouldSubtract(int x, int y, int expectedDifference)
    {
        //This Test is invoked twice, once for each [Input(...)],
        //resulting in two Cases, the first of which passes and the
        //second of which fails.
        var calculator = new Calculator();
        calculator.Subtract(x, y).ShouldBe(expectedDifference);
    }
}

Customizing the Execution Phase

By default, the test class execution phase performs the following steps for each discovered test class:

For each test method in the test class:
    Construct the test class using its default constructor.
    Invoke the test method.
    Dispose of the test class instance, if it is IDisposable.

In other words, your test classes are constructed frequently, like xUnit, meaning that you can use the constructor and Dispose() methods for test setup and test teardown.

However, if you don't want to use this default test class lifecycle, you can define your own. For example, say you wish to instead have the infrequent test class construction familiar to NUnit users:

Construct the test class using its default constructor.
For each test method in the test class:
    Invoke the test method.
Dispose of the test class instance, if it is IDisposable.

To customize the test execution phase, place a subclass of Execution in your test project.

using Fixie;

public class TestingConvention : Execution
{
    public void Execute(TestClass testClass)
    {
        var instance = testClass.Construct();

        testClass.RunCases(@case =>
        {
            @case.Execute(instance);
        });

        instance.Dispose();
    }
}

Lastly, write test classes which use your new testing convention:

public class CalculatorTests
{
    private readonly Calculator calculator;

    public CalculatorTests()
    {
        calculator = new Calculator();
    }

    public void ShouldAdd()
    {
        calculator.Add(2, 3).ShouldBe(5);
    }

    public void ShouldSubtract()
    {
        calculator.Subtract(5, 3).ShouldBe(2);
    }
}

With this testing convention in place, CalculatorTests would be constructed once, and the Calculator instance would be shared across both tests.

For completeness, this means that the default Execution implementation, when you provide no such Execution class yourself, is:

class DefaultExecution : Execution
{
    public void Execute(TestClass testClass)
    {
        testClass.RunCases(@case =>
        {
            var instance = testClass.Construct();

            @case.Execute(instance);

            instance.Dispose();
        });
    }
}

By defining your own Execution, you control every aspect of the test class lifecycle: how to construct the test class (you might have some opinion about the meaning of test class constructor arguments), construction frequency, what steps to take before and after each test class, and what steps to take before and after each test case.

Putting it all Together

Imagine you start testing a new system and you occasionally find that the test methods in your test class all share a few setup steps. You might factor that code into the test class constructor, allowing each test method to focus on the individual scenario at hand.

That's great, until you begin to test a feature that exposes async methods. Test methods can be async, but C# constructors cannot. It might seem that you're stuck with leaving out the test class constructor, and repeating the async setup steps within each test method.

Instead, we can take over both the discovery and execution phases in order to describe a synchronous-or-asynchronous test setup convention:

using System;
using System.Linq;
using System.Reflection;
using Fixie;

public class TestingConvention : Discovery, Execution
{
    public TestingConvention()
    {
        Methods
            .Where(x => x.Name != "SetUp");
    }

    public void Execute(TestClass testClass)
    {
        testClass.RunCases(@case =>
        {
            var instance = testClass.Construct();

            SetUp(instance);

            @case.Execute(instance);
        });
    }

    static void SetUp(object instance)
    {
        instance.GetType().GetMethod("SetUp")?.Execute(instance);
    }
}

Here, we customize the discovery phase so that if a method appears in a test class but is named "SetUp", Fixie will not consider it to be a test method. SetUp(), if present, will no longer be invoked by the test runner during the test class lifecycle.

Next, we customize the execution phase so that we will invoke SetUp() ourselves, exactly when we want to: right after test class instantiation, right before a test case is executed.

The final call to the MethodInfo.Execute(object instance, params object[] arguments) helper method works whether or not a test class's optional SetUp() method is async, ensuring the task is properly awaited.

With this convention, we can write test classes in a few ways:

  1. A test class could have no SetUp() method and will run just like the default behavior.
  2. A test class could have a synchronous public void SetUp(), and it will be invoked right before each test case.
  3. A test class could have an asynchronous public async Task SetUp(), and it will be invoked and awaited right before each test case.

Command Line Arguments

Usage: dotnet fixie [options] [-- [custom arguments]...]


    --configuration name
        The configuration under which to build. When this option
        is omitted, the default configuration is `Debug`.

    --no-build
        Skip building the test project prior to running it.

    --framework name
        Only run test assemblies targeting a specific framework.

    --report path
        Write test results to the specified path, using the
        xUnit XML format.

    --
        Signifies the end of built-in arguments and the beginning
        of custom arguments.

    custom arguments
        Arbitrary arguments made available to custom discovery/execution classes.

Custom Arguments

A discovery or execution customization class can access arbitrary command line arguments. Any command line arguments which appear after the special -- separator argument will be model-bound to your constructor arguments.

For instance, let's say you want to mark some of your tests as being in special categories, such as those which take a long time to run, or those which perform full integration tests with less-than-reliable systems. You'd like these to run only when you explicitly ask for them. First, define a few attributes:

[AttributeUsage(AttributeTargets.Method, Inherited = false)]
public abstract class Category : Attribute
{
    public string Name => GetType().Name;
}

[AttributeUsage(AttributeTargets.Method, Inherited = false)]
public class LongRunning : Category { }

[AttributeUsage(AttributeTargets.Method, Inherited = false)]
public class Integration : Category { }

Next, our testing convention accepts an optional array of category names in its constructor, and uses them to customize the discovery phase: a method with one of these attributes isn't a test method unless the user asked for it to run.

using Fixie;

public class TestingConvention : Discovery
{
    public TestingConvention(string[] include)
    {
        Methods
            .Where(x => MethodShouldRun(x, include));
    }

    static bool MethodShouldRun(MethodInfo method, string[] include)
    {
        var categories = method.GetCustomAttributes<Category>().ToArray();

        if (categories.Length == 0)
            return true;

        return categories.Any(category => include.Contains(category.Name));
    }
}

Armed with this testing convention, test methods can be marked with these optional categories:

public class ExampleTests
{
    [LongRunning]
    public void GetNextChessMove()
    {
        //A long-running test.
    }

    [Integration]
    public void IntegrateWithUnreliableService()
    {
        //An integration test intended for a CI build.
    }

    public void AlwaysRuns()
    {
        //A test that always runs.
    }
}

Lastly, we can optionally include categories at the command line:

dotnet fixie --configuration Release -- --include LongRunning --include Integration

At runtime, the values after -- are bound to the TestingConvention constructor parameters by naming convention and type hints: new string[] { "LongRunning", "Integration" }. Most arguments will naturally appear as --name value pairs. However, if a constructor argument is a bool, then you can avoid an explicit value and simply include the --name to signify true.

Test Runner Integrations

Visual Studio

Tests are discovered and executed in Test Explorer. When a test method has multiple test cases, such as when a custom convention invokes a parameterized test method multiple times with different inputs, each individual pass or failure appears in the detail pane:

Test Explorer

Azure DevOps

When dotnet fixie runs under Azure DevOps, results are reported to the "Tests" tab:

Azure DevOps Report

This support requires that you enable access to DevOps' test results API. To enable that access, ensure your Pipeline script includes the $(System.AccessToken) variable in environment variable SYSTEM_ACCESSTOKEN per the DevOps predefined variables documentation:

pool:
  vmImage: windows-latest

steps:
- script: build
  env:
    SYSTEM_ACCESSTOKEN: $(System.AccessToken)

TeamCity

When dotnet fixie runs under TeamCity, results are reported automatically.

AppVeyor

When dotnet fixie runs under AppVeyor, results are reported automatically.

XML Reports

For other Continuous Integration systems, you can enable XML reporting using the xUnit 2 format as a fallback, though you may find the console logging capture sufficient instead.

TestDriven.NET

The TestDriven.NET plugin itself does not yet support .NET Core runs, and it does not work for test projects that have multiple <TargetFrameworks>. Instead, it can only be expected to work if you have a single netXX framework listed in the singular <TargetFramework> tag.

F# Support

Installation and usage within an F# test project is essentially the same as for C#, though F#'s inherent brevity for all-static modules allows for extra-clean test files:

<!-- Example.Tests.fsproj -->
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Fixie" Version="2.0.2" />
    <PackageReference Include="Shouldly" Version="2.8.3" />
    <DotNetCliToolReference Include="Fixie.Console" Version="2.0.2" />
  </ItemGroup>

  <ItemGroup>
    <Compile Include="CalculatorTests.fs" />
  </ItemGroup>

</Project>
module Example.Tests.CalculatorTests

open Shouldly

let ShouldAdd() =
    let calculator = new Calculator()
    calculator.Add(2, 3).ShouldBe(5)

let ShouldSubtract() =
    let calculator = new Calculator()
    calculator.Subtract(5, 3).ShouldBe(2)
Add a custom footer
⚠️ **GitHub.com Fallback** ⚠️