MS Unittests (C sharp) - NextensArelB/SwaggerGenerationTool GitHub Wiki

[[TOC]]

##Introduction

On this wiki page, you will find an overview of Nextens recommended practices on the topic of Unit Testing.

##Getting started with writing unit tests If you are not yet acquainted with unit tests or would like a refresher on it, we highly recommend reading Microsoft's unit testing best practices.

###Nextens standards for C# Unit tests

• Within Nextens we use MSTest as C# unit test framework.

Note: The test project should have packages Microsoft.NET.Test.Sdk, MSTest.TestAdapter and MSTest.TestFramework installed.

• We use the assertion framework Shouldly
• Mocking frameworks. We use Moq as mocking framework. For solutions that implement AutoFac for dependency injection, it is recommended to also implement the AutoFac.Extras.Moq package for auto-mocking. For other solutions, it is recommended to implement the Moq.Automock package for auto-mocking.
• If you are using Visual Studio Professional we recommend the use of a plugin (like Fine code coverage) to see the coverage in your code and identify risk hotspots. Visual Studio Enterprise has built-in support for code coverage analysis and is also able to read downloaded coverage reports from build pipelines.

Any other testing frameworks and/or libraries are not allowed to use (eg. xUnit, NUnit, RhinoMock, FakeItEasy, NSubstitute, etc.) to ensure a consistent unit testing experience throughout the Nextens code base!

###About your C# code in general In order for your code to be more easily readable, maintainable and testable, keep the following rules in mind:

• Methods should perform one main task
• Methods should be as simple as possible (less if-then-else / for-each etc. statements)
• Methods should be small (less than ~20 lines)
• Methods should be loosely coupled (use dependency injection where possible)

More generally speaking, always follow well-established development paradigms like SOLID, DRY, etc.

###Recommendations for MS Unit tests

• Unit tests should be fast. Keep the individual tests small and simple, to ensure that tests run as quickly as possible.
• Avoid dependencies, use mock objects to remove dependencies on (external) services.
  • Use an auto-mocking framework so that you only need to setup dependencies that are required for your test.
• No logic in the tests (if-statements, loops, etc.)
• You can assert multiple times on one result but do not assert on different result items, create separate tests in that case.
• Test edge cases. Verify that you can test all execution paths and non happy flows in the tested code (Mocking and code coverage analysis can be helpful)
  • Perform tests on error handling: you need to test exceptions and logging
• Make yourself familiar with the extensive documentation provided by MS. It also covers the more complex testable code constructs.
• Run Tests Regularly. Make Unit tests part of your build pipelines so that they run every time new code is submitted. Keep track of the Unit test failures and show coverage reports.
• If a Unit test fails make a bug: either the Unit test needs to be modified or something is wrong in the code (regression)
  • Do not fix the Unit test without understanding why the test is failing: it might be that the reason the test is failing is due to a bug in the code or project.
• It should be clear from the unit test name or description what is being tested
  • If the unit test is complex or testing advanced logic that might not be immediately clear to others, adding comments to explain the tests and the flow could help clarify uncertainties.

  Note: this actually a red flag: if a unit test is complex, this probably means that the method that is being tested is too complex and should be refactored.

• Before you commit code, make sure it is covered by unit tests and all new and existing tests run successfully.
• For parts of (configuration) code that should not part of the code coverage use the [ExcludeFromCodeCoverage] attribute
• Maintaining the unit tests is a team responsibility: when code changes, corresponding unit tests need to be changed and a QA'er should check that the tests are valid again. This can be part of the DoD for each feature that has been delivered.

Note: in rare cases a unit test may be ignored by adding the [Ignore] attribute to a test method or entire test class. However, this must always be done in combination with a reference to a work item or task to fix it! eg. [Ignore("Todo fix this test - task 123456")]. Ignored tests without further explanation should be considered bugs.

• Tests should be independent and not rely on execution order. Avoid shared state between tests. Independent tests can can be parallized to increase executing speed.

##Unit testing best practices

###Data driven tests When you are creating a test for a function that should perform an action based upon different input parameters, it is recommended to use a data driven unit test function, instead of writing multiple almost identical unit tests. More information on how to do so can be found here.

Be careful however with the [DataRow] attribute. As a rule of thumb, for tests that require a few parameters (up to three or four) you can use the inline type of tests with data rows. With more parameters, it is better to create a test model class and use the [DynamicData] attribute.

When using the [DynamicData] attribute, keep the following in mind:

• The easiest way to use the attribute is to provide the name of a method that provides the test input data. The method that provides the data must be static and the return type must be IEnumerable<object[]>. Also you need to specify that the data source is a method, eg.: [DynamicData(nameof(GetOverzichtVerkrijgingTotaalModel), DynamicDataSourceType.Method)]
• For better display names, use the DynamicDataDisplayName property. This requires a public static method that accepts parameters MethodInfo methodInfo, object[] data. This helper method can be put in a separate assembly for easier re-use. In that case, also specify the DynamicDataDisplayNameDeclaringType. For example, Base3 contains an implementation (GetCustomDynamicDataDisplayName) that searches for a property that is decorated with the DisplayName attribute.
• For larger test files with multiple models and provider methods, consider splitting code into multiple files / (partial) classes.
• Copilot can be of great help to migrate existing unit tests.

Note: it is not necessary to use the [DataTestMethod] attribute, the regular TestMethod attribute also supports data driven tests.

###Test coverage for unit tests The best practice is to create two sets of unit tests:

a. Unit tests that test per function and mock the external dependencies:

You can keep the unit test close to the code in the same solution in the folder structure (like described below). They test each function and (external) dependencies are mocked. These tests are executed as part of the build pipeline.

b. Integration Unit tests:

These unit tests are stored in a separate repository and can be run using a separate pipeline. The dependencies need to be installed and configured (for example external API's) before they can manually be started by running the test pipeline.

###Unit tests and functional design Unit tests should be based upon the requirements or functional design, rather than the implemented code. In an ideal world the tests should be written first, based on the requirements (Test Driven Development).

###Where to create your test projects There are two approaches as to where to store your test projects. Regardless of which approach you choose to use It is crucial is that the structure remains consistent for the whole solution.

Note: Test project folder names always have the suffix *.Tests

This approach uses a single folder at the solution level where all your test projects are stored.

Pros:

• Single location makes it easy to locate all the test projects

Cons:

• Replicates the solution structure and can be cumbersome to maintain if the solution is large or projects are moved

Example

├─ MySolution.sln
|
├─ Controllers
│  ├─ SampleControllerProject1
│  └─ SampleControllerProject2
|
├─ DataAccess
|  ├─ SampleDataAccessProject1
|  └─ SampleDataAccessProject2
|
├─ Business
|  ├─ SampleBusinessProject1
|  └─ SampleBusinessProject2
|
└─ UnitTests
  |
  ├─ Controllers
  |  ├─ SampleControllerProject1.Tests
  |  └─ SampleControllerProject2.Tests
  |
  ├─ DataAccess
  |  ├─ SampleDataAccessProject1.Tests
  |  └─ SampleDataAccessProject2.Tests
  |
  └─ Busines
     ├─ SampleBusinessProject1.Tests
     └─ SampleBusinessProject2.Tests

This approach keeps test projects next to the project they test.

Pros:

• When moving a project it is easy to move the associated tests

Cons:

• Overview of all test projects when using a file explorer is cumbersome.

Example

├─ MySolution.sln
|
├─ Controllers
│  ├─ SampleControllerProject1
|  ├─ SampleControllerProject1.Tests
│  ├─ SampleControllerProject2
|  └─ SampleControllerProject2.Tests
|
├─ DataAccess
|  ├─ SampleDataAccessProject1
|  ├─ SampleDataAccessProject1.Tests
|  ├─ SampleDataAccessProject2
|  └─ SampleDataAccessProject2.Tests
|
└─ Business
   ├─ SampleBusinessProject1
   ├─ SampleBusinessProject1.Tests
   ├─ SampleBusinessProject2
   └─ SampleBusinessProject2.Tests

###Naming test files Test file naming is based on classes.

Note: Test file names always have the suffix *Tests without the period .

Example

├─ SampleProject
|  ├─ SampleProject.csproj
|  ├─ SampleClass1.cs
|  └─ SampleClass2.cs
|
└─ SampleProject.Tests
   ├─ SampleClass1Tests.cs
   └─ SampleClass2Tests.cs

###Naming test methods The name of your test method should consist of three parts:

• The name of the method being tested.
• The scenario under which it's being tested.
• The expected behaviour when the scenario is invoked.

Do not worry about your method names becoming "too long". Descriptive test names are valuable for quickly resolving broken tests and getting the correct context of the test at a glance. See also section 'Better test descriptions in reports'

Examples showing the naming of the test methods:

[TestMethod]
public void GetParameterValueFromBlob_BlobBaseUndefined_ThrowsException()
{
    // GetParameterValueFromBlob is the method we are testing
    // BlobBaseUndefined is the scenario we are testing
    // ThrowsException is the result we expect for the scenario we are testing
}

[TestMethod]
public void Resolve_NullTitle_ReturnsTypeNone()
{
    // Resolve is the method we are testing
    // NullTitle is the scenario we are testing
    // ReturnsTypeNone is the result we expect for the scenario we are testing
}

For methods with generic names (eg. 'GetById') that may occur multiple times in your solution. it is recommended to prefix the tests with the class name (ProductController_GetById_ReturnsProduct()). Unit tests in the fiscal domain are prefixed by the fiscal product and year because code is duplicated each year, which would lead to many duplicate tests names (eg. IB2023_Box3_ShouldNotBeEmpty())

###Test method body structure Use the 'Triple A' Arrange, Act, Assert structure within your tests. They are always present as comments in the test method.

The Arrange section is where you arrange the input for your test to execute. Create the minimum required mocks of dependencies and data needed to test the scenario. The Act section is usually one line of code which invokes the method you are testing. The Assert section is where you confirm (assert) that your expectation of the scenario being tested is correct.

Example that shows the test function body:

[TestMethod]
public void Resolve_VolumeType_None_ReturnsCorrectTitle()
{
    // Arrange
    string id = "book-id";
    string title = "Book Title";
    var bookCollection = new BookCollection
    {
        Title = title,
        Books = new List<Book> {
            new Book {
                ContentId = id,
                VolumeType = VolumeType.None,
                ValidFrom = new DateTime(2020, 01, 01)
            }
        }
    };
    var resolver = new BookNameResolver();

    // Act
    var result = resolver.Resolve(id, bookCollection);

    // Assert
    result.ShouldBe($"{title} 2020");
}

###Better test descriptions in reports When your tests are converted into a report, only the method name is used for display. Usually, this is descriptive enough to determine what is being tested. However, sometimes your code will follow a convention and multiple tests with similar naming structure will appear in your test report and you will be unable to deduce which test is for which namespace.

Example

// Modules\Clients\GetController.cs
public IActionResult Get(string id)
{
  // Do stuff
}

// Modules\Tenants\GetController.cs
public IActionResult Get(string id)
{
  // Do stuff
}

In this scenario, our tests will likely be

Get_InvalidId_ReturnsNotFound
Get_InvalidId_ReturnsNotFound

To avoid confusion and provide the necessary context we provide a description to the [TestMethod] attribute.

Example

// Modules\Clients.Tests\GetControllerTests.cs
[TestMethod("clients/{id} with invalid id returns 404")
public void Get_InvalidId_ReturnsNotFound()
{
  // Do stuff
}

// Modules\Tenants.Tests\GetControllerTests.cs
[TestMethod("tenants/{id} with invalid id returns 404")
public void Get_InvalidId_ReturnsNotFound()
{
  // Do stuff
}

The report will now display the descriptive text instead of the method name.

Example pipeline test report:

When you run unit tests in your local environment, it can make a huge difference in execution time -especially in large solutions with a great number of unit tests- to execute unit tests in parallel. With parallel execution enabled, unit tests are executed in parallel per assembly (unit test project). You can enable parallel execution in the settings in Test Explorer:

Note: the settings cog may be hidden if the Explorer pane is too narrow to display all menu items In that case you can click on the two small triangles on the right to reveal the remaining menu items (or resize the pane width)

Note: This option is disabled by default and has to be enabled per solution.

To further increase parallelization, you can add the Parallelize attribute to an assembly (add it to an AssemblyInfo.cs file in your test project):

using Microsoft.VisualStudio.TestTools.UnitTesting;

// This will run all test classes in parallel (using the number of cores available)
[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.ClassLevel)]

The Workers property defines the number of threads (0 means it will use the number of cores of your computer). ExecutionScope is 'ClassLevel' by default, which means that test classes are executed in parallel, but sequentially within the test classes. For maximum parallelization, it can also be set to 'MethodLevel', which means that tests within a test class are also executed in parallel. Be careful here though, as this will only work correctly for stateless unit tests (ie. not depending on shared class variables).

To exclude a method or entire test class from parallelization, the [DoNotParallelize] attribute can be added to the test method or class. With this attribute you can still benefit from parallelization on method level, but exclude tests that should be executed sequentially.

Note: due to a bug in Visual Studio, the displayed test duration will be the sum of all executed tests, instead of the actual test duration. A rough estimate of the real duration would be the test that took the longest time to execute.

It is possible to repeat a unit test run until a test fails. This may be useful to detect rare race conditions, or to verify that tests can safely run in parallel, like explained in the previous section. To use this feature, choose 'Run until failure' after right-clicking on a unit test (assembly, class or single test). The tests do not repeat infinitely, but a predefined number of times that can be configured in the test settings (1000 by default).

###Configure unit tests by using a .runsettings file For advanced scenarios you can add a .runsettings file to your solution to configure how tests are executed. For example to filter out certain tests, set global configuration properties, configure data collectors, logging, etc.
Check the documentation for a complete overview what can be configured. How to configure code coverage

###Use [TestCategory] attribute In large solutions with many unit tests, it may be useful to use the [TestCategory] attribute to group unit tests into categories. You can use categories to group tests in Test Explorer (Group by 'Traits') or in the Unit Test Explorer from Resharper. Unit test filtering can also be done on test categories.

##Unit tests in the CI/CD pipeline The actual execution of the MS Unit tests should be part of the CI/CD pipeline and can be included as part of the pipeline run. If you open the pipeline when it is finished you see the "Tests" tab on top of the page where you can see the results:

##The Unit test review process Unit tests should be reviewed by a developer and a tester/QA engineer. This is to make sure the syntax, structure and naming is verified by the developer and that the coverage of the tests is monitored by the Tester/QA. This way of working will also educate the Tester/QA in the working and syntax of the tests.

More information on the review process done by QA can be found here.

##Further reading Unit testing C# with MSTest and .NET For developers developing Unit tests: Run unit tests with Test Explorer Have a look at efficient unit tests: DRY vs DAMP in Unit tests