Testing Unit Tests

How do we test the tests?

Unit testing is something that comes up a lot in projects, one question that I often get asked is how do we test the tests, it’s a bit similar to the who watches the watchers kind of question which often comes up when talking about things like CCTV, at some point we need to trust… However we can use scripting inside our builds to test the quality of our tests, in days gone past people would use code coverage going for 80%+ but you could cover that with one simple test that hits all the code but doesn’t actually test the quality… So that’s where Stryker .Net comes in.

Stryker .Net

Stryker .Net is a piece of tooling which lets us perform mutation testing on .Net Core and .Net Framework projects, this allows us to test the quality of our tests by inserting bugs

Example

So to use Stryker .Net one possible location would be in the build pipeline, you can the use this to break pull request builds where thresholds aren’t met, in the below example we can see we’re building a .NET Core project, running the unit tests and then there’s an install and run of the Stryker package followed by publishing the results.

 1# Sample Mutation Testing Pipeline
 2trigger:
 3- main
 4
 5pool:
 6  vmImage: ubuntu-latest
 7
 8variables:
 9  buildConfiguration: 'Release'
10  
11steps:
12- task: PowerShell@2
13  displayName: 'Run Build and Unit Tests'
14  inputs:
15    targetType: 'inline'
16    script: |
17      dotnet restore ./UnitTesting.sln 
18      dotnet build ./UnitTesting.sln --configuration $(buildConfiguration)
19      dotnet test ./UnitTesting.sln --configuration $(buildConfiguration) --logger trx      
20    pwsh: true
21
22- task: PublishTestResults@2
23  displayName: Publish Test Results
24  condition: succeededOrFailed()
25  inputs:
26    testRunner: VSTest
27    testResultsFiles: '**/*.trx'
28
29- task: PowerShell@2
30  displayName: 'Run Stryker DotNet'
31  inputs:
32    targetType: 'inline'
33    script: |
34      dotnet tool install dotnet-stryker --tool-path $(Agent.BuildDirectory)/tools     
35      $(Agent.BuildDirectory)/tools/dotnet-stryker      
36    pwsh: true
37    workingDirectory: '$(Build.SourcesDirectory)/Testing'
38
39- task: PublishMutationReport@0
40  condition: succeededOrFailed()
41  inputs:
42    reportPattern: '**/mutation-report.html'

When the results are published you can see the unit tests run -

As you can see from the above all the tests passed, however the build by the icon failed… Why?

Lets take a look at why the build failed… Lets see what the output from the build was…

Logs showing that it failed due to Stryker not meeting the minimum criteria

As we can see this failed due to mutations score being below the target, the target can be configured through the config file which is a JSON file kept in the test project root (not the solution root), the Stryker tool will then read it and apply those targets and error as needed.

As you can see in the original YAML file we also publish the output from the Stryker tool no matter if we have a success or failure status for the build, this produces an output similar to the below.

Results showing that the mutation score was 66.67%

We can now use Stryker to help ensure we validate our unit tests in our build pipelines.

More information is available from the Stryker site.

Tue 30 Mar 2021

How do we test the tests?

Stryker .Net

Example