How do we test the tests?
Unit testing is something that comes up a lot in projects, one question that I often get asked is how do we test the tests, it’s a bit similar to the who watches the watchers kind of question which often comes up when talking about things like CCTV, at some point we need to trust… However we can use scripting inside our builds to test the quality of our tests, in days gone past people would use code coverage going for 80%+ but you could cover that with one simple test that hits all the code but doesn’t actually test the quality… So that’s where Stryker .Net comes in.
Stryker .Net is a piece of tooling which lets us perform mutation testing on .Net Core and .Net Framework projects, this allows us to test the quality of our tests by inserting bugs
So to use Stryker .Net one possible location would be in the build pipeline, you can the use this to break pull request builds where thresholds aren’t met, in the below example we can see we’re building a .NET Core project, running the unit tests and then there’s an install and run of the Stryker package followed by publishing the results.
1# Sample Mutation Testing Pipeline 2trigger: 3- main 4 5pool: 6 vmImage: ubuntu-latest 7 8variables: 9 buildConfiguration: 'Release' 10 11steps: 12- task: PowerShell@2 13 displayName: 'Run Build and Unit Tests' 14 inputs: 15 targetType: 'inline' 16 script: | 17 dotnet restore ./UnitTesting.sln 18 dotnet build ./UnitTesting.sln --configuration $(buildConfiguration) 19 dotnet test ./UnitTesting.sln --configuration $(buildConfiguration) --logger trx 20 pwsh: true 21 22- task: PublishTestResults@2 23 displayName: Publish Test Results 24 condition: succeededOrFailed() 25 inputs: 26 testRunner: VSTest 27 testResultsFiles: '**/*.trx' 28 29- task: PowerShell@2 30 displayName: 'Run Stryker DotNet' 31 inputs: 32 targetType: 'inline' 33 script: | 34 dotnet tool install dotnet-stryker --tool-path $(Agent.BuildDirectory)/tools 35 $(Agent.BuildDirectory)/tools/dotnet-stryker 36 pwsh: true 37 workingDirectory: '$(Build.SourcesDirectory)/Testing' 38 39- task: PublishMutationReport@0 40 condition: succeededOrFailed() 41 inputs: 42 reportPattern: '**/mutation-report.html'
When the results are published you can see the unit tests run -
As you can see from the above all the tests passed, however the build by the icon failed… Why?
Lets take a look at why the build failed… Lets see what the output from the build was… As we can see this failed due to mutations score being below the target, the target can be configured through the config file which is a JSON file kept in the test project root (not the solution root), the Stryker tool will then read it and apply those targets and error as needed.
As you can see in the original YAML file we also publish the output from the Stryker tool no matter if we have a success or failure status for the build, this produces an output similar to the below.
We can now use Stryker to help ensure we validate our unit tests in our build pipelines.
More information is available from the Stryker site.