Monitor Test Coverage and Identify Which Code Needs more Test

Knowing which code is exercised or not by your automatic tests suite is essential to significantly reduce the amount of bugs, avoid regressions and boost your team productivity.

You can import your code coverage data within the NDepend model from your preferred coverage technology: Visual Studio coverage, NCrunch, dotCover, NCover or OpenCover. With this data NDepend lets you:

  • Monitor test coverage
  • and Identify which code needs more test

Monitor test coverage

It is not enough to develop a test suite and obtain a high code coverage ratio. You must ensure that with time and evolution your code coverage ratio will remain high. This is why NDepend proposes a series of default rules related to test coverage. These rules are executed only if code coverage data is imported:

  • Code should be tested: For each method not 100% covered by tests an issue is created. For each issue, some smart formulas estimate the cost to write new tests to reach 100% coverage. On a poorly tested code base, this rule itself can generate a significant portion of the total technical-debt. It makes sense since originally the technical-debt metaphor concerned only untested code. Then other kind of code-smells added up.
  • New Methods should be tested and Methods refactored should be tested : On a large legacy code base it can be costly, tedious and unproductive to write tests to cover most of the code. However it is important to write tests for new code written and also test existing code refactored. This way developers learn to develop well with tests and the overall quality and productivity will increase progressively.
  • Types almost 100% tested should be 100% tested: Often covering the few percents of remaining uncovered code of a class, requires as much work as covering the first 90%. For this reason, often teams estimate that 90% coverage is enough. However untestable code usually means poorly designed code which usually leads to error prone code. So it might be worth refactoring and making sure to cover the few uncovered lines of code because most tricky bugs might come from this small portion of hard-to-test code.
  • Types that used to be 100% covered by tests should still be 100% covered and Types tagged with FullCoveredAttribute should be 100% covered: If, by chance or thanks to effort, a class is 100% covered by tests, it is a good idea to make sure that in the future it will remain 100% covered when it will be modified. This way you avoid coverage holes that are often a nest for tricky bugs that reach production. The first rule strategy is to make sure that types that were 100% covered by tests in the baseline remain 100% covered. The second rule strategy is stronger: You can define a FullCoveredAttribute or use the NDepend.Attributes.FullCoveredAttribute one and tag classes 100% covered by test. This way the developers can explicitly declare and comment their full test strategy intent and make sure it'll be respected in the future.

Code coverage rules might not be enough. You might want to fail the build upon some code coverage conditions not met. This is why some default quality gates are here to monitor code coverage:

Since quality gates are C# LINQ code queries it is easy to change thresholds and create your own quality gate.

Identify which code needs more test

NDepend provides a unique code metric visualization tool that relies on colored treemaping. For example the screenshot below shows code coverage by tests on the NDepend code base.

  • Rectangles represent the code hierarchy: assemblies, namespaces, types and methods
  • A rectangle area is here proportional to the number of lines of code of the corresponding element.
  • A rectangle color represents the code coverage ratio of the corresponding element: green means 100% ; red means 0%
NDepend Code Coverage visualized with a colored treemap

We are proud that the NDepend code is 86%+ covered by tests, including UI code. This high global coverage ratio is well illustrated by the colored treemap.

However we can easily identify which component is not covered enough. The assembly NDepend.Addin is almost all red and clearly needs more tests.

Both the size metric and the color metric can be chosen independently among a wide range of code metrics. For example it might be even more useful to compare method complexity with code coverage. The more complex a method is, the more likely it contains a bug, the more important it is to test it. On this particular complexity vs. coverage topic a default code is proposed: Methods should have a low C.R.A.P score.

You can watch this 2 minutes video about NDepend and code coverage: