To generate a reasonably accurate JaCoCo code coverage report for a Scala project.
A Maven plug-in for creating JaCoCo code coverage reports is available on github and will be a helpful resource for following this article. This project contains an example that integrates Maven, JaCoCo and ScalaTest with this plug-in and can be used as a template for your own projects. The interesting technologies showcased include JaCoCo and Scala.
Suppose our development team would like to track how much of our Scala code is covered by our test suite. Now suppose we would like to configure Maven to generate these reports with each build.
The JaCoCo Maven plug-in can easily be configured to record and report coverage metrics for compiled Scala because JaCoCo uses a java agent to instrument bytecode on the fly. For example, given a project with standard layout, where Example.scala and ExampleSpec.scala represent a worker and its ScalaTest specification, then the pom configuration is fairly straightforward.
The project layout
Unfortunately, this configuration will soon begin reporting false negative results when even the most basic Scala features are employed. For example, suppose our Example class extends a Scala trait.
The coverage results show a false negative result for the mixed-in method ExampleTrait#thisIsMixedIn. We expect to see 100% coverage but instead see 75%.
We can see from the bytecode for ExampleTrait that the Scala compiler mixes in traits by generating bytecode for methods in the extending classes and these generated methods delegate to the implemented traits via static calls.
We can also see from the bytecode that the line numbers for the mixed-in method and the constructor are the same. Perhaps this coincidence can give us enough information to filter out these types of false negatives from our coverage report.
Similarly, case classes generate even more bytecode for methods that can be excluded from our coverage reports.
The coverage results show significantly more false negative results for mixed-in and generated methods. We expect to see coverage results closer to 100% than to 32%.
Again, we can see from the bytecode that the line numbers for the mixed-in methods and the constructor are the same. We can also see from the bytecode that generated methods follow a pattern: curried(), tupled() and (\w|\$)+\$default\$\d+\(\) methods. These methods unfortunately cannot be filtered by their debug line numbers, but they can be identified consistently by name. This information may be sufficient for eliminating these false negative results so that our coverage trend can at least look more reasonable even if not exact.
The JaCoCo project maintainers are currently in the process of collecting use cases for the types of configurable filtering options that the community needs. Until that general-purpose solution has been implemented, the jacoco-scala-maven-plugin can be used to fulfill two of those specific filtering needs, i.e., SCALAC.MIXIN and SCALAC.CASE. These two filters eliminate methods that have the same debug line numbers as the constructor and the names tupled(), curried() and (\w|\$)+\$default\$\d+\(\), as noted above.
In our pom, instead of configuring the jacoco-maven-plugin to emit a report, we can now pass these two filtering options to the jacoco-scala-maven-plugin.
When using these two filters, the coverage results for the TraitExample and CaseExample no longer include generated and mixed-in methods, and instead give us a more accurate coverage score of 100% for both.
By examining and resolving anomalies in the code coverage statistics for our project through the introduction of a jacoco-scala-maven-plugin, we have gained an insight into how the Scala compiler mixes and injects methods into generated bytecode (which can also give insight into how Scala is able to chain super calls when multiple traits are linearized). While there are certainly other generated instructions that can be ignored during a coverage run, these two filters get us closer to being able to gather useful coverage trend information over time.