Yesterday I finished writing refactorings planned for 1.5 and decided to diverse my activity by investigating possibility to add automated code coverage analysis to our build process. I chose Clover .NET as coverage tool mainly because there are not too many alternatives (NCover seems a bit raw) and that a team next door (Fabrique) successfully uses its big Java brother - Clover.
Integrating it turned out to be a tricky task as our build process is fairy complicated: we use NAnt as general workflow engine, VS.NET for building and multiple custom tools for code generation, etc. So I ended up writing another tool which takes our solution file, goes through its projects and creates copies of the projects which contain instrumented sources. At the end we get copy of our original solution which references instrumented projects.
But unfortunately Clover doesn't seem to handle projects of such size well: when I run unit tests on clovered binaries it starts allocating too much memory and fails with OutOfMemoryException when memory consumption reaches 1.5Gb. First I though we really have so much code and added another gig of RAM to my machine but this didn't help. I even added /3GB switch to the boot.ini file to extend address space for the process. No effect. I only could get results when I instrumented only one of our 40+ projects (medium sized one - about 42000 NCLOC). I got quite reasonable results (however, I have to confess that I wish the coverage was much better).
Then I decided to read through the instrumented sources to learn what the instrumentation process does. It actually creates static array for each instrumented class (of course, it is covered in special reporter class, but details don't matter now). Then at each statement and condition it increments certain elements of array. Indexes are presented as numeric literals in code so it must keep some database which matches these numbers to particular lines in code. First I thought that the problem was with these indexes, but after carefully reading the code through Reflector (it was reallly simple - only three classes in the runtime) I changed my mind. Finally it turned out that the problem was that Clover instrumented a class which was used in COM interop and the changes added static constructor to it which caused TypeLoadException in our code deeps. The exception somehow got uncaught and resulted in maintaining many thousands copies of test data in memory. When I excluded that file from instrumentation everything went ok except one problem with instrumenting boolean expressions containing assignments (I know that Clover has special options for it, but it didn't seem to help).
As a bottomline I don't regret that I spent so much time making it work as the test coverage report seems to be very helpful. We will try to make it run automatically and play with the results for a month or so before deciding whether we're going to use it in our development.