Posts Tagged ‘unit testing’

TeamCity plugin for Visual Studio

Tuesday, March 12th, 2013

When working with TeamCity and Visual Studio, we can do a lot of things right from within our IDE. We can trigger a remote run, fetch dotCover code coverage information from the server, view changes and builds from a tool window, navigate to unit tests and a lot more. In this post, we’ll be looking at some of these features. But first things first: let’s install this nifty tool!

Installing the plugin

Every TeamCity installation ships with several tools for IDE integration (with our IntelliJ IDEA based tools as well as with Visual Studio). We can find the plugin on every TeamCity server under the My Settings & Tools page. We can download the plugin right there!

Finding the TeamCity plugin for Visual Studio

After downloading and installing the plugin, we can find a new menu item in our Visual Studio.

TeamCity plugin for Visual Studio menu entry

From this menu, we can connect to our TeamCity server using the Login… menu item. After entering the URL to our TeamCity server and providing the correct credentials, we can start exploring.

Login to TeamCity

Remote run from Visual Studio

When developing a project in Visual Studio, we can initiate a personal build using the TeamCity plugin for Visual Studio. We call this a “remote run” because the build that is triggered runs on a TeamCity build agent, not on the developer’s machine. The interesting thing here is that this remote build uses the current version control repository sources plus the changed files in the developer’s IDE. All steps from the build configuration are executed for this personal build as well.

After changing some files locally, we can use the TeamCity | Remote Run (Local Changes) menu to trigger a remote build. In the dialog that opens, we can select the changes we made locally that should be included in this personal build. We can select all changes or cherry-pick just the changes we want to verify on the build server.

Local changes for personal build - remote run

Note that we’re using Subversion as the source control system here. Remote Run is available for TFS, Subversion and Perforce. When using Git or Mercurial, the workflow is slightly different. Check the documentation on branch remote run for more information.

When we click the Configure personal build… icon in the toolbar, we have to make some other decisions. First of all, we must select the build configuration we want to use for the personal build. Next, we can provide a comment for this personal build. This comment will be shown in the TeamCity web UI afterwards to describe the personal build.

One interesting option is the Pre-tested Commit checkbox and its related commit if setting. Using this, submitted code changes first go through testing. If all tests pass, TeamCity will automatically commit the changes to version control and integrate it in the next build. When tests fail, the code is not committed and the developer who made the change is notified about this. Here’s a chart of the pre-tested commit workflow.

Personal build configuration and pre-tested commit

We can even customize our build: put it at the top of the queue or add additional build parameters.

After clicking the Run button, TeamCity will run the selected build configuration for the included changes. We can see the results in the TeamCity web UI, consult the build log, check unit test results and so on.

TeamCity web UI personal build

My Changes

Since this post is about the TeamCity plugin for Visual Studio, we can also verify the status of builds triggered because of our changes by using the TeamCity | My Changes menu.

Overview of My Changes

Build log

From the toolbar, we can consult the build log for every personal build listed in the My Changes window. Clicking the Show Build Log icon (or right-clicking the build and selecting the appropriate context menu) will instruct the TeamCity plugin for Visual Studio to download the build log directly from TeamCity.

Build log from TeamCity downloaded in Visual Studio

Open Failed Tests

Did your changes cause a unit test to fail? No worries: we can use the Open Failed Tests context menu from the My Changes window in order to see what is going on. From the window that opens we can re-run the failing tests locally using the ReSharper test runner.

Unit tests

Code Coverage

When you have dotCover installed on your machine, the TeamCity plugin for Visual Studio enables us to view code coverage results. Using the TeamCity | Coverage menu item we can select a code coverage snapshot to open.

TeamCity dotCover code coverage

After selecting and opening a snapshot, we get dotCover’s test runner showing code coverage. We can even double-click a class from the snapshot shown and explore code coverage at the statement level.

dotCover snapshot from TeamCity

Investigate a build

Whenever a build fails, we can volunteer to fix the build by starting an investigation. From the TeamCity | My Investigations menu, we can manage our investigations and take action on open investigations by either fixing it or giving up (when working in developer teams in Belgium, that last option typically results in having to bring pastries for the team).

Investigate a build

Open in IDE

The TeamCity web UI features an Open in IDE button on many places. For example, when inspecting changes that were included in a build, we can open the file that was built in our IDE by clicking the IDE icon.

Open in IDE from TeamCity web UI

We can open tests in the IDE as well, again using the Open in IDE function. When working with tests, this will trigger Visual Studio to open the ReSharper test runner and display the selected test.

Open test in IDE

Give the TeamCity plugin for Visual Studio a go and let us know what you think!

ReSharper plugins for unit testing - MSpec, xUnit.net and Silverlight

Monday, January 21st, 2013

ReSharper provides comprehensive support for running and debugging unit tests. Out of the box, it works with NUnit and Microsoft’s Visual Studio Unit Testing Framework (henceforth mstest for brevity) for .Net languages, and supports QUnit and Jasmine for JavaScript.

The same APIs that ReSharper uses to implement this support are fully available to third parties, and there are currently plugins to expand this support to include Machine.Specifications (MSpec), xUnit.net and support for Silverlight projects. More can be added at any time, through the use of the SDK.

The MSpec project is a “Context/Specification” testing framework, which is a form of BDD tool, focussed more on driving correct behaviour than on testing functionality – building the right thing, rather than building the thing right. It’s a significant change of style to anyone used to the NUnit/mstest style of attributed methods, but it’s not too difficult to follow – test methods are replaced with fields that take lambdas that arrange, act and assert. It’s worth checking out the example projects in the source repository – BankingSpecs is a good place to start.

Machine.Specifications ships as a NuGet package, which also includes the ReSharper runner. Simply run the InstallReSharperRunner batch file in your packages\Machine.Specifications\tools folder (it includes support for ReSharper 6 and 7). Once installed, ReSharper now recognises MSpec tests in your source code, adding the unit test icons to the gutter in the editor:

mspec test recognised in the editor

One of MSpec’s nice features it that the names of the test and the information from the SubjectAttribute are used to generate a natural language description, which, when used with the different grouping options in the test runner can give very readable test results:

mspec results with natural language descriptions grouped by category

The xUnit.net framework is a more traditional unit testing framework, similar to NUnit and mstest, but arguably more idiomatic .Net. For example, it still uses attributes to mark methods as tests, but uses constructors and IDisposable instead of SetUp and TearDown attributes. Support is added by the xunitcontrib plugin, giving ReSharper the ability to locate and run xUnit.net tests. Like NUnit and mstest, it can handle parameterised row tests, dynamically adding rows into the results as the tests are run.

xUnit.net parameterised tests, using random data

It also works with the “Show Unit Test Usages” option in the filter drop down of the Find Results window. When viewing the results of a find usage search, any usage in a test method or test class is marked with a test icon, rather than an icon representing a method, class or property. Unchecking the “Show Unit Test Usages” option in the “Filter Usages” drop down will hide the test usages in the results, allowing you to focus on the usages in the production codebase.

Filtering xUnit.net test usages out of Find Usages results

Unit test providers don’t need to be source based. Most of them are, and display the test icon in the editor gutter, but they also examine the metadata of the compiled assembly, and can potentially find more tests here. Using this technique, it’s possible to run tests created in a source file that ReSharper doesn’t support. For example, we can create an xUnit.net test in F#, and while it’s not detected in the editor, it is still found and run when the assembly is compiled.

xUnit.net test in F#

Since the test hasn’t come from source code, you can’t double click the test in the test results to navigate to it. But a failing test will display a clickable stack trace that will navigate to the failure.

F# test results showing clickable call stack

This technique can be used to run F# tests using other F# test frameworks, as long as they are based on a unit test framework that ReSharper supports (and this technique can also be used to write a plugin that supports a framework that isn’t based on one of these frameworks).

Silverlight support is provided by AgUnit, which enables running and debugging your unit tests in Silverlight projects, and of course handles the Silverlight test framework’s AsynchronousAttribute.

AgUnit debugging an asynchronous Silverlight test

Perhaps surprisingly, a couple of our own products are also plugins for ReSharper’s unit test runner – dotTrace and dotCover. Although the products register themselves with ReSharper somewhat differently to normal plugins, these products use the same open APIs that the third party plugins use, albeit for a slightly different purpose. Instead of examining source code and metadata to find and run tests, they provide the environment in which tests are run; dotTrace collects profiling information, and dotCover collects coverage information. Both products extend the UI to provide means of running the tests, but dotCover also adds an output pane to the test runner to display the coverage information (and highlights covered and uncovered code in the editor), and of course, this works with third party test providers.

Unit test runner showing dotCover custom code coverage results pane

And, just to bring the extensibility full circle, dotCover includes ReSharper’s unit test runner, which is used if ReSharper isn’t installed. Again, NUnit and mstest support is provided out of the box, and there are separate dotCover plugins for xUnit.net and MSpec support (the dotCover MSpec runner is also distributed in the NuGet package, just use the InstallDotCoverRunner batch file in the tools folder).

ReSharper provides comprehensive unit testing support, in an environment that is very flexible. There is a lot of great support, provided in the box, and just a download away. And it’s all open and extensible – if you don’t see something you want, grab the SDK and get coding!

Unit Testing Improvements in ReSharper 7

Friday, July 27th, 2012

The ReSharper unit test runner has become an indispensable part of ReSharper. The ability to run and debug tests from a location in code, together with an integrated unit test runner make testing with ReSharper a streamlined and pleasant experience, and a rich variety of plug-ins supporting many popular unit test frameworks makes ReSharper relevant whatever unit testing framework you choose.

Here is an overview of some of the new unit testing features that we are shipping in ReSharper 7.

Jasmine Support

The previous version of ReSharper has shipped with support for the QUnit JavaScript test framework. In this release, we have added support for Jasmine — a behavior-driven development framework for testing JavaScript code:

Naturally, just like any other type of unit tests, Jasmine tests are presented in the ReSharper unit test runner:

PhantomJS Test Runner

In addition to being able to run QUnit and Jasmine tests in a browser, ReSharper 7 offers an alternative mechanism: one can use PhantomJS – a command-line interface to WebKit – to run JavaScript unit tests instead. To start using it, download and unpack PhantomJS, then open up ReSharper options (the Unit Testing|JavaScript Tests tab) and specify the install location:

Now, any all JavaScript tests will be executed in a PhantomJS environment. Since a full browser is not needed for the tests, ReSharper will not open up a summary page inside a web browser.

NUnit Improvements

The new version of ReSharper comes with support for the TestCaseSource attribute. This attribute lets test authors define external variables that store test data. For example:

Since the argument to the attribute references a variable by name, ReSharper comes with appropriate inspections…

…as well as code completion:

As of this release, ReSharper supports NUnit version 2.6 by default.

Support for Other Frameworks

If you are looking for a test framework that isn’t supported by ReSharper out of the box, don’t worry – many plugin authors have already prepared the corresponding ReSharper plug-ins. Frameworks supported by ReSharper plugins today include AgUnit, Gallio/MbUnit, MSpec, and XUnit.

That’s it — happy coding! (And testing, of course.)

dotCover 2.0 is Released

Monday, June 18th, 2012

JetBrains developers and QA engineers are finally going to have some sleep since dotCover 2.0 RTM is available right now for you to download.

Here’s the list of features that make this a worthy upgrade:

  • Support for Visual Studio 2012 RC along with Visual Studio 2005, 2008, and 2010. As soon as Visual Studio 2012 RTM comes out, we’re going to release a minor update to support it.
  • Integrated unit test runner. You don’t have to have ReSharper installed to execute unit test coverage analysis anymore: dotCover 2.0 bundles the same unit test runner that ReSharper provides. However if you do have ReSharper installed, dotCover will use ReSharper’s implementation of the unit test runner by default. Similar to its older brother, dotCover’s unit test runner supports NUnit and MSTest by default. For xUnit and MSpec, there are up-to-date plug-ins available right from the start at dotCover download page.
  • dotCover 2.0 adds attribute filters and a redesigned filter management UI. Attribute filters are there to help you exclude entities marked with certain attributes (for example, ObsoleteAttribute or TestFixtureAttribute) from coverage analysis.
  • Locate in coverage tree: a new shortcut to navigate from the caret position in Visual Studio text editor to its matching node in the coverage tree.

For a bit more detail on the new features, please read What’s New in dotCover 2.0, and be quick to download the new release!

Note that if your dotCover maintenance period or upgrade subscription is still valid, this would be a free upgrade for you!

dotCover 2.0 Early Access Open

Thursday, April 5th, 2012

dotCover 1.x is all but history. Starting this week, you can download and play with dotCover 2.0 early builds.

Why would you want to do that? Here’s why:

  • dotCover 2.0 can be installed into 4 versions of Visual Studio, including Visual Studio 11 Beta (with Visual Studio 11 RTM support to follow as soon as it’s available.)
  • dotCover 2.0 bundles a unit test runner. This means, even if a developer doesn’t have ReSharper installed for whatever reason, he/she can still use dotCover for unit test coverage — and of course, for simply running unit tests as well. That said, if dotCover is installed into Visual Studio that also has ReSharper, ReSharper’s implementation of unit test runner is used by default (which you can change at any time.) At this point, dotCover’s own unit test runner supports NUnit and MSTest. If you’re using MSpec or xUnit, you still have to use the ReSharper implementation. However, we’ll be working with plug-in authors to ensure compatibility with dotCover as well.
  • dotCover 2.0 extends filtering capabilities with attribute filters. This enables you to exclude entities marked with certain attributes from coverage analysis. Attribute filters are specified via dotCover | Edit Attribute Filters as fully qualified type names or masks. This could be useful for filtering out test fixtures or obsolete code.

One more feature that is not yet available but is being worked on is the Hot Spots view, which is designed to provide a list of most risky methods in your solution, in terms of high cyclomatic complexity and low unit test coverage. Stay tuned for more details when we’re done with the feature.

Meanwhile, download dotCover 2.0 early builds. If anything goes wrong, please don’t forget to file bug reports in dotCover issue tracker.

New Features in ReSharper 6 Unit Test Runner

Tuesday, August 23rd, 2011

Previous posts introducing ReSharper 6:

ReSharper 6 includes a series of new features and improvements in the test runner, as outlined below.

NUnit TestCase

We now fully support NUnit’s TestCase, TestCaseSource, Values, Ranges and other attributes. Given the following tests:

image

ReSharper outputs the results as individual tests, appropriately grouped.

image

This allows us to now run and view tests individually.

image

As expected, dotCover and dotTrace also provide support for this.

Support for Coded UI Tests with MSTest

Coded UI tests (when appropriate) are also supported now with the ReSharper test runner

image

Default Keyboard Bindings and Test Re-Runs

Out of the box ReSharper now ships with pre-defined shortcut keys for running, debugging, re-running tests, among other things. This eases the pain of having to map keys on new installations (as of ReSharper 6, you no longer loose your key bindings during upgrades!). By default, the key-bindings for the schemes are:

Operation Visual Studio IntelliJ IDEA
Run Unit Tests. Context Sensitive Ctrl+U, R Ctrl+T, R
Debug Unit Tests. Context Sensitive Ctrl+U, D Ctrl+T, D
Run ALL Unit Tests Ctrl+U, L Ctrl+T, L
Run Current Session Ctrl+U, Y Ctrl+T, Y
Rerun last test. Context Insensitiveimage Ctrl+U, U Ctrl+T, T

This last shortcut is a new feature added in 6 which allows us to re-run the last test we just ran. And we can run it from anywhere! This makes TDD for instance very easy as we can write some code, be in the actual code and run the test, without having to switch back to the test or open up the unit text explorer.

Running Tests from File Structure and Find Results Windows

You can now run Tests directly from the File Structure Window

SNAGHTML14c0e0a

as well as the Find Results Window

SNAGHTML14d0b1d

again, removing some of the friction when it comes to testing.

Categories

Certain Unit Testing frameworks provide test categories. For example, in NUnit there is the [Category] attribute. ReSharper now allows us to ignore specific categories and not run them during a test run. This is accomplished by indicating the category under ReSharper | Options | Unit Testing

image

When running unit tests, these appear as ignored.

image

This is useful for instance when we want to avoid running certain tests during development sessions.

Categories can also be used to run a specific group of tests by selecting these from the context menus:

image

Other improvements

Some other minor improvement or previous features we have blogged about include:

  • QUnit Support
  • Unit testing Sessions are now persisted between Visual Studio restarts
  • Ability to run tests in parallel from different assemblies, decreasing testing time (can be turned off in ReSharper | Options | Unit Testing)
  • General performance improvements
  • Update Progress indicator on Windows task bar

image

As always, we welcome your feedback

Machine.Specifications for ReSharper 6 now available

Wednesday, July 20th, 2011

Many, including myself have been eagerly awaiting the release of the ReSharper Runner of Machine.Specification (MSpec) for ReSharper 6. I’m glad to announce that this is now available, which is the result of a team-effort between Victor Kropp, Chris Nicola, myself and of course Alexander Gross, long-time contributor and main maintainer of the MSpec project.

NuGet packages have already been updated to support ReSharper 6. The source code can also be obtained via SymbolSource.org if using NuGet or directly from Alexander’s GitHub account.

Installing the Runner in ReSharper 6

If you’re new to MSpec or have simply forgotten how to install the runner under ReSharper, here are some simple steps to use Mspec in your project and integrate it with ReSharper:

1. Run the command install-package Machine.Specifications (or Machine.Specifications-Signed if you want the signed version). If this is an existing project, issue the command update-package instead of install-package.

2. Inside the packages folder corresponding to Machine.Specifications, a tools folder is created with a series of assemblies and some batch files. There’s a series of batch files named InstallResharperRunner.X.X. – VS20XX.bat where X’s correspond to the version of ReSharper and of Visual Studio. By executing the corresponding one, it will copy the assemblies to the ReSharper plug-in folder.

You can of course do the second step manually, by copying the Machine.Specifications.ReSharperRunner.X.Y files to the ReSharper Plugins folder, located either under the ReSharper bin folder in %Program Files%\JetBrains\ReSharperX\bin (for all users) or under %APPDATA%\JetBrains\ReSharperX\bin (user-specific settings).

With that you should be up and running.

Enjoy.

ReSharper 6 Introduces Support for JavaScript Unit Testing with QUnit

Friday, March 25th, 2011

Previous posts introducing ReSharper 6:

In ReShaper 6 we have taken the first step to make it easier for developers to work with JavaScript. By providing support for analysis, refactoring and code-navigation, we try and remove some of the difficulties developers might encounter when working with this language. Now we’re taking the next step!

Running QUnit from ReSharper

When it comes to Unit Testing, even some of the more experienced JavaScript developers tend to not pay too much attention to testing their code. Reasons for this include too much friction, lack of understanding on how to test, what framework to use, etc. Although we can’t solve all of the problems, we are going to try and remove a few of them by providing support for running JavaScript Unit Tests in an easier way with ReSharper. And we are starting this by providing support for QUnit, the unit testing framework used by jQuery. You will be able to run QUnit JavaScript tests using the same interfaces you know and love in ReSharper. And the best part of it all, is that it’s really easy! Let’s see how.

1. Define the code under test

First step is to have some code that we want to test. This code can be located in the same file where are tests are (not recommended) or in different files. In our case, we want to test some simple calculator operations:

We have placed this code in a file called CodeUnderTest.js but it can be named absolutely anything.

2. Write some tests

Writing Unit Tests in QUnit is pretty straightforward. Whereas with most test frameworks, you’d normally write a class that contains test methods, with QUnit, you call a function test which takes two parameters, a description of the test and the actual code to test, which is passed in as a function:

We place this code in a file called CalculatorTests.js (again, can be named anything). Our project structure now looks like this:

Since our tests are located in a file different to that of our tests, we do need to tell ReSharper which code we are referring to by using a specific reference notation at the top of the test file:

As soon as we add the tests, ReSharper will detect that these are QUnit tests and provide us with the ability to run them by placing the Test Runner menu in the left gutter:

providing us with the ability to run the tests:

We can now run individual tests and get the output displayed in the ReSharper test runner:

Notice how the tests are grouped under “Calculator tests”. This is done using the module function which allows grouping of tests:

When the tests are run, ReSharper launches the default system browser which runs the tests and reports the information back to ReSharper. Normally, to run QUnit tests, you would need to create an HTML file that runs the them. This is handled internally by ReSharper, so it’s one less step to perform.

If you prefer to use a specific browser to run the tests, you can do so by changing the browser under the ReSharper Options | Tools | Qunit

There is still more to come…

Currently, ReSharper is bundling the latest available version of QUnit (at the time of building). We will be adding support in upcoming releases which will allow you to use your own version of QUnit. Also, we will be looking at providing access to the HTML file that ReSharper generates, allowing better support for Continuous Integration environments, as well as the possibility of running the tests from the browser directly.

When can you play with it?

Right now! The public EAP build includes this functionality. Download it from here the ReSharper 6 Early Access page.

And as always, feedback welcome. Feel free to provide comments here or using our issue tracking system where you’ll not only be able to monitor the status of a feature but also get people to vote for it.

Advanced scenarios with dotCover Console Runner

Thursday, August 5th, 2010

In an earlier post, we saw how to use dotCover in simple scenarios, whereby we have a single unit test project for which we want to obtain coverage reports on. There are times however, when this is not the case; that is, we might have multiple tests projects. It is very common (and recommended practice) to separate Unit test projects from Integration test projects, allowing us to run the former more frequently during development and the latter during an Automated Continuous Integration setup. In these cases, running the dotCover analyse command, as shown in the previous post might not be ideal.

Fine tuning the coverage steps

For the scenarios described, whereas we have multiple test projects, dotCover allows us to fine-tune the coverage process by offering us a series of commands that breakdown the coverage, merging and reporting into individual steps. Suppose we have the following Project Layout

image

The AppCode is our application code that we want to run coverage for, and TestProject1 and TestProject2 are specific the test projects that cover AppCode. These can be Unit Test or Integration Test projects. The process we need to follow is shown in the figure below:

image

We run coverage on each of the tests projects by using the cover (c) command using the following configuration file (corresponds to testProject1.xml. testProject2.xml is similar)

<?xml version=”1.0” encoding=”utf-8“?>
<CoverageParams>
<Executable>
X:\Libraries\nUnit\nunit-console.exe
</Executable>
<Arguments>TestProject1.dll</Arguments>
<WorkingDir>TestProject1\bin\debug</WorkingDir>
<Output>output1.xml</Output>
<Filters>
<IncludeFilters>
<FilterEntry>
<ModuleMask>AppCode</ModuleMask>
<ClassMask>*</ClassMask>
<FunctionMask>*</FunctionMask>
</FilterEntry>
</IncludeFilters>
<ExcludeFilters>
</ExcludeFilters>
</Filters>
</CoverageParams>

[Command to execute: dotcover c testProject1.xml]

Executing this command generates a snapshot file with coverage data, the Coverage Results Descriptor (by default, unless specified in the configuration file using the TempDir element, the snapshots are saved in the User temporary folder). These snapshots are not the final XML report that we are after but intermediate information that dotCover uses to generate reports from. There is a correspondence between snapshots and processes, such that each process generates a single snapshot. In our case, since we are calling a unit testing framework to launch a single test project, there will only be one snapshot. However there are cases, for instance when running coverage report on an application, whereby the application can launch another process  itself. In this case, multiple snapshots will be generated.

When we run this command, once for each Project, we end up with two snapshot files. We now need to merge the snapshots in order to generate the XML report. This is accomplished using the merge (m) command

<?xml version=”1.0” encoding=”utf-8“?>
<MergeParams>
<Source>
<string>output1.xml</string>
<string>output2.xml</string>
</Source>
<Output>output_merged.xml</Output>
</MergeParams>

[Command to execute: dotcover m merge.xml]

which takes the output of the two cover commands (output1.xml and output2.xml) and generates a single snapshot file. The only thing remaining is to generate an XML coverage report based off of this data, using the report command

<?xml version=”1.0” encoding=”utf-8“?>
<ReportParams>
<Source>output_merged.xml</Source>
<Output>results_merged.xml</Output>
</ReportParams>

[Command to execute: dotcover r report.xml]

providing us with the XML report containing coverage information (percentage of code covered, statements covered), which like in the previous post we can apply an XSLT to format into HTML or JSON.

Other Coverage Commands

dotCover provides two more commands:

  • list
  • delete

The list command simply obtains a list of all snapshot files given the coverage descriptors, that is, the output of the different cover commands. The configuration file lists these descriptors:

<?xml version=”1.0” encoding=”utf-8“?>
<ListParams>
<Source><!– Required. At least one child element expected. –>
<string><!– Coverage results descriptor 1 file name –></string>
<string><!– Coverage results descriptor 2 file name –></string>
<string><!– Coverage results descriptor N file name –></string>
</Source>
<Output><!– Required. Resulting file name. Stores plain list of all snapshot files. –></Output>
</ListParams>

The delete command deletes the list of given snapshots:

<?xml version=”1.0” encoding=”utf-8“?>
<DeleteParams>
<Source><!– Required. At least one child element expected. –>
<string><!– Coverage results descriptor 1 file name –></string>
<string><!– Coverage results descriptor 2 file name –></string>
<string><!– Coverage results descriptor N file name –></string>
</Source>
</DeleteParams>

Summary

As we can see, dotCover allows us to fine-tune the coverage process by letting us control the different steps. One question that might arise is why couldn’t we have done the same by running the analyse command twice, once for each Test Project. If we were to do that, we would get two independent reports, whereas here the results are all merged into a single report. The analyse command is a great fit for simple scenarios that only have a single project. Once we move onto more complex setups, having a finer control over the coverage process has its benefits.

Running Code Coverage from the Console with dotCover

Wednesday, July 28th, 2010

As of the beta* of dotCover, we included a Console runner to run coverage using the command line, allowing for instance, setup of dotCover in a Continuous Integration environment. Let’s see how it works.

*To get all these features in this post, you need to download the latest nightly build

 

Console Runner

The Console runner is located under the installation folder (%Program Files%\JetBrains\dotCover\v1.0\bin\dotCover.exe). The best option is to add it to the system path so as to be able to run it from anywhere. The runner accepts several commands based on the operation we want to perform. Each of these commands in turn takes one parameter, which is an XML configuration file. The commands are:

  • cover: Coverage of specified application
  • merge: Merges snapshots
  • report: Creates an XML report of the coverage results
  • list: Lists snapshots
  • delete: Deletes snapshots
  • analyse: Provides an all-in-one analysis and output

We can also obtain a list of commands by typing dotCover.exe help on the command line:

image

As shown in the figure above, we can find out more about each command by typing help followed by the command. All commands (including help) have corresponding shortcuts. For instance, to get information about analyse, we can type:

dotCover.exe help analyse

or

dotCover.exe h a

Since all commands take as parameter an XML file, when requesting help, all they do is generate a sample XML with comments indicating what each element means. The previous command would therefore output:

image

The obvious advantage to this is that we can easily get a new configuration file setup by just asking for help and piping the output to some file. In fact, adding a third filename parameter, dotCover will do this for us, without us having to touch up the output (remove headers, etc.).

Coverage in one command

The dotCover console runner is flexible in that it complies with most common requirements in a Continuous Integration setup. Normally in projects, we have multiple test projects we need to cover. We might then want to see the results separately or merge them as one report. dotCover allows for this kind of flexibility. However, in many cases, all we want to do is run coverage on a single test project and see the results. We’ll see how to do that in a single step in this blog post and in a future post we’ll get into the details or merging, listing, etc.

Let’s see how to do this step by step. I’ll be using a sample app that I’ve prepared for this blog post (using MSpec). You can adjust the parameters to your own project paths and unit testing framework as required.

 

  1. Type dotcover help analyse coverage.xml on the command line. I’m doing this in the root folder of my solution.

    image

  2. Add coverage.xml to the Solution and open it up in Visual Studio
  3. Eliminate all elements except the following

     image

  4. Fill in the elements with the corresponding values:

    Executable: Is the path to the unit testing framework runner (in this case mspec.exe)
    Arguments: Arguments passed to the runner, the test assemblies (specs.dll)
    WorkingDir: Optional path to the working folder (provide this to not have to fully qualify test assemblies)
    Output: XML Report containing coverage information. Relative to the coverage.xml path (output.xml)

     image

  5. That’s it in terms of configuration. Now all we need to do is run it by typing dotCover analyse coverage.xml. If all is configured correctly, we should see the following output:

    image

    and there should be an output.xml file located in the same folder. Opening it up we can see the coverage results

    image

 

Filtering from the Console

Looking at the results, we can see that there is a lot of noise, that is, assemblies that we’re not interested in such as the tests, MSpec assemblies, etc. In a previous blog post, we saw how to filter these out using dotCover GUI. Let’s see how we can do this via the command line. If we generate the default coverage configuration, we can see that there is a filter section:

image

These are used to filter assemblies. We can indicate which assemblies we want covered and which we don’t. In order to filter out all assemblies and concentrate only on our actual code (ClassLib), we can do one of two things:

  1. Filter out all non-required assemblies individually by adding them to the ExcludeFilters section, as in:

     image

  2. Exclude everything that’s not specifically in the Include section. This is done by leaving an ExcludeFilter section blank and placing what we want covered in the IncludeFilter:

    image

Based on the needs and depending on which of the two lists (Include or Exclude) are smaller, we can opt for option 1 or 2. The ouput should now be filtered:

image

 

Giving it some style

Only thing left to do is to format the output so that it can be viewed nicely inside a browser. This can be done easily using XSLT and some appropriate design skills (something I lack).

image 

Included in the project you have an XSLT to create some HTML as well as one to covert the XML to JSON. Having JSON output allows us to combine it with something like jQuery to display a nice treeview or grid with the results.