Incremental testing with TeamCity
Tuesday, March 13th, 2012When you build a large project, it is often important that previously built components that are still up-to-date are not rebuilt. Otherwise, if all targets are built every time, each build will take a long time to complete.
The longer it takes to make a build, the later you get feedback on your changes, the more difficult it is to setup reliable and meaningful Continuous Integration process.
And the faster your builds are, the higher the rate at which your changes are integrated, the less is the feedback time, and the more value you get from your Continuous Integration.
TeamCity 7.0 introduces a new feature: support for incremental builds for IntelliJ IDEA, Gradle and Maven projects, primarily aimed at incremental testing.
Overview
Most of the changes developers make to the source code don’t affect the whole system. The better components and their dependencies are organized within the project, the more localized the impact of a change will be.
Incremental builds are all about re-building only those parts of project which have been changed or affected by the changes, leaving the rest intact. It’s the same good old concept of incremental compilation based on a file timestamp, taken one step further to affect all other build stages, including packaging, automated testing, etc. Especially testing, because automated tests may take really long time.
In most of the big projects with large amount of tests, building everything each time we change something could be a huge waste of time. Granted, from time to time we do need full builds, as we do need full re-compile. That’s because incremental builds are less reliable, less informative, since they show only a part of the whole picture, and they may accumulate errors. Still, in most cases, full builds are not required, and incremental builds can be real time saver for the cost of a bit lesser reliability. That’s especially true for well-organized project, where tests are properly distributed among modules.
Incremental Builds in TeamCity
TeamCity knows about a change when the file changes in the version control. After all changes are collected for the build, TeamCity calculates which parts of the project are affected by the changes based on the module dependencies.
TeamCity uses modules as a level of granularity for making incremental builds. Why did we choose modules? Here are just some of the few reasons:
- Modules provide easy to analyse project structure.
- Modules are the first class citizens in most of the modern build systems and IDEs (not in all of them they’re called ‘modules’ though).
- Modules have clearly defined dependencies, in contrast to those between language entities, some of which can be revealed only at the runtime (like interface implementations created by reflection).
- As a result of 3., module dependency graph can be easily extracted from project files and analyzed without understanding the language semantics.
Now that we have a module dependency graph, all we need is to map the changed file to a module. This is an easy task, given that the module structure is (usually) unambiguously reflected in the project file structure.
Theoretic example
Lets consider a project consisting of eight modules with dependencies between them shown on the picture below.
When we detect change in file “src/x.java“, we can easily identify that it belongs to module X. Based on this knowledge, and the knowledge of module dependencies, we can quickly discover all other modules affected by the changes, which are in our case: C, D and (transitively) E. We can also assume that modules A, B, F and G stay unaffected.
If we only re-build modules X, C, D, and E now, the odds are high that we’ll get the very same result as with re-building the whole project from scratch, but with much less time spent.
TeamCity is primarily focused on making incremental testing possible. And that is for the reason: from our own experience, and in most of our clients’ installations, automated testing is usually the longest part of a build. Nevertheless, same approach may apply for other build stages. For instance, with Maven incremental builds, you only deploy to remote repository SNAPSHOT artifacts, affected by the change.
Real example with Spring Framework
To demonstrate how TeamCity users can benefit form incremental building we took a well known open source project Spring Framework (http://www.springsource.org/), which builds with Gradle, as an example.
Let’s consider a small source code modification. In our case, we changed the ReflectionTestUtils.java in spring-test module, which has two dependent modules: spring-struts and spring-aspects.
The full build of Spring Framework upon our change runs over 10361 tests, and lasts for 4 min. 53 sec.
If we enable TeamCity incremental builds, we get the following results: only 805 tests are run, and the build takes 1 min. 12 sec.
What has happened?
For the full build, we have a usual Build Configuration set up within TeamCity. Each next run of this Build Configuration results in execution of the full set of tests no matter how big was the change since the previous build. This is generally not a bad thing except that it may take very long time to perform such builds.
As a result, over 10000 tests are run, almost 5 minutes are spent, and the build is RED: 16 test runs have failed.
Now let’s copy this Build Configuration and make only one modification in the runner settings – enable incremental building:
In case of Gradle, this setting tells TeamCity to execute Gradle’s buildDependents task for each module directly affected by the change. buildDependents is a magic Gradle standard task, which in a turn executes build task for the given module, and all the dependent modules, both directly and transitively.
With this new Build Configuration, only 805 tests were run, and the build took a little over 1 minute. Note that this time the build is GREEN. That’s because only those tests were executed, which are relevant to our change. This GREEN gives us clear indication that our change was OK, without cluttering with the rest of tests.
Was it worth the hustle?
For those thinking 3.5 minutes not a big deal I want to show a picture from our real life:
This is an excerpt from the build history of an incremental IntelliJ IDEA build configuration in Faradi (TeamCity 7.0 code name).
The build marked with red underline is a full build (because of a change in a core component), and it lasted for 2h:48m.
The build marked with green underline is an incremental build, which only contains small local changes, and lasted for less than 20 minutes.
That is a considerable difference (almost ten times!). Although most of the builds in this picture are not that quick, as their changes are often big and distributed, they still take much less than 2.5 hour to build, providing huge time saving.
Incremental personal builds
As was demonstrated before, incremental builds can save you a lot of time. Still, the biggest benefit incremental builds provide being combined with TeamCity personal builds.
For a personal build only your personal changes are taken into account. All required dependencies are built upon concurrently made changes, but tests will be run only for yours.
The screenshot below shows the results of my personal build with a trivial change in maven-runner module (don’t be surprised with the duration — those tests are quite slow). Regardless of the fact that one more change in completely different module (from Kirill Maximov) was included into the same build, only tests affected by my own personal changes were run.
Even if Kirill’s change breaks the full build, my personal build is still marked as GREEN. So, no cluttering with the changes of others.
When to use incremental builds
Incremental builds aren’t always good and helpful. Below is some general advice.
Incremental builds ARE NOT generally good for:
- Release and deployment builds, where you have to make sure that the build is clean, and that you see the whole picture in the build results.
- Less frequent tasks such as nightly builds, code inspections, heavy integration tests.
- Projects with poor componentization.
Incremental builds ARE good for:
- Build configurations supposed to be used for unit-testing of developers’ commits.
- Personal builds.
Individual build tools specifics
TeamCity 7.0 supports incremental testing for IntelliJ IDEA, Maven and Gradle runners. And yes, we’re planning to enhance this list in future versions by including .NET runners.
There are some individual specifics of incremental building in different runners forced by underlying build system limitations.
IntelliJ IDEA Projects
At this moment IntelliJ IDEA project runner only supports running tests incrementally, and doesn’t support incremental compilation or any other build stages. It is still great because we feel running test is the most time consuming build activity in majority of software projects.
Gradle
Gradle can incrementally run any task, which is great. The problem is that it only calculates local file system changes, which is not usable for the distributed architecture of TeamCity. So the current implementation is limited to only executing Gradle’s buildDependents task.
This is still enough for most cases, because, as I mentioned earlier, this task executes the build task for every affected module. The build task in its turn executes a full set of build activities for a module.
Additionally, you can use standard Gradle ‘-x‘ command line parameter to exclude an arbitrary set of tasks. This is particularly helpful when you want to only run tests.
Maven
Maven is able to run any build activity (goal) for selected modules, but requires splitting a build into two phases with an intermediate installation of artifacts required by affected modules into the local repository. This brings in some overhead, making the incremental building feature not very usable (in terms of speed) for projects with small number of tests. If you want to know why Maven runner needs this you’re welcome to read a detailed post about Maven incremental building specifics that will follow shortly.
To be continued…
