Since version 7.0 TeamCity offers native support for module-based incremental building possible with some build system like Maven and Gradle. Thanks to the ability to track developers’ changes in the project files, TeamCity is now able to build and test only those parts of systems affected by the changes.
The overview of this feature is available here, which is recommended for reading by those not familiar with the concept of incremental building.
This article describes in detail the technical aspects and difficulties of implementing incremental building in TeamCity Maven runner, and can be useful for TeamCity users extensively working with Maven.
Maven own abilities
Maven has some limited support for incremental builds. In Maven architecture all build procedures (compilation, testing, code generation, etc.) are performed by plugins. Plugins are also responsible for making decisions whether results of their previous work are up to date and can be used in the current execution. At the moment of writing this I know only one Maven plugin that actually performs such analysis – Maven compile plugin. Before compiling a .java file it looks into the target directory and checks if the corresponding .class file exists and its modification time is greater than the modification time of the source file. If it’s true, the source file is considered not requiring recompilation.
Apart from problems with this trivial approach (it’s enough to mention that Maven compiler plugin doesn’t detect deletion of source files keeping the obsolete class files in place) incremental compilation is definitely not enough. I don’t think it would be too far from truth if I say that in most projects the longest build phase is running tests, not compilation.
Maven Surefire plugin (which is responsible for running tests by default in the Maven ecosystem) doesn’t provide any file-level incrementality similar to the compiler plugin.
But how about the module based dependency analysis? In the theoretic example from the previous post we want to execute tests for modules X, C, D, E. Can Maven employ the technique described above to achieve the desired effect? The answer is — yes, to some extent.
For Maven 2.x prior to 2.1.0 this can be done using the Maven reactor plugin. In later Maven 2 and Maven 3 releases this mechanism is embedded into the core. Without digging into the details, the essence of this feature is that you provide the list of modules of interest in the command line and depending on additional switches Maven builds them alone, with upstream modules, or with downstream modules.
Maven 3 command line would look like this:
mvn test --projects X --also-make-dependents
or like this with short key versions:
mvn test -pl X -amd
This will execute tests in modules X, C, D, E sequentially going down the dependency graph starting from module X. (One important note: to have this build succeed one need all the dependencies of X — A and B — to be installed into the local repository or deployed into a remote repository. Otherwise Maven won’t resolve these dependencies, since they won’t be included in build reactor)
So it looks like all TeamCity has to do is translate a build change list into a module list and run mvn -pl --amd, right? Not really. There are some problems making things not that simple.
The first problem is that when analyzing dependency graph Maven doesn’t take into account the parent relationship, which is a kind of dependency. Changes in the parent pom indeed affect all child projects. But if you try to specify parent pom in the --project list you’ll get a build error “Couldn’t find specified project in module list”, because Maven allows placing there only modules producing artifacts.
The second problem is that Maven doesn’t distinguish dependency scopes. Is there a difference between a “compile” dependency and a “test” dependency in terms of change impact analysis? The first one is transitive, while the second one is not.
Let’s consider another example:
In this project there are 4 modules. Subsystem1 and Subsystem2 depend on CommonUtil, which provides a common use library with utility methods. CommonUtil contains tests, which use classes from MyTestFramework. This is not a production dependency. It doesn’t affect CommonUtil production code, so a change in MyTestFramework affects only tests in CommonUtil and does not Subsystem1 and Subsystem2. To test this change we need to run tests only in MyTestFramework and CommonUtil.
Maven doesn’t take this into account. If you put MyTestFramework in the --project list it will run tests in all 4 modules.
Yet another problem is also related to limiting the dependency scope. Assume that in the last example we’ve changed only tests in CommonUtil. Obviously, to test this change we need to run tests in CommonUtil only. Unfortunately we can’t tell Maven not to go down the dependency graph deeper for this module. One may argue that in this case we can simply omit the --also-make-dependents switch, and Maven will build only this module. But a real change list is usually a mix of test and production modifications, often in more than one module. So we cannot apply the --also-make-dependents switch to some modules in the list and not apply to the others. At least, we cannot do this in a single Maven execution.
The final issue isn’t related directly to Maven. It’s rather an intrinsic issue of the distributed architecture of TeamCity – it cannot guarantee that the results of the previous build are available for the current build, because the older build might have run on a different agent machine, and Maven rightly doesn’t care about it.
Let’s get back to the first example. TeamCity can schedule this build to a fresh agent on which this project has never been built. Since this build is incremental, Maven starts building modules X, C, D, E. But when building module X it immediately fails because modules A and B, which X depends on, simply haven’t been compiled.
This means that before running tests for some set of modules we should build their own dependencies and install them into a local repository. It effectively means running one goal for one module set and another goal for another module set in a single execution, which Maven with its current architecture is unable to do.
With all said above it becomes clear that TeamCity cannot rely on Maven’s --also-make-dependents feature. So, how does it work in TeamCity?
What TeamCity adds
First of all TeamCity implements its own change impact analysis algorithm for determining the set of affected modules, taking into account parent relationships and different dependency scopes. Second, TeamCity performs a special preliminary phase for making dependencies of the affected modules.
The build is split into two sequential Maven executions.
The first Maven execution called preparation phase is intended for building the dependencies of affected modules and installing them into the local repository. This ensures our snapshot dependencies are (1) available and (2) up-to-date. Thus the preparation goal is always “install”. But we don’t want tests to be executed for dependencies, which aren’t affected by changes, so the command line for the preparation phase is accompanied with the -DskipTests switch. The preparation phase is to assure there will be no compiler or other errors during the second execution caused by absence or inconsistency of dependency classes.
The second Maven execution called main phase is pretty clear. It executes the main goal “test” (in our case), thus performing only those tests affected by the change.
Effectively it’s transformed into the following Maven 3 commands:
mvn install --projects A,B -DskipTests
mvn test --projects X,C,D,E
For Maven 2.x the command lines looks like this:
mvn install -N -r -Dmaven.reactor.includes=A,B -DskipTests
mvn test -N -r -Dmaven.reactor.includes=A,B
Finally, after the main execution TeamCity removes the installed dependency artifacts from the local repository to avoid possible interference in future builds.
Enabling Maven incremental building
The best part of the whole story is how to enable incremental building for Maven in TeamCity, because it’s done with one click. Just turn on the “Build only projects affected by changes” check box in the Maven runner configuration UI:
Enjoy!
