Yet another bug-fix update is out – TeamCity 6.5.6. In this build we fixed issues related to Duplicates Finder (.NET) runner, once again improved general performance and addressed a couple of issues in TeamCity plugin for IntelliJ IDEA platform. You can find the complete list of changes in release notes.
Download the build and share your feedback with us.
Happy building!
TeamCity 7.0 EAP (Early Access Program) was recently opened and one of the new features is the built-in support for NuGet. I recently blogged about setting up TeamCity to pack and publish NuGet packages via a plug-in and this plug-in is now included by default in TeamCity 7. However, the real new interesting feature is that TeamCity is now a native NuGet repository too!
Many of those that have been using NuGet, have most likely been using it to consume packages from nuget.org where there are currently over 3800 unique packages, most of which are open source.
What happens however if for some reason or another you do not want to submit packages to nuget.org? For instance, think that you want to use NuGet to modularize and distribute code inside your own organization, or create libraries for private consumption. In this case, publishing to nuget.org does not make sense. This leaves you with basically two options:
Both of these options come with their own share of overhead. With the local share you now require sharing of folders and permissions. Setting up your own NuGet repository also requires managing permissions and whatnot separately. At the end of the day, its another service to manage.
Fortunately, you now have a third option: TeamCity. The same server that builds your projects, runs your tests, packs and publishes your packages can now also serve them. The best part of it is that it is so simple, that I had to take up the rest of this blog with the previous nonsense just to give it some meat.
I am not going to cover how to pack and publish packages in this post. All that is covered in detail in the previous post I wrote, so please read that first if you’re not familiar with the process. Enabling TeamCity as NuGet and making packages available consists of two steps:
Go to Administration | Server Configuration | NuGet tab
Click on the Enable button to enable it. The same screen with then display two different feeds: a public and a private one:
If by chance the Public Url is not available, you will probably see a message telling you that you need to enable the Guest account in TeamCity, which can be done from the General tab.
Since TeamCity itself is going to be a NuGet server, the step to publish a package is no longer required. However, packing the package is. In this step (NuGet Pack Build Type), we can just configure the output for the package to point to some specific folder, for instance packages
We need instruct TeamCity to ouput the results of this folder as artifacts. This is done in the General Settings step of the Build Configuration
and with that, we’re done. Next up is to configure Visual Studio to consume from this feed.
Although this step is optional, it is recommended to add your repositories to Visual Studio to avoid having to type long URL’s in each time you want to read from a specific package repository. To do this, click on Options | Library Package Manager | Package Manager Settings
We need to add a new NuGet Repository. I’ve called it Local TeamCity and the URL corresponds to the public URL provided to me by TeamCity in Step 1:
Notice that I have another entry which is Local TeamCity Auth which corresponds to the authenticated version.
Once we have this, we can now easily consume packages from our repository by merely specifying it in the Package Manager Console, either via the Combobox or explicitly in each call:
That’s all there is to it. By merely publishing our packages as artifacts, TeamCity now provides a full-fledged nuget server which opens up great possibilities when it comes to working and managing dependencies between projects. TeamCity is currently in EAP and much of what I’ve described here is in open to improvements. That is why your feedback is very important. Download 7 and start playing with it today. Let us know what you think.
Over the last few weeks we have made a great deal of improvements in .NET inspections support. As a result .NET Inspections (powered by JetBrains ReSharper) and FxCop inspections runners produce much more clear results.
In case you missed, some time ago we announced new .NET inspections runner based on ReSharper inspections engine, and we already received feedback on it. Thanks to our EAP users we were able to fix a number of serious problems:
- LINQ usages are marked as unresolved symbols;
- TW-18724 NUnit attributes usages (and all External Annotations);
- TW-18730 Usages of resources in WPF, Silverlight projects.
Now the updated plug-in is available for download: follow the installation instructions and share your feedback with us.
As for FxCop runner, it now has a new setting – ‘FxCop version’.
It is really useful in case you have several FxCop versions installed in your build farm: selecting the version of FxCop to use in the runner settings will allow to keep the results consistent. If you’re not sure which FxCop versions you have installed, just go to Agents > Parameters Report page and use ’system.FxCopCmdFileVersion’ parameter name. You’ll get something like this.
Updated plug-in is available to download too.
Note that both plugins are compatible with the latest TeamCity 7.0 EAP and are not intended to work with earlier TeamCity versions.
Enjoy!
TeamCity uses Service Messages to provide an easy API to allow for build scripts integration, i.e. the build providing status reports, artifacts publishing, test reporting, etc. A Service Message has the following format:
##teamcity[hello message='teamcity' version='6.5.x' comment='awesome']
Service Messages introduce a communication protocol for data exchanges between processes using streams, sockets, HTTP or pipes.
I’ve created a wrapper to allow for easier interaction with TeamCity Service Messages, providing it as a package available on NuGet.
The library contains two main classes. One for reading and one for writing service messages.
Reading service messages can be done using a TextReader or string with minimal memory requirements:
using JetBrains.TeamCity.ServiceMessages.Read;
...
var parser = new ServiceMessageParser();
using(TextReader reader = ... ) {
foreach(var message in parser.ParseServiceMessages(reader)) {
/// process IServiceMessage object
}
}
...
Parse method gives us an IEnumerable of IServiceMessage:
public interface IServiceMessage
{
[NotNull]
string Name { get; }
[CanBeNull]
string DefaultValue { get; }
IEnumerable<string> Keys { get; }
[CanBeNull]
string GetValue([NotNull] string key);
}
The Name is the name of service message, i.e. ‘name’ in ##teamcity[name'aaa'].
The DefaultValue contains simple value, i.e. ‘value’ for ##teamcity[name 'value'] or null for service message with attributes, i.e. ##teamcity[name tea='hot' ice='cold']. To access service message attributes use Keys property and GetValue method.
To create a service message you may use anonymous type for service message attributes:
using JetBrains.TeamCity.ServiceMessages.Write;
....
string serializedMessage
= new ServiceMessageFormatter().FormatMessage(
"rulez",
new {
Version = "3.2.2",
Mode = "zoom"
}));
/// returns: ##teamcity[rulez Version='3.2.2' Mode='zoom']
...
The previous is the simplest way for creating service messages, although you may use a strongly typed approach:
using JetBrains.TeamCity.ServiceMessages.Write;
....
string serializedMessage
= new ServiceMessageFormatter().FormatMessage(
"rulez",
new ServiceMessageProperty("Apple", "239"),
new ServiceMessageProperty("Pie", "42")
));
...
The source code for the project is available on GitHub and the build is at TeamCity.CodeBetter.Com and with TeamCity support for NuGet, it automatically publishes the library as NuGet package on NuGet.org. For a full description of the format and usages of Service Messages see the Build Script Integration with TeamCity article.
Previously described Fail build on metric change feature is not the only new way of controlling build status and detecting build failures. In TeamCity 7.0 we introduce one more useful feature that allows to mark build as failed, in case when certain line is met in build log.
Note, that you can instruct TeamCity to look for a line containing or matching a Java Regular Expression regexp text that appears/doesn’t appear during the build. When the condition is set, the interface looks like:
Here’s how a build failure looks when caused by such condition:
The feature is not completed yet, so your comments and feedback are very much welcome. To try it live, install TeamCity 7.0 EAP build, we recommend the next EAP build which is about to be released in the nearest future.
Enjoy!
When we released TeamCity 1.0 in October 2006, we had about 20 agents on local server. Since then TeamCity has come a long way, we’ve added a lot of features, and we have been increasing our agents pool. Now we have 110 agents connected, a tripled user base, bigger artifacts, significantly increased number of check-ins to version control systems monitored by TeamCity.
This is how our Agents page looks like during the typical working day now:
To support this number of agents in our installation we’ve made several configuration changes. In this blog post I would like to share some general recommendations how to achieve better performance and scalability with your own TeamCity installation.
First of all, switch your server to external database. We are using MySQL with InnoDB storage. TeamCity can also work with Oracle, MSSQL, PostgreSQL and Sybase. If you choose to use Oracle or MSSQL, it is better to install them on a separate server, as these databases usually require a lot of resources. MySQL and PostgreSQL are much more lightweight and can be installed on the same machine as TeamCity server but we recommend to use dedicated disk for the database files. If you are not sure what database to use, or you do not have experienced DBA who would help you, choose MySQL (with InnoDB storage), it’s fast, easy to install and use and most likely will suite your needs.
TeamCity is mostly I/O bound system. The more agents you have, the more builds will be executed concurrently, which means more build logs and artifacts are stored on disk, more data saved in the database.
To achieve the best performance install several disks to your server. At least separate OS and TeamCity binaries from TeamCity data directory (.BuildServer), i.e. place them on different disks. Moreover .BuildServer directory can be further split onto different disks too, as TeamCity can perform significant I/O activity there. In particular the following directories can be accessed frequently:
.BuildServer/system/caches – this directory holds search indexes and version control specific caches, in case of Git or Mercurial, TeamCity stores cloned repositories there.BuildServer/system/artifacts – this directory contains artifacts and is accessed each time when build publishes its artifacts or retrieves them via artifact dependencies.BuildServer/system/messages – if you have many agents, this is probably the most frequently used directory, as it holds build logs<TeamCity installation directory>/temp – this directory stores temporary files created by TeamCity, it is also used as a temporary directory for external processes started by the server, like hg, p4 and so onNote that artifacts, and messages directories can hold huge amount of data.
Given that, the following setup seems reasonable: use a large disk for the whole .BuildServer except caches directory and another medium size disk for caches and temp. If you really need faster access to artifacts, you can further separate artifacts and messages directories.
In our case we have 2×2Tb disks combined in RAID1 where we placed the whole .BuildServer directory except .BuildServer/system/caches and 400Gb disk for caches and temp.
We also have 2×145Gb SAS disks RAID1 for the MySQL database files.
Most likely you won’t need anything extraordinary here. Choose the best CPU in terms of its value for the money. Note that it also makes sense to have several CPUs, TeamCity is multi-threaded application and is able to leverage their power.
Our server has one Intel Xeon E5520 CPU (8M Cache, 2.26 GHz, 5.86 GT/s) and I would say it is enough for us. We also have MySQL database installed on the same server, so this CPU is shared between TeamCity and MySQL.
It always makes sense to have a server where you can add more memory easily. Also it makes sense to have 64-bit hardware, OS and 64-bit JVM. For example, in case of Windows, 32-bit JVM can’t allocate more than 1,2 – 1,3Gb of memory just because of OS limitations.
Unfortunately, if Java application is started by 64-bit version of JVM, it requires 20%-50% more memory. For example, if you ran TeamCity with 32-bit JVM and 1Gb Xmx, you need to increase this Xmx up to 1,5Gb with 64-bit JVM. This is the price for better scalability.
Let’s return to TeamCity memory usage. Probably the most common factors affecting memory usage are the length of the builds history and the number of tests per build. TeamCity caches test results to be able to perform fast calculations to find a build where the test failed for the first time, or where it was fixed. If you have relaxed cleanup policy and prefer to have many builds in the history and these builds have many tests (tens of thousands) expect memory usage higher than usual.
If you are using Git, you should also expect higher memory usage on the server side. Opening a Git repository, accessing objects stored in Git database may require a lot of memory. That was the reason why we moved some of the operations with Git outside of TeamCity Java process.
It may seem obvious, but if you have a big number of active users (hundreds) and many agents you may also see an increased memory footprint. The reason for that is that more active threads process user and agent requests concurrently, this means more memory is allocated on the threads stacks. Also with a big number of users you may need to increase the number of connections in the database pool. Plus this impacts memory too, as each connection holds various caches.
I would say that for production purposes, if TeamCity is running under 64-bit JVM, 2Gb Xmx should be enough for the start. If needed you’ll be able to increase it later. Since TeamCity 6.5 we have memory usage information in thread dumps taken from Diagnostics page. Memory information block is at the bottom of thread dump and looks like this:
Memory usage: Code Cache: used = 35.43Mb committed = 35.75Mb max = 48.0Mb used/max = 73.81% PS Eden Space: used = 542.27Mb committed = 1088.18Mb max = 1091.68Mb used/max = 49.67% PS Survivor Space: used = 18.39Mb committed = 38.0Mb max = 38.0Mb used/max = 48.42% PS Old Gen: used = 1036.67Mb committed = 1653.50Mb max = 2333.37Mb used/max = 44.43% PS Perm Gen: used = 122.66Mb committed = 202.12Mb max = 250.0Mb used/max = 49.07% Total: used = 1755.44Mb max = 3761.06Mb used/max = 46.67%
In general, total percentage of the used memory should be about 50%-60% most of the time. If during several hours it is closer to 80%-90%, consider increasing Xmx.
Our own server has 12Gb, of those 4Gb are dedicated to TeamCity server, and 2Gb are used by MySQL. The rest is mostly unused or holds OS file caches (Windows Server 2008 R2).
We are not using superfast hardware locally, as we prefer to see bottlenecks faster. We keep paying attention to overall server performance, and each release should be better or at least not worse in this aspect. If you are still using TeamCity 5 or even 4 and are experiencing performance issues, consider upgrade to version 6.5, in general it should work faster than previous versions. Do not hesitate to contact us if you have performance problems. If you choose to do so, try to gather as much details as possible: http://confluence.jetbrains.net/display/TCD6/Reporting+Issues#ReportingIssues-HangsandThreadDumps.
I hope these recommendations will be useful for those who need some estimations for the server hardware, and for those who would like to increase server performance, but do not know where to start.
One of TeamCity’s remarkable features is a possibility to analyze the code of your application, find duplicates, show corresponding reports and charts. You can calculate code coverage metrics, run code inspections, and see results right in the TeamCity’s UI. And in most cases, these features work out of the box, without setting up additional external plugins.
All these code examining tools generate various numeric metrics, and you can see the dynamic of their change in the TeamCity UI.
What you could not do before (well, almost could not) is to specify metric thresholds, which would fail the build upon exceeding them. For instance, you couldn’t tell TeamCity that the build must fail if code coverage or code duplicates number is worse than in the previous build.
Corresponding requests stayed for a while in our tracker and collected quite a few number of votes:
- TW-2539 Inspection Build: fail if more inspection errors than in last successful build
- TW-3040 Make build fail if code coverage is below certain threshold
- TW-3041 Make build fail if code coverage is below previous code coverage value
- TW-4953 Setting to fail a build when number of found code duplicates exceeds some threshold
- TW-15679 Option to fail a build if a specific statistics value is below/greater then in last successful/tagged build
When we started pondering over the task, it became obvious, that it can be generalized. And besides code quality, we could consider other metrics, such as number of tests in a build or volume of published artifacts:
- TW-12449 Option to fail a build if test number drops
- TW-5451 Option to fail a build when not all artifacts are published (artifacts validation)
Moreover, if a build publishes its own numeric metric (and it is very easy to set up), this metric can also be used to determine build status.
So now you can specify several metric-based build failure rules for a build configuration:
As you see, you can create rules for both the absolute metric values, and for the difference with a previous build in the same build configuration. As a previous build anchor, you can use:
- last finished build
- last successful build
- last pinned build
- build with a specified build number
- last finished build with given tag
The default list of available metrics is:
- artifact bytes (you can fail a build, if not all artifacts were published, basing on their volume)
- number of tests in a build
- number of ignored tests
- code inspections: errors, warnings
- code duplicates count
- code coverage percentage – lines, blocks, methods, classes
To add your custom metric to this list, you have to change TeamCity’s configuration file main-config.xml (<TeamCity data directory>/config/main-config.xml) and add the following section there:
<build-metrics>
<statisticValue key="my_file_count" description="number of files"/>
</build-metrics>
So, if your build will publish value my_file_count, you can use it as a criteria for a build failure.
When build failure metrics are set up, the interface will look like:
When a build fails because of the bad metric value, it looks like this:
The feature is not completed yet, so your comments and feedback are very much welcome. To try it live, install TeamCity 7.0 EAP build, which is about to be released in the nearest future.
We are glad to announce another TeamCity 6.5 EAP build. In this EAP you’ll find mute test failure feature, further improvements in tests grouping, parameters completion in the Administration area, new build runner and new build feature for .NET users.
With this EAP we bundle new version of Windows Tray notifier and new, pre-release version of dotCover 1.1. We also made significant improvements in the way how IntelliJ IDEA plugin works with server, for example, the network traffic produced by plugin significantly reduced, and the plugin itself loads much less data in the collapsed state. Also remote run trigger on branches now allows more flexible branch names.
See the full list of changes in the release notes, download the build and send us your feedback!
Happy building!
This isn’t a brand new feature, but one might not yet know that besides email, Jabber, rss, IDE and Windows tray notifications, you can configure TeamCity to sent notifications about builds status via Google Talk instead of Jabber.
The only thing you need to do is to set the following options on the Administration | Server Configuration | Jabber Notifier tab:
talk.google.comNote, that if you use Google Apps for domain, you should specify full username with domain part in the Server user field. Naturally, these settings are available only for System Administrators.
As with the rest notifiers, you can tune up the messages sent: the FreeMarker templates used for generating them are fully customizable (refer to our docs for the details).
Once this is done, a user will need only to specify his account and notification rules at My Settings&Tools | Jabber notifier tab to subscribe to Google Talk notifications.
Enjoy!
As we promised, we continue the configuration parameters story started in the previous post. Today, I’d like to tell you about .NET related properties.
TeamCity .NET plugins provide a bunch of predefined properties indicating that .NET Framework/SDK/Visual Studio/Mono are detected on build agent. Previously you could refer to these properties as to system properties. Now we’ve changed it.
We decided to turn all these properties into configuration parameters, and therefore we’ve converted all refernces to such properties detected in TeamCity to keep builds running. However, it’s not possible to convert references used from a build script. To workaroud the issue you can add system property in build agent configuration or use compatibility mode switch in TeamCity properties:
teamcity.dotnet.properties.compatibility.mode with value truedotNetPropertiesCompatibilityMode configuration parameter with value true in a build configuration settings.Here is the list of changed parameters:
| Before | Now |
system.DotNetFrameworkX.Y |
DotNetFrameworkX.Y_x86 |
system.DotNetFrameworkX.Y_Path |
DotNetFrameworkX.Y_x86_Path |
system.DotNetFrameworkX.Y_xZZ |
DotNetFrameworkX.Y_xZZ |
system.DotNetFrameworkX.Y_xZZ_Path |
DotNetFrameworkX.Y_xZZ_Path |
system.DotNetFrameworkSDKX.Y |
DotNetFrameworkSDKX.Y_x86 |
system.VS200X |
VS200X |
system.Mono |
Mono |
system.MonoVersion |
MonoVersion |
system.WindowsSDKX.Y |
WindowsSDKX.Y |
Note, there will no longer reported .NET Framework configuration parameters without explicit bitness (i.e. x86 or x64).
Twitts about TeamCity