TeamCity Build Dependencies

The subject of build dependencies is neither a trivial nor a minor one. Various build tools approach this subject from different perspectives contributing various solutions, each with its own strengths and weaknesses.

Maven and Gradle users who are familiar with release and snapshot dependencies may not know about TeamCity snapshot dependencies or assume they’re somehow related to Maven (which isn’t true). TeamCity users who are familiar with artifact and snapshot dependencies may not know that adding an Artifactory plugin allows them to use artifact and build dependencies as well, on top of those provided by TeamCity.

Some of the names mentioned above seem not to be established enough while others may require a discussion about their usage patterns. Having this in mind I’ve decided to explore each solution in its own blog post, setting a goal of providing enough information so that people can choose what works best.

The first post explored Maven snapshot and release dependencies. This is the second post, which covers artifact and snapshot dependencies provided by TeamCity and the third and final part will cover the artifact and build dependencies provided by TeamCity Artifactory plugin.

Non-Maven Dependencies

While Maven-based dependencies management and artifact repositories are very common and widespread in Java, there are cases where you may still find them insufficient or inadequate for your needs. For starters, you may not be developing in Java or perhaps your build tool is not providing built-in integration with Maven repositories, as is the case with Ant (or its Gant and NAnt spin-offs), SCons, Rake or MSBuild. Secondly, snapshot Maven dependencies provide their own set of challenges covered in the previous blog post, making it harder to ensure correct snapshot dependency is used in a chain of builds.

In order to address these scenarios, TeamCity provides two ways to connect dependent build configurations and their outcomes: artifact and snapshot dependencies.

TeamCity Artifact Dependencies

The idea of artifact dependencies in TeamCity is very simple: download the artifacts produced by an other build before the current one begins. After the artifacts are downloaded to the folder specified (checkout directory by default), your build script can use them to achieve its goals. You can find configuration details in TeamCity documentation.

Naturally, this scheme is not suitable for build tools with automatic dependencies management, but it works well with build or shell scripts accepting and expecting local paths, relative to the checkout directory. Note that the copying works not only for the produced build binaries, but for any kind of binary or text files, like the TeamCity coverage report as demonstrated on the screenshot above.

There is one important detail about specifying artifact dependencies and that is “Get artifacts from” configuration where you specify what type of build should files be taken from. Possible values of this field are “last successful”, “finished”, “pinned”, or “tagged build”, as well as the build number or “Build from the same chain”. While most values should be trivial to understand with “Last successful build” being the default and generally suitable option, the definition of “same chain” build is directly related to TeamCity snapshot dependencies.

TeamCity Snapshot Dependencies

Imagine a monolithic multi-step build process (build, test, package, deploy) which you decide to split into multiple smaller builds, invoked sequentially, forming a chain of executions. Doing so allows one to configure or trigger every chain step separately and run certain steps in parallel in order to speedup the process (like executing tests or building independent components). Most of all, it makes the overall maintenance significantly easier. However, while doing so you need to ensure every chain step uses the same consistent set of sources pulled from VCS even if newer commits are made all the while chain steps are running. That’s what TeamCity snapshot dependencies are for: they connect several build configurations into a single chain of execution, called build chain, with every step using the same set of sources, regardless of VCS updates. Note that the TeamCity use of the term “snapshot dependencies” may confuse people familiar with Maven snapshot dependencies which are two unrelated concepts.

Snapshot dependencies are configured similarly to artifact dependencies. You can find configuration details in TeamCity documentation.

Using Artifact and Snapshot Dependencies Together

When applicable, it is recommended to define both kinds of dependencies between build configurations, as this ensures not only a consistent set of sources used throughout a chain steps but also a consistent flow of artifacts produced. Now the definition of “Build from the same chain” in artifact dependency mentioned above becomes clear, as this is the only meaningful option in this scenario.

In a way, you can think of build chain steps running in isolation from VCS updates after the first sources’ “snapshot” is taken. Chain artifacts are either re-created from the same sources or passed through chain steps with artifact dependencies. This makes chain steps consistent, reproducible and always up-to-date (when applied to using chain artifacts), something that can’t be easily achieved with Maven snapshot dependencies.

Build Chains Visibility in TeamCity 7.0

TeamCity 7.0 took the notion of build chains to a whole new level by providing build chains a new UI, making chain steps visible and re-runnable. Once you have snapshot dependencies defined, a new “Build Chains” tab appears in project reports, providing a visual representation of all related build chains and a way to re-run any chain step manually, using the same set of sources pulled originally.

Build Chain Triggering

Having build configurations connected with snapshot dependencies and, therefore, their builds grouped into build chains not only makes them more consistent regarding the sources used, it also impacts the way builds are added to the build queue: after a certain chain step is triggered, the default behavior is to add all preceding chain steps as well, keeping their respective order, in addition to the one that was triggered initially. Let me repeat it for more clarity: triggering certain chain configuration adds preceding (those to the left of it) and not subsequent (to the right of it) configurations to the build queue, although it may seem counterintuitive at first. The idea is to mark the location where chain execution stops, which is exactly the configuration that was triggered initially; it becomes the last execution step.

To trigger subsequent chain steps upon VCS changes found in a chain configuration, you can add a VCS trigger with the “Trigger on changes in snapshot dependencies” option to the configuration that would be the last execution step. This configuration is then triggered whenever any of the preceding chain steps is updated, which schedules the whole chain for execution.

Having this behavior in mind, you therefore need to decide which configurations are triggered automatically and which should be run manually. Usually, earlier chain steps having no impact on external environment can be triggered automatically by VCS trigger but final chain steps, potentially modifying external systems, are invoked manually after a human verification of the previous chain results. The process of running the final chain steps manually is usually referred to as “promoting” previously finished builds.

Sample Build Chain: Compile, Test, Deploy

Imagine three sample build configurations, "Compile", "Test" and "Deploy" connected into a build chain: "Deploy" is snapshot dependent on "Test" which is snapshot dependent on "Compile".

In this sample scenario the "Compile" and "Test" configurations are triggered automatically while "Deploy" is triggered manually, following the recommendations given above. VCS changes in "Compile" configuration only trigger an execution of this chain step, while VCS changes in "Test" configuration trigger "Compile" and "Test" execution (in that order).

Once a "Compile" configuration is added to the builds queue, its sources’ timestamp is recorded on the server to be used in all subsequent chain steps. If any of the chain steps is connected to a different VCS root, its sources are also pulled according to the same timestamp.

Promoting Finished Builds

As soon as the automatic chain execution stops (after running "Test"), you can continue it by clicking the corresponding “Run” button on the "Deploy" configuration that was not triggered (see the build chain screenshot above). Alternatively, it is possible to promote a finished "Test" build through its “Build Actions” and invoke configurations which are snapshot dependent on it – "Deploy" configuration in this case.

Summary

This article has provided an overview of TeamCity artifact and snapshot dependencies, build chains, how their steps are triggered and how finished builds are promoted. I hope you now have a good understanding of how it works and of when it is appropriate (or not) to use TeamCity build dependencies in addition to those provided by build tools such as Maven.

Please, refer to the TeamCity documentation for more information about this subject:

• Dependent Build
• Build Chain

The final blog post in the series will uncover how you can use the TeamCity Artifactory plugin in order to achieve a behavior which is similar to build chains for projects with Maven-based dependency management. Stay tuned!