Chapter 23. Multi-project Builds

The powerful support for multi-project builds is one of Gradle's unique selling points. This topic is also the intellectually most challenging.

23.1. Cross Project Configuration

Let's start with a very simple multi-project build. After all Gradle is a general purpose build tool at its core, so the projects don't have to be java projects. Our first examples are about marine life.

23.1.1. Defining Common Behavior

We have the following project tree. This is a multi-project build with a root project water and a subproject bluewhale.

Figure 23.1. Project layout

water/
  build.gradle
  settings.gradle
  bluewhale/

Note: The code for this example can be found at samples/userguide/multiproject/firstExample/water

Example 23.1.  settings.gradle

include 'bluewhale'

And where is the build script for the bluewhale project? In Gradle build scripts are optional. Obviously for a single project build, a project without a build script doesn't make much sense. For multiproject builds the situation is different. Let's look at the build script for the water project and execute it:

Example 23.2.  build.gradle

Closure cl = { task -> println "I'm $task.project.name" }
task hello(cl)
project(':bluewhale') {
    task hello(cl)
}

Example 23.3. Output of gradle -q hello

> gradle -q hello
I'm water
I'm bluewhale

Gradle allows you to access any project of the multi-project build from any build script. The Project API provides a method called project(), which takes a path as an argument and returns the Project object for this path. The capability to configure a project build from any build script we call cross project configuration. Gradle implements this via configuration injection.

We are not that happy with the build script of the water project. It is inconvenient to add the task explicitly for every project. We can do better. Let's first add another project called krill to our multi-project build.

Figure 23.2. Project layout

water/
  build.gradle
  settings.gradle
  bluewhale/
  krill/

Note: The code for this example can be found at samples/userguide/multiproject/addKrill/water

Example 23.4.  settings.gradle

include 'bluewhale', 'krill'

Now we rewrite the water build script and boil it down to a single line.

Example 23.5.  build.gradle

allprojects {
    task hello { task -> println "I'm $task.project.name" }
}

Example 23.6. Output of gradle -q hello

> gradle -q hello
I'm water
I'm bluewhale
I'm krill

Is this cool or is this cool? And how does this work? The Project API provides a property allprojects which returns a list with the current project and all its subprojects underneath it. If you call allprojects with a closure, the statements of the closure are delegated to the projects associated with allprojects. You could also do an iteration via allprojects.each, but that would be more verbose.

Other build systems use inheritance as the primary means for defining common behavior. We also offer inheritance for projects as you will see later. But Gradle uses configuration injection as the usual way of defining common behavior. We think it provides a very powerful and flexible way of configuring multiproject builds.

23.2. Subproject Configuration

The Project API also provides a property for accessing the subprojects only.

23.2.1. Defining Common Behavior

Example 23.7.  build.gradle

allprojects {
    task hello {task -> println "I'm $task.project.name" }
}
subprojects {
    hello.doLast {println "- I depend on water"}
}

Example 23.8. Output of gradle -q hello

> gradle -q hello
I'm water
I'm bluewhale
- I depend on water
I'm krill
- I depend on water

23.2.2. Adding Specific Behavior

You can add specific behavior on top of the common behavior. Usually we put the project specific behavior in the build script of the project where we want to apply this specific behavior. But as we have already seen, we don't have to do it this way. We could add project specific behavior for the bluewhale project like this:

Example 23.9.  build.gradle

allprojects {
    task hello {task -> println "I'm $task.project.name" }
}
subprojects {
    hello.doLast {println "- I depend on water"}
}
project(':bluewhale').hello.doLast {
    println "I'm the largest animal that has ever lived on this planet."
}

Example 23.10. Output of gradle -q hello

> gradle -q hello
I'm water
I'm bluewhale
- I depend on water
I'm the largest animal that has ever lived on this planet.
I'm krill
- I depend on water

As we have said, we usually prefer to put project specific behavior into the build script of this project. Let's refactor and also add some project specific behavior to the krill project.

Figure 23.3. Project layout

water/
  build.gradle
  settings.gradle
  bluewhale/
    build.gradle
  krill/
    build.gradle

Note: The code for this example can be found at samples/userguide/multiproject/spreadSpecifics/water

Example 23.11.  settings.gradle

include 'bluewhale', 'krill'

Example 23.12.  bluewhale/build.gradle

hello.doLast { println "- I'm the largest animal that has ever lived on this planet." }

Example 23.13.  krill/build.gradle

hello.doLast {
    println "- The weight of my species in summer is twice as heavy as all human beings."
}

Example 23.14.  build.gradle

allprojects {
    task hello {task -> println "I'm $task.project.name" }
}
subprojects {
    hello.doLast {println "- I depend on water"}
}

Example 23.15. Output of gradle -q hello

> gradle -q hello
I'm water
I'm bluewhale
- I depend on water
- I'm the largest animal that has ever lived on this planet.
I'm krill
- I depend on water
- The weight of my species in summer is twice as heavy as all human beings.

23.2.3. Project Filtering

To show more of the power of configuration injection, let's add another project called tropicalFish and add more behavior to the build via the build script of the water project.

23.2.3.1. Filtering By Name

Figure 23.4. Project layout

water/
  build.gradle
  settings.gradle
  bluewhale/
    build.gradle
  krill/
    build.gradle
  tropicalFish/

Note: The code for this example can be found at samples/userguide/multiproject/addTropical/water

Example 23.16.  settings.gradle

include 'bluewhale', 'krill', 'tropicalFish'

Example 23.17.  build.gradle

allprojects {
    task hello {task -> println "I'm $task.project.name" }
}
subprojects {
    hello.doLast {println "- I depend on water"}
}
configure(subprojects.findAll {it.name != 'tropicalFish'}) {
    hello.doLast {println '- I love to spend time in the arctic waters.'}
}

Example 23.18. Output of gradle -q hello

> gradle -q hello
I'm water
I'm bluewhale
- I depend on water
- I love to spend time in the arctic waters.
- I'm the largest animal that has ever lived on this planet.
I'm krill
- I depend on water
- I love to spend time in the arctic waters.
- The weight of my species in summer is twice as heavy as all human beings.
I'm tropicalFish
- I depend on water

The configure() method takes a list as an argument and applies the configuration to the projects in this list.

23.2.3.2. Filtering By Properties

Using the project name for filtering is one option. Using dynamic project properties is another.

Figure 23.5. Project layout

water/
  build.gradle
  settings.gradle
  bluewhale/
    build.gradle
  krill/
    build.gradle
  tropicalFish/
    build.gradle

Note: The code for this example can be found at samples/userguide/multiproject/tropicalWithProperties/water

Example 23.19.  settings.gradle

include 'bluewhale', 'krill', 'tropicalFish'

Example 23.20.  bluewhale/build.gradle

arctic = true
hello.doLast { println "- I'm the largest animal that has ever lived on this planet." }

Example 23.21.  krill/build.gradle

arctic = true
hello.doLast {
    println "- The weight of my species in summer is twice as heavy as all human beings."
}

Example 23.22.  tropicalFish/build.gradle

arctic = false

Example 23.23.  build.gradle

allprojects {
    task hello {task -> println "I'm $task.project.name" }
}
subprojects {
    hello {
        doLast {println "- I depend on water"}
        afterEvaluate { Project project ->
            if (project.arctic) { doLast {
                println '- I love to spend time in the arctic waters.' }
            }
        }
    }
}

Example 23.24. Output of gradle -q hello

> gradle -q hello
I'm water
I'm bluewhale
- I depend on water
- I'm the largest animal that has ever lived on this planet.
- I love to spend time in the arctic waters.
I'm krill
- I depend on water
- The weight of my species in summer is twice as heavy as all human beings.
- I love to spend time in the arctic waters.
I'm tropicalFish
- I depend on water

In the build file of the water project we use an afterEvaluate notification. This means that the closure we are passing gets evaluated after the build scripts of the subproject are evaluated. As the property arctic is set in those build scripts, we have to do it this way. You will find more on this topic in Section 23.6, “Dependencies - Which dependencies?”

23.3. Execution rules for multi-project builds

When we have executed the hello task from the root project dir things behaved in an intuitive way. All the hello tasks of the different projects were executed. Let's switch to the bluewhale dir and see what happens if we execute Gradle from there.

Example 23.25. Output of gradle -q hello

> gradle -q hello
I'm bluewhale
- I depend on water
- I'm the largest animal that has ever lived on this planet.
- I love to spend time in the arctic waters.

The basic rule behind Gradle's behavior is simple. Gradle looks down the hierarchy, starting with the current dir, for tasks with the name hello an executes them. One thing is very important to note. Gradle always evaluates every project of the multi-project build and creates all existing task objects. Then, according to the task name arguments and the current dir, Gradle filters the tasks which should be executed. Because of Gradle's cross project configuration every project has to be evaluated before any task gets executed. We will have a closer look at this in the next section. Let's now have our last marine example. Let's add a task to bluewhale andkrill.

Example 23.26.  bluewhale/build.gradle

arctic = true
hello.doLast { println "- I'm the largest animal that has ever lived on this planet." }

task distanceToIceberg {
    println '20 nautical miles'
}

Example 23.27.  krill/build.gradle

arctic = true
hello.doLast { println "- The weight of my species in summer is twice as heavy as all human beings." }

task distanceToIceberg {
    println '5 nautical miles'
}

Example 23.28. Output of gradle -q distanceToIceberg

> gradle -q distanceToIceberg
20 nautical miles
5 nautical miles

Here the output without the -q option:

Example 23.29. Output of gradle distanceToIceberg

> gradle distanceToIceberg
:bluewhale:distanceToIceberg
20 nautical miles
:krill:distanceToIceberg
5 nautical miles

BUILD SUCCESSFUL

Total time: 1 secs

The build is executed from the water project. Neither water nor tropicalFish have a task with the name distanceToIceberg. Gradle does not care. The simple rule mentioned already above is: Execute all tasks down the hierarchy which have this name. Only complain if there is no such task!

23.4. Running Tasks by their Absolute Path

As we have seen, you can run a multi-project build by entering any subproject dir and execute the build from there. All matching task names of the project hierarchy starting with the current dir are executed. But Gradle also offers to execute tasks by their absolute path (see also Section 23.5, “Project and Task Paths”):

Example 23.30. Output of gradle -q :hello :krill:hello hello

> gradle -q :hello :krill:hello hello
I'm water
I'm krill
- I depend on water
- The weight of my species in summer is twice as heavy as all human beings.
- I love to spend time in the arctic waters.
I'm tropicalFish
- I depend on water

The build is executed from the tropicalFish project. We execute the hello tasks of the water, the krill and the tropicalFish project. The first two tasks are specified by there absolute path, the last task is executed on the name matching mechanism described above.

23.5. Project and Task Paths

A project path has the following pattern: It starts always with a colon, which denotes the root project. The root project is the only project in a path that is not specified by its name. The path :bluewhale corresponds to the file system path water/project in the case of the example above.

The path of a task is simply its project path plus the task name. For example :bluewhale:hello. Within a project you can address a task of the same project just by its name. This is interpreted as a relative path.

Originally Gradle has used the '/' character as a natural path separator. With the introduction of directory tasks (see Section 10.3, “Directory Creation”) this was no longer possible, as the name of the directory task contains the '/' character.

23.6. Dependencies - Which dependencies?

The examples from the last section were special, as the projects had no Execution Dependencies. They had only Configuration Dependencies. Here is an example where this is different:

23.6.1. Execution Dependencies

23.6.1.1. Dependencies and Execution Order

Figure 23.6. Project layout

messages/
  settings.gradle
  consumer/
    build.gradle
  producer/
    build.gradle

Note: The code for this example can be found at samples/userguide/multiproject/dependencies/firstMessages/messages

Example 23.31.  settings.gradle

include 'consumer', 'producer'

Example 23.32.  consumer/build.gradle

task action {
    println "Consuming message: " +  System.getProperty('org.gradle.message')
}

Example 23.33.  producer/build.gradle

task action {
    println "Producing message:"
    System.setProperty('org.gradle.message', 'Watch the order of execution.')
}

Example 23.34. Output of gradle -q action

> gradle -q action
Consuming message: null
Producing message:

This did not work out. If nothing else is defined, Gradle executes the task in alphanumeric order. Therefore :consumer:action is executed before :producer:action. Let's try to solve this with a hack and rename the producer project to aProducer.

Figure 23.7. Project layout

messages/
  settings.gradle
  aProducer/
    build.gradle
  consumer/
    build.gradle

Example 23.35.  settings.gradle

include 'consumer', 'aProducer'

Example 23.36.  aProducer/build.gradle

task action {
    println "Producing message:"
    System.setProperty('org.gradle.message', 'Watch the order of execution.')
}

Example 23.37.  consumer/build.gradle

task action {
    println "Consuming message: " +  System.getProperty('org.gradle.message')
}

Example 23.38. Output of gradle -q action

> gradle -q action
Producing message:
Consuming message: Watch the order of execution.

Now we take the air out of this hack. We simply switch to the consumer dir and execute the build.

Example 23.39. Output of gradle -q action

> gradle -q action
Consuming message: null

For Gradle the two action tasks are just not related. If you execute the build from the messages project Gradle executes them both because they have the same name and they are down the hierarchy. In the last example only one action was down the hierarchy and therefore it was the only task that got executed. We need something better than this hack.

23.6.1.2. Declaring Dependencies

Figure 23.8. Project layout

messages/
  settings.gradle
  consumer/
    build.gradle
  producer/
    build.gradle

Note: The code for this example can be found at samples/userguide/multiproject/dependencies/messagesWithDependencies/messages

Example 23.40.  settings.gradle

include 'consumer', 'producer'

Example 23.41.  consumer/build.gradle

dependsOn(':producer')

task action {
    println "Consuming message: " +  System.getProperty('org.gradle.message')
}

Example 23.42.  producer/build.gradle

task action {
    println "Producing message:"
    System.setProperty('org.gradle.message', 'Watch the order of execution.')
}

Example 23.43. Output of gradle -q action

> gradle -q action
Producing message:
Consuming message: Watch the order of execution.

Running this from the consumer directory gives:

Example 23.44. Output of gradle -q action

> gradle -q action
Producing message:
Consuming message: Watch the order of execution.

We have now declared that the consumer project has an execution dependency on the producer project. For Gradle declaring execution dependencies between projects is syntactic sugar. Under the hood Gradle creates task dependencies out of them. You can also create cross project tasks dependencies manually by using the absolute path of the tasks.

23.6.1.3. The Nature of Project Dependencies

Let's change the naming of our tasks and execute the build.

Example 23.45.  consumer/build.gradle

dependsOn(':producer')

task consume {
    println "Consuming message: " +  System.getProperty('org.gradle.message')
}

Example 23.46.  producer/build.gradle

task produce {
    println "Producing message:"
    System.setProperty('org.gradle.message', 'Watch the order of execution.')
}

Example 23.47. Output of gradle -q consume

> gradle -q consume
Consuming message: null

Uhps. Why does this not work? The dependsOn command is created for projects with a common lifecycle. Provided you have two Java projects were one depends on the other. If you trigger a compile for for the dependent project you don't want that all tasks of the other project get executed. Therefore a dependsOn creates dependencies between tasks with equal names. To deal with the scenario above you would do the following:

Example 23.48.  consumer/build.gradle

task consume(dependsOn: ':producer:produce') {
    println "Consuming message: " +  System.getProperty('org.gradle.message')
}

Example 23.49.  producer/build.gradle

task produce {
    println "Producing message:"
    System.setProperty('org.gradle.message', 'Watch the order of execution.')
}

Example 23.50. Output of gradle -q consume

> gradle -q consume
Producing message:
Consuming message: Watch the order of execution.

23.6.2. Configuration Time Dependencies

Let's have one more example with our producer-consumer build before we enter Java land. We add a property to the producer project and create now a configuration time dependency from consumer on producer.

Example 23.51.  consumer/build.gradle

key = 'unknown'
if (project(':producer').hasProperty('key')) {
    key = project(':producer').key
}
task consume(dependsOn: ':producer:produce') {
    println "Consuming message from key '$key': " +  System.getProperty(key)
}

Example 23.52.  producer/build.gradle

key = 'org.gradle.message'

task produce {
    println "Producing message:"
    System.setProperty(key, 'Watch the order of execution.')
}

Example 23.53. Output of gradle -q consume

> gradle -q consume
Producing message:
Consuming message from key 'unknown': null

The default evaluation order of the projects is alphanumeric (for the same nesting level). Therefore the consumer project is evaluated before the producer project and the key value of the producer is set after it is read by the consumer project. Gradle offers a solution for this.

Example 23.54.  consumer/build.gradle

evaluationDependsOn(':producer')

key = 'unknown'
if (project(':producer').hasProperty('key')) {
    key = project(':producer').key
}
task consume(dependsOn: ':producer:produce') {
    println "Consuming message from key '$key': " +  System.getProperty(key)
}

Example 23.55. Output of gradle -q consume

> gradle -q consume
Producing message:
Consuming message from key 'org.gradle.message': Watch the order of execution.

The command evaluationDependsOn triggers the evaluation of producer before consumer is evaluated. The example is a bit contrived for the sake of showing the mechanism. In this case there would be an easier solution by reading the key property at execution time.

Example 23.56.  consumer/build.gradle

task consume(dependsOn: ':producer:produce') {
    String key = project(':producer').key 
    println "Consuming message from key '$key': " +  System.getProperty(key)
}

Example 23.57. Output of gradle -q consume

> gradle -q consume
Producing message:
Consuming message from key 'org.gradle.message': Watch the order of execution.

Configuration dependencies are very different to execution dependencies. Configuration dependencies are between projects whereas execution dependencies are always resolved to task dependencies. Another difference is that always all projects are configured, even when you start the build from a subproject. The default configuration order is top down, which is usually what is needed.

On the same nesting level the configuration order depends on the alphanumeric position. The most common use case is to have multi-project builds that share a common lifecycle (e.g. all projects use the Java plugin). If you declare with dependsOn a execution dependency between different projects, the default behavior of this method is to create also a configuration dependency between the two projects. Therefore it is likely that you don't have to define configuration dependencies explicitly.

23.6.3. Real Life examples

Gradle's multi-project features are driven by real life use cases. The first example for describing such a use case, consists of two webapplication projects and a parent project that creates a distribution out of them. [31] For the example we use only one build script and do cross project configuration.

Figure 23.9. Project layout

webDist/
  settings.gradle
  build.gradle
  date/
    src/main/java/org/gradle/sample/DateServlet.java
  hello/
    src/main/java/org/gradle/sample/HelloServlet.java

Note: The code for this example can be found at samples/userguide/multiproject/dependencies/webDist

Example 23.58.  settings.gradle

include 'date', 'hello'

Example 23.59.  build.gradle

dependsOnChildren()

allprojects {
    usePlugin('java')
    group = 'org.gradle.sample'
    version = '1.0'
}

subprojects {
    usePlugin('war')
    repositories {
        mavenCentral()
    }
    dependencies {
        compile "javax.servlet:servlet-api:2.5"
    }
}

task explodedDist(dependsOn: libs) {
    File explodedDist = mkdir(buildDir, 'explodedDist')
    subprojects.each {project ->
        project.libs.archiveTasks.each {archiveTask ->
            ant.copy(file: archiveTask.archivePath, todir: explodedDist)
        }
    }
}

We have an interesting set of dependencies. Obviously the date and hello task have a configuration dependency on webDist, as all the build logic for the webapp projects is injected by webDist. The execution dependency is in the other direction, as webDist depends on the build artifacts of date and hello. There is even a third dependency. webDist has a configuration dependency on date and hello because it needs to know the archivePath. But it asks for this information at execution time. Therefore we have no circular dependency.

Such and other dependency patterns are daily bread in the problem space of multi-project builds. If a build system does not support such patterns, you either can't solve your problem or you need to do ugly hacks which are hard to maintain and massively afflict your productivity as a build master.

There is one more thing to note from the current example. We have used the command dependOnChildren(). It is a convenience method and calls the dependsOn method of the parent project for every child project (not every sub project). It declares a execution dependency of webDist on date and hello.

Another use case would be a situation where the subprojects have a configuration and execution dependency on the parent project. This is the case when the parent project does configuration injection into its subprojects, and additionally produces something at execution time that is needed by its child projects (e.g. code generation). In this case the parent project would call the childrenDependOnMe method to create an execution dependency for the child projects. We might add an example for this in a future version of the userguide.

23.7. Project Lib Dependencies

What if one projects needs the jar produced by another project in its compile path. And not just the jar but also the transitive dependencies of this jar. Obviously this is a very common use case for Java multi-project builds. As already mentioned in Section 20.2.4, “Project dependencies”, Gradle offers project dependencies for this.

Figure 23.10. Project layout

java/
  settings.gradle
  build.gradle
  api/
    src/main/java/org/gradle/sample/api/Person.java
    src/main/java/org/gradle/sample/apiImpl/PersonImpl.java
  services/
    personService/
      src/main/java/org/gradle/sample/services/PersonService.java
      src/test/java/org/gradle/sample/services/PersonServiceTest.java
  shared/
    src/main/java/org/gradle/sample/shared/Helper.java

Note: The code for this example can be found at samples/userguide/multiproject/dependencies/java

We have the projects shared, api and personService. personService has a lib dependency on the other two projects. api has a lib dependency on shared. [32]

Example 23.60.  settings.gradle

include 'api', 'shared', 'services:personService'

Example 23.61.  build.gradle

subprojects {
    usePlugin('java')
    group = 'org.gradle.sample'
    version = '1.0'
}

project(':api') {
    dependencies.compile project(':shared')
}

project(':services:personService') {
    dependencies {
        compile project(':shared'), project(':api')
        testCompile "junit:junit:3.8.2"
    }
}

All the build logic is in the build.gradle of the root project. [33] A lib dependency is a special form of an execution dependency. It causes the other project to be build first and adds the jar with the classes of the other project to the classpath. It also add the dependencies of the other project to the classpath. So you can enter the api folder and trigger a gradle compile. First shared is build and then api is build. Project dependencies enable partial multi-project builds.

If you come from Maven land you might be perfectly happy with this. If you come from Ivy land, you might expect some more fine grained control. Gradle offers this to you:

Example 23.62.  build.gradle

subprojects {
    usePlugin('java')
    group = 'org.gradle.sample'
    version = '1.0'
}

project(':api') {
    configurations {
        spi
    }
    dependencies {
        compile project(':shared')
    }
    task spiJar(type: Jar) {
        baseName = 'api-spi'
        confs = ['spi']
        fileSet() {
            include('org/gradle/sample/api/**')
        }
    }
}

project(':services:personService') {
    dependencies {
        compile project(':shared')
        compile project(':api') {
            dependencyConfiguration = 'spi'
        }
        testCompile "junit:junit:3.8.2", project(':api')
    }
}

The Java plugin adds per default a jar to your project libraries which contains all the classes. In this example we create an additional library containing only the interfaces of the api project. We assign this library to a new dependency configuration. For the person service we declare that the project should be compiled only against the api interfaces but tested with all classes from api.

23.8. Property and Method Inheritance

Properties and methods declared in a project are inherited to all its subprojects. This is an alternative to configuration injection. But we think that the model of inheritance does not reflect the problem space of multi-project builds very well. In a future edition of this userguide we might write more about this.

Method inheritance might be interesting to use as Gradle's Configuration Injection does not support methods yet (but will in a future release.).

You might be wondering why we have implemented a feature we obviously don't like that much. One reason is that it is offered by other tools and we want to have the check mark in a feature comparison :). And we like to offer our users a choice.

23.9. Summary

Writing this chapter was pretty exhausting and reading it might have a similar effect. Our final message for this chapter is that multi-project builds with Gradle are usually not difficult. There are six elements you need to remember: allproject, subprojects, dependsOn, childrenDependOnMe, dependOnChildren and project lib dependencies. [34] With those elements, and keeping in mind that Gradle has a distinct configuration and execution phase, you have already a lot of flexibility. But when you enter steep territory Gradle does not become an obstacle and usually accompanies and carries you to the top of the mountain.



[31] The real use case we had, was using http://lucene.apache.org/solr, where you need a separate war for each index your are accessing. That was one reason why we have created a distribution of webapps. The Resin servlet container allows us, to let such a distribution point to a base installation of the servlet container.

[32] services is also a project, but we use it just as a container. It has no build script and gets nothing injected by another build script.

[33] We do this here, as it makes the layout a bit easier. We usually put the project specific stuff into the buildscript of the respective projects.

[34] So we are well in the range of the 7 plus 2 Rule :)