# Testing Build Logic with TestKit

version 6.5-milestone-1

The Gradle TestKit (a.k.a. just TestKit) is a library that aids in testing Gradle plugins and build logic generally. At this time, it is focused on functional testing. That is, testing build logic by exercising it as part of a programmatically executed build. Over time, the TestKit will likely expand to facilitate other kinds of tests.

## Usage

To use the TestKit, include the following in your plugin’s build:

Example 1. Declaring the TestKit dependency
build.gradle

    dependencies {
        testImplementation gradleTestKit()
    }

build.gradle.kts

    dependencies {
        testImplementation(gradleTestKit())
    }

The `gradleTestKit()` encompasses the classes of the TestKit, as well as the Gradle Tooling API client. It does not include a version of JUnit, TestNG, or any other test execution framework. Such a dependency must be explicitly declared.

Example 2. Declaring the JUnit dependency
build.gradle

    dependencies {
        testImplementation 'junit:junit:4.13'
    }

build.gradle.kts

    dependencies {
        testImplementation("junit:junit:4.13")
    }

## Functional testing with the Gradle runner

The GradleRunner facilitates programmatically executing Gradle builds, and inspecting the result.

A contrived build can be created (e.g. programmatically, or from a template) that exercises the “logic under test”. The build can then be executed, potentially in a variety of ways (e.g. different combinations of tasks and arguments). The correctness of the logic can then be verified by asserting the following, potentially in combination:

- The build’s output;

- The build’s logging (i.e. console output);

- The set of tasks executed by the build and their results (e.g. FAILED, UP-TO-DATE etc.).

After creating and configuring a runner instance, the build can be executed via the GradleRunner.build() or GradleRunner.buildAndFail() methods depending on the anticipated outcome.

The following demonstrates the usage of the Gradle runner in a Java JUnit test:

### Example: Using GradleRunner with Java and JUnit

BuildLogicFunctionalTest.java

    import org.gradle.testkit.runner.BuildResult;
    import org.gradle.testkit.runner.GradleRunner;
    import org.junit.Before;
    import org.junit.Rule;
    import org.junit.Test;
    import org.junit.rules.TemporaryFolder;

    import java.io.BufferedWriter;
    import java.io.File;
    import java.io.FileWriter;
    import java.io.IOException;
    import java.util.Collections;

    import static org.junit.Assert.assertEquals;
    import static org.junit.Assert.assertTrue;

    import static org.gradle.testkit.runner.TaskOutcome.*;

    public class BuildLogicFunctionalTest {
        @Rule public final TemporaryFolder testProjectDir = new TemporaryFolder();
        private File settingsFile;
        private File buildFile;

        @Before
        public void setup() throws IOException {
            settingsFile = testProjectDir.newFile("settings.gradle");
            buildFile = testProjectDir.newFile("build.gradle");
        }

        @Test
        public void testHelloWorldTask() throws IOException {
            writeFile(settingsFile, "rootProject.name = 'hello-world'");
            String buildFileContent = "task helloWorld {" +
                                      "    doLast {" +
                                      "        println 'Hello world!'" +
                                      "    }" +
                                      "}";
            writeFile(buildFile, buildFileContent);

            BuildResult result = GradleRunner.create()
                .withProjectDir(testProjectDir.getRoot())
                .withArguments("helloWorld")
                .build();

            assertTrue(result.getOutput().contains("Hello world!"));
            assertEquals(SUCCESS, result.task(":helloWorld").getOutcome());
        }

        private void writeFile(File destination, String content) throws IOException {
            BufferedWriter output = null;
            try {
                output = new BufferedWriter(new FileWriter(destination));
                output.write(content);
            } finally {
                if (output != null) {
                    output.close();
                }
            }
        }
    }

The following demonstrates the usage of the Gradle runner in a Kotlin JUnit test:

### Example: Using GradleRunner with Kotlin and JUnit

BuildLogicFunctionalTest.kt

    import org.gradle.testkit.runner.BuildResult
    import org.gradle.testkit.runner.GradleRunner
    import org.gradle.testkit.runner.TaskOutcome
    import org.junit.Assert.assertEquals
    import org.junit.Assert.assertTrue
    import org.junit.Before
    import org.junit.Rule
    import org.junit.Test
    import org.junit.rules.TemporaryFolder
    import kotlin.jvm.JvmField
    import java.io.File

    class BuildLogicFunctionalTest {

        @Rule @JvmField
        val testProjectDir: TemporaryFolder = TemporaryFolder()
        private lateinit var settingsFile: File
        private lateinit var buildFile: File

        @Before
        fun setup() {
            settingsFile = testProjectDir.newFile("settings.gradle.kts")
            buildFile = testProjectDir.newFile("build.gradle.kts")
        }

        @Test
        fun `test helloWorld task`() {

            settingsFile.writeText("""
                rootProject.name = "hello-world"
            """.trimIndent())
            buildFile.writeText("""
                tasks.register("helloWorld") {
                    doLast {
                        println("Hello world!")
                    }
                }
            """.trimIndent())

            val result = GradleRunner.create()
                .withProjectDir(testProjectDir.root)
                .withArguments("helloWorld")
                .build()

            assertTrue(result.output.contains("Hello world!"))
            assertEquals(TaskOutcome.SUCCESS, result.task(":helloWorld")?.outcome)
        }
    }

Any test execution framework can be used.

As Gradle build scripts can also be written in the Groovy programming language, it is often a productive choice to write Gradle functional tests in Groovy. Furthermore, it is recommended to use the (Groovy based) Spock test execution framework as it offers many compelling features over the use of JUnit.

The following demonstrates the usage of the Gradle runner in a Groovy Spock test:

### Example: Using GradleRunner with Groovy and Spock

BuildLogicFunctionalTest.groovy

    import org.gradle.testkit.runner.GradleRunner
    import static org.gradle.testkit.runner.TaskOutcome.*
    import org.junit.Rule
    import org.junit.rules.TemporaryFolder
    import spock.lang.Specification

    class BuildLogicFunctionalTest extends Specification {
        @Rule TemporaryFolder testProjectDir = new TemporaryFolder()
        File settingsFile
        File buildFile

        def setup() {
            settingsFile = testProjectDir.newFile('settings.gradle')
            buildFile = testProjectDir.newFile('build.gradle')
        }

        def "hello world task prints hello world"() {
            given:
            settingsFile << "rootProject.name = 'hello-world'"
            buildFile << """
                task helloWorld {
                    doLast {
                        println 'Hello world!'
                    }
                }
            """

            when:
            def result = GradleRunner.create()
                .withProjectDir(testProjectDir.root)
                .withArguments('helloWorld')
                .build()

            then:
            result.output.contains('Hello world!')
            result.task(":helloWorld").outcome == SUCCESS
        }
    }

It is a common practice to implement any custom build logic (like plugins and task types) that is more complex in nature as external classes in a standalone project. The main driver behind this approach is bundle the compiled code into a JAR file, publish it to a binary repository and reuse it across various projects.

## Getting the plugin-under-test into the test build

The GradleRunner uses the Tooling API to execute builds. An implication of this is that the builds are executed in a separate process (i.e. not the same process executing the tests). Therefore, the test build does not share the same classpath or classloaders as the test process and the code under test is not implicitly available to the test build.

Starting with version 2.13, Gradle provides a conventional mechanism to inject the code under test into the test build.

### Automatic injection with the Java Gradle Plugin Development plugin

The Java Gradle Plugin development plugin can be used to assist in the development of Gradle plugins. Starting with Gradle version 2.13, the plugin provides a direct integration with TestKit. When applied to a project, the plugin automatically adds the `gradleTestKit()` dependency to the test compile configuration. Furthermore, it automatically generates the classpath for the code under test and injects it via GradleRunner.withPluginClasspath() for any `GradleRunner` instance created by the user. It’s important to note that the mechanism currently only works if the plugin under test is applied using the plugins DSL. If the target Gradle version is prior to 2.8, automatic plugin classpath injection is not performed.

The plugin uses the following conventions for applying the TestKit dependency and injecting the classpath:

- Source set containing code under test: `sourceSets.main`

- Source set used for injecting the plugin classpath: `sourceSets.test`

Any of these conventions can be reconfigured with the help of the class GradlePluginDevelopmentExtension.

The following Groovy-based sample demonstrates how to automatically inject the plugin classpath by using the standard conventions applied by the Java Gradle Plugin Development plugin.

Example 3. Using the Java Gradle Development plugin for generating the plugin metadata
build.gradle

    plugins {
        id 'groovy'
        id 'java-gradle-plugin'
    }

    dependencies {
        testImplementation('org.spockframework:spock-core:1.3-groovy-2.4') {
            exclude module: 'groovy-all'
        }
    }

build.gradle.kts

    plugins {
        groovy
        `java-gradle-plugin`
    }

    dependencies {
        testImplementation("org.spockframework:spock-core:1.3-groovy-2.4") {
            exclude(module = "groovy-all")
        }
    }

### Example: Automatically injecting the code under test classes into test builds

src/test/groovy/org/gradle/sample/BuildLogicFunctionalTest.groovy

    def "hello world task prints hello world"() {
        given:
        settingsFile << "rootProject.name = 'hello-world'"
        buildFile << """
            plugins {
                id 'org.gradle.sample.helloworld'
            }
        """

        when:
        def result = GradleRunner.create()
            .withProjectDir(testProjectDir.root)
            .withArguments('helloWorld')
            .withPluginClasspath()
            .build()

        then:
        result.output.contains('Hello world!')
        result.task(":helloWorld").outcome == SUCCESS
    }

The following build script demonstrates how to reconfigure the conventions provided by the Java Gradle Plugin Development plugin for a project that uses a custom `Test` source set.

Example 4. Reconfiguring the classpath generation conventions of the Java Gradle Development plugin
build.gradle

    plugins {
        id 'groovy'
        id 'java-gradle-plugin'
    }

    sourceSets {
        functionalTest {
            groovy {
                srcDir file('src/functionalTest/groovy')
            }
            resources {
                srcDir file('src/functionalTest/resources')
            }
            compileClasspath += sourceSets.main.output + configurations.testRuntimeClasspath
            runtimeClasspath += output + compileClasspath
        }
    }

    task functionalTest(type: Test) {
        testClassesDirs = sourceSets.functionalTest.output.classesDirs
        classpath = sourceSets.functionalTest.runtimeClasspath
    }

    check.dependsOn functionalTest

    gradlePlugin {
        testSourceSets sourceSets.functionalTest
    }

    dependencies {
        functionalTestImplementation('org.spockframework:spock-core:1.3-groovy-2.4') {
            exclude module: 'groovy-all'
        }
    }

build.gradle.kts

    plugins {
        groovy
        `java-gradle-plugin`
    }

    sourceSets {
        create("functionalTest") {
            withConvention(GroovySourceSet::class) {
                groovy {
                    srcDir(file("src/functionalTest/groovy"))
                }
            }
            resources {
                srcDir(file("src/functionalTest/resources"))
            }
            compileClasspath += sourceSets.main.get().output + configurations.testRuntimeClasspath
            runtimeClasspath += output + compileClasspath
        }
    }

    tasks.register<Test>("functionalTest") {
        testClassesDirs = sourceSets["functionalTest"].output.classesDirs
        classpath = sourceSets["functionalTest"].runtimeClasspath
    }

    tasks.check { dependsOn(tasks["functionalTest"]) }

    gradlePlugin {
        testSourceSets(sourceSets["functionalTest"])
    }

    dependencies {
        "functionalTestImplementation"("org.spockframework:spock-core:1.3-groovy-2.4") {
            exclude(module = "groovy-all")
        }
    }

### Working with Gradle versions prior to 2.13

For earlier versions of Gradle (before 2.13), it is possible to manually make the code under test available via some extra configuration. The following example demonstrates having the build generate a file containing the implementation classpath of the code under test, and making it available at test runtime.

Example 5. Making the code under test classpath available to the tests
build.gradle

    // Write the plugin's classpath to a file to share with the tests
    task createClasspathManifest {
        def outputDir = file("$buildDir/$name")

        inputs.files(sourceSets.main.runtimeClasspath)
            .withPropertyName("runtimeClasspath")
            .withNormalizer(ClasspathNormalizer)
        outputs.dir(outputDir)
            .withPropertyName("outputDir")

        doLast {
            outputDir.mkdirs()
            file("$outputDir/plugin-classpath.txt").text = sourceSets.main.runtimeClasspath.join("\n")
        }
    }

    // Add the classpath file to the test runtime classpath
    dependencies {
        testRuntimeOnly files(createClasspathManifest)
    }

build.gradle.kts

    // Write the plugin's classpath to a file to share with the tests
    tasks.register("createClasspathManifest") {
        val outputDir = file("$buildDir/$name")

        inputs.files(sourceSets.main.get().runtimeClasspath)
            .withPropertyName("runtimeClasspath")
            .withNormalizer(ClasspathNormalizer::class)
        outputs.dir(outputDir)
            .withPropertyName("outputDir")

        doLast {
            outputDir.mkdirs()
            file("$outputDir/plugin-classpath.txt").writeText(sourceSets.main.get().runtimeClasspath.joinToString("\n"))
        }
    }

    // Add the classpath file to the test runtime classpath
    dependencies {
        testRuntimeOnly(files(tasks["createClasspathManifest"]))
    }

The tests can then read this value, and inject the classpath into the test build by using the method GradleRunner.withPluginClasspath(java.lang.Iterable). This classpath is then available to use to locate plugins in a test build via the plugins DSL (see Plugins). Applying plugins with the plugins DSL requires the definition of a plugin identifier. The following is an example (in Groovy) of doing this from within a Spock Framework `setup()` method, which is analogous to a JUnit `@Before` method.

### Example: Injecting the code under test classes into test builds

src/test/groovy/org/gradle/sample/BuildLogicFunctionalTest.groovy

        List<File> pluginClasspath

        def setup() {
            settingsFile = testProjectDir.newFile('settings.gradle')
            buildFile = testProjectDir.newFile('build.gradle')

            def pluginClasspathResource = getClass().classLoader.findResource("plugin-classpath.txt")
            if (pluginClasspathResource == null) {
                throw new IllegalStateException("Did not find plugin classpath resource, run `testClasses` build task.")
            }

            pluginClasspath = pluginClasspathResource.readLines().collect { new File(it) }
        }

        def "hello world task prints hello world"() {
            given:
            buildFile << """
                plugins {
                    id 'org.gradle.sample.helloworld'
                }
            """

            when:
            def result = GradleRunner.create()
                .withProjectDir(testProjectDir.root)
                .withArguments('helloWorld')
                .withPluginClasspath(pluginClasspath)
                .build()

            then:
            result.output.contains('Hello world!')
            result.task(":helloWorld").outcome == SUCCESS
        }

This approach works well when executing the functional tests as part of the Gradle build. When executing the functional tests from an IDE, there are extra considerations. Namely, the classpath manifest file points to the class files etc. generated by Gradle and not the IDE. This means that after making a change to the source of the code under test, the source must be recompiled by Gradle. Similarly, if the effective classpath of the code under test changes, the manifest must be regenerated. In either case, executing the `testClasses` task of the build will ensure that things are up to date.

Some IDEs provide a convenience option to delegate the "test classpath generation and execution" to the build. In IntelliJ you can find this option under Preferences…​ > Build, Execution, Deployment > Build Tools > Gradle > Runner > Delegate IDE build/run actions to gradle. Please consult the documentation of your IDE for more information.

### Working with Gradle versions prior to 2.8

The GradleRunner.withPluginClasspath(java.lang.Iterable) method will not work when executing the build with a Gradle version earlier than 2.8 (see The version used to test), as this feature is not supported on such Gradle versions.

Instead, the code must be injected via the build script itself. The following sample demonstrates how this can be done.

### Example: Injecting the code under test classes into test builds for Gradle versions prior to 2.8

src/test/groovy/org/gradle/sample/BuildLogicFunctionalTest.groovy

        List<File> pluginClasspath

        def setup() {
            settingsFile = testProjectDir.newFile('settings.gradle')
            buildFile = testProjectDir.newFile('build.gradle')

            def pluginClasspathResource = getClass().classLoader.findResource("plugin-classpath.txt")
            if (pluginClasspathResource == null) {
                throw new IllegalStateException("Did not find plugin classpath resource, run `testClasses` build task.")
            }

            pluginClasspath = pluginClasspathResource.readLines().collect { new File(it) }
        }

        def "hello world task prints hello world with pre Gradle 2.8"() {
            given:
            def classpathString = pluginClasspath
                .collect { it.absolutePath.replace('\\', '\\\\') } // escape backslashes in Windows paths
                .collect { "'$it'" }
                .join(", ")

            buildFile << """
                buildscript {
                    dependencies {
                        classpath files($classpathString)
                    }
                }
                apply plugin: "org.gradle.sample.helloworld"
            """

            when:
            def result = GradleRunner.create()
                .withProjectDir(testProjectDir.root)
                .withArguments('helloWorld')
                .withGradleVersion("2.7")
                .build()

            then:
            result.output.contains('Hello world!')
            result.task(":helloWorld").outcome == SUCCESS
        }

## Controlling the build environment

The runner executes the test builds in an isolated environment by specifying a dedicated "working directory" in a directory inside the JVM’s temp directory (i.e. the location specified by the `java.io.tmpdir` system property, typically `/tmp`). Any configuration in the default Gradle user home directory (e.g. `~/.gradle/gradle.properties`) is not used for test execution. The TestKit does not expose a mechanism for fine grained control of all aspects of the environment (e.g., JDK). Future versions of the TestKit will provide improved configuration options.

The TestKit uses dedicated daemon processes that are automatically shut down after test execution.

## The Gradle version used to test

The Gradle runner requires a Gradle distribution in order to execute the build. The TestKit does not depend on all of Gradle’s implementation.

By default, the runner will attempt to find a Gradle distribution based on where the `GradleRunner` class was loaded from. That is, it is expected that the class was loaded from a Gradle distribution, as is the case when using the `gradleTestKit()` dependency declaration.

When using the runner as part of tests being executed by Gradle (e.g. executing the `test` task of a plugin project), the same distribution used to execute the tests will be used by the runner. When using the runner as part of tests being executed by an IDE, the same distribution of Gradle that was used when importing the project will be used. This means that the plugin will effectively be tested with the same version of Gradle that it is being built with.

Alternatively, a different and specific version of Gradle to use can be specified by the any of the following `GradleRunner` methods:

- GradleRunner.withGradleVersion(java.lang.String)

- GradleRunner.withGradleInstallation(java.io.File)

- GradleRunner.withGradleDistribution(java.net.URI)

This can potentially be used to test build logic across Gradle versions. The following demonstrates a cross-version compatibility test written as Groovy Spock test:

### Example: Specifying a Gradle version for test execution

BuildLogicFunctionalTest.groovy

    import org.gradle.testkit.runner.GradleRunner
    import static org.gradle.testkit.runner.TaskOutcome.*
    import org.junit.Rule
    import org.junit.rules.TemporaryFolder
    import spock.lang.Specification
    import spock.lang.Unroll

    class BuildLogicFunctionalTest extends Specification {
        @Rule final TemporaryFolder testProjectDir = new TemporaryFolder()
        File settingsFile
        File buildFile

        def setup() {
            settingsFile = testProjectDir.newFile('settings.gradle')
            buildFile = testProjectDir.newFile('build.gradle')
        }

        @Unroll
        def "can execute hello world task with Gradle version #gradleVersion"() {
            given:
            buildFile << """
                task helloWorld {
                    doLast {
                        logger.quiet 'Hello world!'
                    }
                }
            """

            when:
            def result = GradleRunner.create()
                .withGradleVersion(gradleVersion)
                .withProjectDir(testProjectDir.root)
                .withArguments('helloWorld')
                .build()

            then:
            result.output.contains('Hello world!')
            result.task(":helloWorld").outcome == SUCCESS

            where:
            gradleVersion << ['2.6', '2.7']
        }
    }

### Feature support when testing with different Gradle versions

It is possible to use the GradleRunner to execute builds with Gradle 1.0 and later. However, some runner features are not supported on earlier versions. In such cases, the runner will throw an exception when attempting to use the feature.

The following table lists the features that are sensitive to the Gradle version being used.

Table 1. Gradle version compatibility
Feature | Minimum Version | Description
--- | --- | ---
Inspecting executed tasks | 2.5 | Inspecting the executed tasks, using BuildResult.getTasks() and similar methods.
Plugin classpath injection | 2.8 | Injecting the code under test viaGradleRunner.withPluginClasspath(java.lang.Iterable).
Inspecting build output in debug mode | 2.9 | Inspecting the build’s text output when run in debug mode, using BuildResult.getOutput().
Automatic plugin classpath injection | 2.13 | Injecting the code under test automatically via GradleRunner.withPluginClasspath() by applying the Java Gradle Plugin Development plugin.
Setting environment variables to be used by the build. | 3.5 | The Gradle Tooling API only supports setting environment variables in later versions.

## Debugging build logic

The runner uses the Tooling API to execute builds. An implication of this is that the builds are executed in a separate process (i.e. not the same process executing the tests). Therefore, executing your tests in debug mode does not allow you to debug your build logic as you may expect. Any breakpoints set in your IDE will be not be tripped by the code being exercised by the test build.

The TestKit provides two different ways to enable the debug mode:

- Setting “`org.gradle.testkit.debug`” system property to `true` for the JVM using the `GradleRunner` (i.e. not the build being executed with the runner);

- Calling the GradleRunner.withDebug(boolean) method.

The system property approach can be used when it is desirable to enable debugging support without making an adhoc change to the runner configuration. Most IDEs offer the capability to set JVM system properties for test execution, and such a feature can be used to set this system property.

## Testing with the Build Cache

To enable the Build Cache in your tests, you can pass the `--build-cache` argument to GradleRunner or use one of the other methods described in Enable the build cache. You can then check for the task outcome TaskOutcome.FROM_CACHE when your plugin’s custom task is cached. This outcome is only valid for Gradle 3.5 and newer.

### Example: Testing cacheable tasks

BuildLogicFunctionalTest.groovy

    def "cacheableTask is loaded from cache"() {
        given:
        buildFile << """
            plugins {
                id 'org.gradle.sample.helloworld'
            }
        """

        when:
        def result = runner()
            .withArguments( '--build-cache', 'cacheableTask')
            .build()

        then:
        result.task(":cacheableTask").outcome == SUCCESS

        when:
        new File(testProjectDir.root, 'build').deleteDir()
        result = runner()
            .withArguments( '--build-cache', 'cacheableTask')
            .build()

        then:
        result.task(":cacheableTask").outcome == FROM_CACHE
    }

Note that TestKit re-uses a Gradle user home between tests (see GradleRunner.withTestKitDir(java.io.File)) which contains the default location for the local build cache. For testing with the build cache, the build cache directory should be cleaned between tests. The easiest way to accomplish this is to configure the local build cache to use a temporary directory.

### Example: Clean build cache between tests

BuildLogicFunctionalTest.groovy

    @Rule final TemporaryFolder testProjectDir = new TemporaryFolder()
    File buildFile
    File localBuildCacheDirectory

    def setup() {
        localBuildCacheDirectory = testProjectDir.newFolder('local-cache')
        testProjectDir.newFile('settings.gradle') << """
            buildCache {
                local {
                    directory '${localBuildCacheDirectory.toURI()}'
                }
            }
        """
        buildFile = testProjectDir.newFile('build.gradle')
    }