Kover Plugin (0.8.2) Incompatible with Gradle TestKit for Functional Tests in Gradle 8.7+ and Missing Configuration in Aggregation Plugin

[tyl3r-ch]

Kover Plugin (0.8.2) Incompatible with Gradle TestKit for Functional Tests in Gradle 8.7+ and Missing Configuration in Aggregation Plugin

Environment

• Gradle Version: 8.7, 8.8, 8.9-rc-1
• Kover Version: 0.8.2
• Kotlin Version: 1.9.22, 1.9.24, 2.0.0
• OS: macOS
• JDK Version: 21

Description

1. The Kover plugin (version 0.8.2) is causing failures in functional tests that use Gradle TestKit when running on Gradle versions 8.7 and above. This issue occurs in a Gradle plugin project using Kotlin, and affects both the Kover aggregation plugin applied in settings.gradle.kts and the Kover plugin applied directly in a project.

2. Additionally, the new Kover aggregation plugin is missing a crucial configuration that prevents coverage for functional tests. This configuration is available in the project-level Kover plugin but not in the aggregation plugin.

Steps to Reproduce

1. Clone the repository: [Repository URL]
2. Ensure Gradle 8.7+ is installed
3. Apply Kover plugin in one of two ways:
   a. In settings.gradle.kts:

   plugins {
       id("org.jetbrains.kotlinx.kover.aggregation") version "0.8.2"
   }
   
   kover {
       enableCoverage()
       reports {
           includedProjects.add(":plugin-test")
       }
   }

   b. Or directly in build.gradle.kts:

   plugins {
       id("org.jetbrains.kotlinx.kover") version "0.8.2"
   }

4. Run the functional tests: ./gradlew :plugin-test:functionalTest

Expected Behavior

1. All functional tests in GradleFunctionalTestPluginFunctionalTest.kt should execute successfully, including tests for Gradle versions 8.7, 8.8, and 8.9-rc-1.
2. Coverage reports should include data from functional tests when using either the project-level Kover plugin or the aggregation plugin.

Actual Behavior

1. Functional tests fail to run correctly. Specific error messages indicate issues with loading the plugin extension:

> Configure project :
w: Classpath entry points to a non-existent location: /Users/.../Downloads/gradle-functional-test/plugin-test/build/classes/java/main
w: Classpath entry points to a non-existent location: /Users/.../Downloads/gradle-functional-test/plugin-test/build/classes/java/main
e: file:///private/var/folders/bf/zs2dlk055j11mz9wwxr369th0000gn/T/junit8183199304049029516/build.gradle.kts:1:12: Unresolved reference: example
e: file:///private/var/folders/bf/zs2dlk055j11mz9wwxr369th0000gn/T/junit8183199304049029516/build.gradle.kts:8:5: Unresolved reference: test
e: file:///private/var/folders/bf/zs2dlk055j11mz9wwxr369th0000gn/T/junit8183199304049029516/build.gradle.kts:11:1: Unresolved reference: helloWorld
test value from greeting extension: INFO

2. When using the Kover aggregation plugin, it's not possible to configure coverage for functional tests due to a missing configuration option.

Relevant Code

settings.gradle.kts (for aggregation plugin)

plugins {
    id("org.jetbrains.kotlinx.kover.aggregation") version "0.8.2"
}

kover {
    enableCoverage()
    reports {
        includedProjects.add(":plugin-test")
    }
    // The following configuration is missing and needed for functional test coverage
    // currentProject {
    //     sourceSets.add(functionalTestSourceSet)
    //     sources {
    //         excludedSourceSets.add(functionalTestSourceSet.name)
    //     }
    // }
}

plugin-test/build.gradle.kts (for project-level plugin)

plugins {
    `java-gradle-plugin`
    alias(libs.plugins.jvm.ontt)
    id("org.jetbrains.kotlinx.kover") version "0.8.2"
}

// ... [Other relevant build configurations]

val functionalTestSourceSet = sourceSets.create("functionalTest") {
    compileClasspath += sourceSets.main.get().output + configurations.testRuntimeClasspath.get()
    runtimeClasspath += output + compileClasspath
}

kover {
    currentProject {
        sourceSets.add(functionalTestSourceSet)
        sources {
            excludedSourceSets.add(functionalTestSourceSet.name)
        }
    }
}

plugin-test/src/functionalTest/kotlin/org/example/GradleFunctionalTestPluginFunctionalTest.kt

class GradleFunctionalTestPluginFunctionalTest {
    @Test
    fun `can run task dot eight`() {
        // ... [Test implementation]
    }

    // ... [Other test methods]
}

Troubleshooting Steps Taken

1. Tested with multiple Kotlin versions (1.9.22, 1.9.24, 2.0.0)
2. Verified functionality by commenting out Kover plugin in settings.gradle.kts
3. Attempted to apply Kover plugin directly in plugin-test/build.gradle.kts
4. Reviewed Kover documentation for known issues or configuration requirements
5. Added configuration for functional test source set in Kover block (for project-level plugin)
6. Attempted to add similar configuration for aggregation plugin (not possible due to missing option)

Possible Causes

1. Incompatibility between Kover 0.8.2 and Gradle TestKit for Gradle 8.7+
2. Kover's instrumentation potentially interfering with Gradle TestKit's classpath or runtime behavior
3. Possible issues with how Kover interacts with subprojects in a multi-project build
4. Incorrect or incomplete configuration of Kover for functional tests
5. Missing configuration options in the Kover aggregation plugin

Questions

1. Is this a known issue with Kover 0.8.2 when used alongside Gradle TestKit in Gradle 8.7+?
2. Are there specific configuration options for Kover that need to be adjusted for compatibility with newer Gradle versions?
3. Could the application of Kover in settings.gradle.kts or directly in the project be causing unexpected behavior in subprojects or functional tests?
4. Are there any known workarounds or alternative approaches to achieve code coverage in Gradle plugin functional tests?
5. Is the added configuration for functional test source set correct and sufficient for the project-level plugin?
6. Is there a plan to add the missing configuration options to the Kover aggregation plugin to support functional test coverage?

Additional Information

• This issue is blocking our ability to measure code coverage while ensuring compatibility with the latest Gradle versions.
• The problem occurs both when using the Kover aggregation plugin in settings.gradle.kts and when applying the Kover plugin directly in a project.
• The error messages suggest that the plugin extension is not being loaded correctly when Kover is active in newer Gradle versions.
• The Kover aggregation plugin is missing the following configuration, which is necessary for including functional tests in coverage reports:

  kover {
      currentProject {
          sourceSets.add(functionalTestSourceSet)
          sources {
              excludedSourceSets.add(functionalTestSourceSet.name)
          }
      }
  }

  This configuration is available in the project-level plugin but not in the aggregation plugin, making it impossible to properly configure coverage for functional tests when using the aggregation plugin.

build.gradle.kts.txt
settings.gradle.kts.txt
libs.versions.toml.txt
GradleFunctionalTestPluginFunctionalTest.kt.txt
GradleFunctionalTestPlugin.kt.txt
GradleFunctionalTestPluginTest.kt.txt