Improve gradleCheckFlakyTestDetector library

Signed-off-by: Prudhvi Godithi <[email protected]>

README.md

@@ -38,10 +38,13 @@ lib = library(identifier: 'jenkins@<tag>', retriever: modernSCM([
 | [publishToRubyGems.groovy](./vars/publishToRubyGems.groovy)                                                 | A library to publish gems to [rubygems.org](https://rubygems.org/) with [opensearchproject](https://rubygems.org/profiles/opensearchproject) as the owner. Please note that this library expects the gems to be pre-signed. You can use [PublishToRubyGemsLibTester](./tests/jenkins/lib-testers/PublishToRubyGemsLibTester.groovy) to add tests in your repository. See how to use the lib in your [jenkinsFile](./tests/jenkins/jobs/PublishToRubyGems_JenkinsFile).                                                                 |
 | [publishToMaven.groovy](./vars/publishToMaven.groovy)                                                       | A library to sign and deploy opensearch maven artifacts to sonatype staging repository, it also has an optional parameter `autoPublish` to auto-release artifacts from staging repo to prod without manual intervention. You can use [PublishToMavenLibTester](./tests/jenkins/lib-testers/PublishToMavenLibTester.groovy) to add tests in your repository. See how to use the lib in your [jenkinsFile](./tests/jenkins/jobs/PublishToMaven_JenkinsFile).                                                                                  |
 | [publishToNuget.groovy](./vars/publishToNuget.groovy)                                                       | A library to build, sign and publish dotnet artifacts to [Nuget Gallery](https://www.nuget.org/). Please check if the [default docker](https://github.com/opensearch-project/opensearch-build/blob/main/docker/ci/dockerfiles/current/release.centos.clients.x64.arm64.dockerfile) file contains the required dotnet sdk. You can use [PublishToNugetLibTester](./tests/jenkins/lib-testers/PublishToNugetLibTester.groovy) to add tests in your repository. See how to use the lib in your [jenkinsFile](./tests/jenkins/jobs/PublishToNuget_Jenkinsfile).
-| [publishToArtifactsProdBucket.groovy](./vars/publishToArtifactsProdBucket.groovy)                                                       | This library signs and uploads the artifacts to production S3 bucket which points to artifacts.opensearch.org. Please make sure the role that you use to upload exists and has the right permission. For artifacts of different types like macos, linux and windows, call this lib for each artifact with different signing parameters. You can use [PublishToArtifactsProdBucketLibTester](./tests/jenkins/lib-testers/PublishToArtifactsProdBucketLibTester.groovy) to add tests in your repository. See how to use the lib in your [jenkinsFile](./tests/jenkins/jobs/PublishToArtifactsProdBucket_Jenkinsfile). 
+| [publishToArtifactsProdBucket.groovy](./vars/publishToArtifactsProdBucket.groovy)                                                     | This library signs and uploads the artifacts to production S3 bucket which points to artifacts.opensearch.org. Please make sure the role that you use to upload exists and has the right permission. For artifacts of different types like macos, linux and windows, call this lib for each artifact with different signing parameters. You can use [PublishToArtifactsProdBucketLibTester](./tests/jenkins/lib-testers/PublishToArtifactsProdBucketLibTester.groovy) to add tests in your repository. See how to use the lib in your [jenkinsFile](./tests/jenkins/jobs/PublishToArtifactsProdBucket_Jenkinsfile). 
 | [buildMessage.groovy](./vars/buildMessage.groovy)                                                       | This library that can parse the jenkins build log based on the user defined input query string. 
-| [closeBuildSuccessGithubIssue.groovy](./vars/closeBuildSuccessGithubIssue.groovy)                                                       | This library that identifies the successfully built components and closes the created [AUTOCUT] issues.
+| [closeBuildSuccessGithubIssue.groovy](./vars/closeBuildSuccessGithubIssue.groovy)                                                     | This library that identifies the successfully built components and closes the created [AUTOCUT] issues.
 | [createGithubIssue.groovy](./vars/createGithubIssue.groovy)                                                       | This library that identifies the failed components and creates the [AUTOCUT] issues.
+| [publishGradleCheckTestResults.groovy](./vars/publishGradleCheckTestResults.groovy)                                                       | This library runs part of Gradle Check and publishes the failed test data to the [OpenSearch Metrics Cluster](https://metrics.opensearch.org/_dashboards/app/dashboards#/view/e5e64d40-ed31-11ee-be99-69d1dbc75083).
+| [gradleCheckFlakyTestDetector.groovy](./vars/gradleCheckFlakyTestDetector.groovy)                                                       | This library detects the flaky tests from [OpenSearch Metrics Cluster](https://metrics.opensearch.org/_dashboards/app/dashboards#/view/e5e64d40-ed31-11ee-be99-69d1dbc75083) and generates a test report.
+| [gradleCheckFlakyTestGitHubIssue.groovy](./vars/gradleCheckFlakyTestGitHubIssue.groovy)                                                       | This library is used in [gradleCheckFlakyTestDetector.groovy](./vars/gradleCheckFlakyTestDetector.groovy) to create/edit the GitHub Issue using the generated test report.
 
 ## Contributing
 

src/gradlecheck/FetchPostMergeFailedTestClass.groovy

@@ -27,7 +27,7 @@ class FetchPostMergeFailedTestClass {
         this.script = script
     }
 
-    def getQuery() {
+    def getQuery(timeFrame) {
         def queryMap = [
                 size: 200,
                 query: [
@@ -52,11 +52,33 @@ class FetchPostMergeFailedTestClass {
                                                 match: [
                                                         test_status: [
                                                                 query: "FAILED",
-                                                                operator: "OR"
+                                                                operator: "OR",
+                                                                prefix_length: 0,
+                                                                max_expansions: 50,
+                                                                fuzzy_transpositions: true,
+                                                                lenient: false,
+                                                                zero_terms_query: "NONE",
+                                                                auto_generate_synonyms_phrase_query: true,
+                                                                boost: 1
                                                         ]
                                                 ]
                                         ]
-                                ]
+                                ],
+                                filter: [
+                                        [
+                                                range: [
+                                                        build_start_time: [
+                                                                from: "now-${timeFrame}",
+                                                                to: "now",
+                                                                include_lower: true,
+                                                                include_upper: true,
+                                                                boost: 1
+                                                        ]
+                                                ]
+                                        ]
+                                ],
+                                adjust_pure_negative: true,
+                                boost: 1
                         ]
                 ],
                 aggregations: [
@@ -68,13 +90,12 @@ class FetchPostMergeFailedTestClass {
                         ]
                 ]
         ]
-
         def query = JsonOutput.toJson(queryMap)
         return query.replace('"', '\\"')
     }
 
-    def getPostMergeFailedTestClass() {
-         def jsonResponse = new OpenSearchMetricsQuery(metricsUrl,awsAccessKey, awsSecretKey, awsSessionToken, script).fetchMetrics(getQuery())
+    def getPostMergeFailedTestClass(timeFrame) {
+         def jsonResponse = new OpenSearchMetricsQuery(metricsUrl,awsAccessKey, awsSecretKey, awsSessionToken, script).fetchMetrics(getQuery(timeFrame))
          def keys = jsonResponse.aggregations.test_class_keyword_agg.buckets.collect { it.key }
          return keys
     }

src/gradlecheck/MarkdownComparator.groovy

@@ -0,0 +1,33 @@
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package gradlecheck
+
+class MarkdownComparator {
+
+    ArrayList<String> testReportMarkDownTable
+    ArrayList<String> gitHubMarkDownTable
+
+    MarkdownComparator(ArrayList<String> testReportMarkDownTable, ArrayList<String> gitHubMarkDownTable) {
+        this.testReportMarkDownTable = testReportMarkDownTable
+        this.gitHubMarkDownTable = gitHubMarkDownTable
+    }
+
+    def markdownComparison() {
+        def differences = testReportMarkDownTable.findAll { ghRow ->
+            !gitHubMarkDownTable.any { compRow ->
+                ghRow['Git Reference'] == compRow['Git Reference'] &&
+                        ghRow['Merged Pull Request'] == compRow['Merged Pull Request'] &&
+                        ghRow['Build Details'] == compRow['Build Details'] &&
+                        ghRow['Test Name'] == compRow['Test Name']
+            }
+        }
+        return differences
+    }
+}

src/gradlecheck/OpenSearchMetricsQuery.groovy

@@ -26,14 +26,14 @@ class OpenSearchMetricsQuery {
         this.script = script
     }
 
+    // Ensure the alias `gradle-check` is created targeting all the gradle-check-* indices.
     def fetchMetrics(String query) {
         def response = script.sh(
             script: """
             set -e
             set +x
             MONTH_YEAR=\$(date +"%m-%Y")
-            INDEX_NAME="gradle-check-\$MONTH_YEAR"
-            curl -s -XGET "${metricsUrl}/\$INDEX_NAME/_search" --aws-sigv4 "aws:amz:us-east-1:es" --user "${awsAccessKey}:${awsSecretKey}" -H "x-amz-security-token:${awsSessionToken}" -H 'Content-Type: application/json' -d "${query}" | jq '.'
+            curl -s -XGET "${metricsUrl}/gradle-check/_search" --aws-sigv4 "aws:amz:us-east-1:es" --user "${awsAccessKey}:${awsSecretKey}" -H "x-amz-security-token:${awsSessionToken}" -H 'Content-Type: application/json' -d "${query}" | jq '.'
         """,
                 returnStdout: true
         ).trim()

src/gradlecheck/ParseMarkDownTable.groovy

@@ -0,0 +1,32 @@
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package gradlecheck
+
+class ParseMarkDownTable {
+    String markdown
+
+    ParseMarkDownTable(String markdown) {
+        this.markdown = markdown
+    }
+
+    def parseMarkdownTableRows() {
+        def rows = markdown.split("\\|\\s*\\n")
+        rows = rows[2..-2] // Skipping headers and footer
+        return rows.collect { row ->
+            def cells = row.split("\\|\\s*")
+            return [
+                    'Git Reference': cells[1].trim(),
+                    'Merged Pull Request': cells[2].trim(),
+                    'Build Details': cells[3].trim(),
+                    'Test Name': cells[4].trim()
+            ]
+        }
+    }
+}

tests/gradlecheck/CreateMarkDownTableTest.groovy

@@ -22,11 +22,8 @@ class CreateMarkDownTableTest  {
                 [gitReference: "def456", pullRequestLink: "https://github.com/opensearch-project/OpenSearch/pull/2", buildDetailLink: "https://ci.opensearch.org/2", testNames: ["test3"]]
         ]
         def additionalPullRequests = ["3", "4"]
-
         def createMarkDownTable = new CreateMarkDownTable(failedTest, tableData, additionalPullRequests)
-
         def result = createMarkDownTable.createMarkdownTable()
-
         def expectedOutput = """
 ## Flaky Test Report for `ExampleTest`
 

tests/gradlecheck/FetchPostMergeFailedTestClassTest.groovy

@@ -71,11 +71,33 @@ class FetchPostMergeFailedTestClassTest {
                                                 match: [
                                                         test_status: [
                                                                 query: "FAILED",
-                                                                operator: "OR"
+                                                                operator: "OR",
+                                                                prefix_length: 0,
+                                                                max_expansions: 50,
+                                                                fuzzy_transpositions: true,
+                                                                lenient: false,
+                                                                zero_terms_query: "NONE",
+                                                                auto_generate_synonyms_phrase_query: true,
+                                                                boost: 1
                                                         ]
                                                 ]
                                         ]
-                                ]
+                                ],
+                                filter: [
+                                        [
+                                                range: [
+                                                        build_start_time: [
+                                                                from: "now-15d",
+                                                                to: "now",
+                                                                include_lower: true,
+                                                                include_upper: true,
+                                                                boost: 1
+                                                        ]
+                                                ]
+                                        ]
+                                ],
+                                adjust_pure_negative: true,
+                                boost: 1
                         ]
                 ],
                 aggregations: [
@@ -88,16 +110,16 @@ class FetchPostMergeFailedTestClassTest {
                 ]
         ]).replace('"', '\\"')
 
-        def result = fetchPostMergeFailedTestClass.getQuery()
+        def result = fetchPostMergeFailedTestClass.getQuery("15d")
 
         assert result == expectedOutput
     }
 
     @Test
     void testGetPostMergeFailedTestClassReturnsKeys() {
         def expectedOutput = ["testClass1", "testClass2"]
-
-        def result = fetchPostMergeFailedTestClass.getPostMergeFailedTestClass()
+        def timeFrame = "15d"
+        def result = fetchPostMergeFailedTestClass.getPostMergeFailedTestClass(timeFrame)
 
         assert result == expectedOutput
     }

tests/gradlecheck/MarkdownComparatorTest.groovy

@@ -0,0 +1,57 @@
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package gradlecheck;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+class MarkdownComparatorTest {
+
+    @Test
+    public void testMarkdownComparisonWithDifferences() {
+        List<Map<String, String>> testReportMarkdown = new ArrayList<>();
+        testReportMarkdown.add(createRow("abc123", "PR1", "Build1", "Test1"));
+        testReportMarkdown.add(createRow("def456", "PR2", "Build2", "Test2"));
+        List<Map<String, String>> gitHubMarkdown = new ArrayList<>();
+        gitHubMarkdown.add(createRow("abc123", "PR1", "Build1", "Test1"));
+        MarkdownComparator comparator = new MarkdownComparator(testReportMarkdown, gitHubMarkdown);
+        List<Map<String, String>> differences = comparator.markdownComparison();
+        Assert.assertEquals(1, differences.size());
+        Assert.assertEquals("def456", differences.get(0).get("Git Reference"));
+        Assert.assertEquals("PR2", differences.get(0).get("Merged Pull Request"));
+        Assert.assertEquals("Build2", differences.get(0).get("Build Details"));
+        Assert.assertEquals("Test2", differences.get(0).get("Test Name"));
+    }
+
+    @Test
+    public void testMarkdownComparisonWithoutDifferences() {
+        List<Map<String, String>> testReportMarkdown = new ArrayList<>();
+        testReportMarkdown.add(createRow("abc123", "PR1", "Build1", "Test1"));
+        List<Map<String, String>> gitHubMarkdown = new ArrayList<>();
+        gitHubMarkdown.add(createRow("abc123", "PR1", "Build1", "Test1"));
+        MarkdownComparator comparator = new MarkdownComparator(testReportMarkdown, gitHubMarkdown);
+        List<Map<String, String>> differences = comparator.markdownComparison();
+        Assert.assertEquals(0, differences.size());
+    }
+
+    private Map<String, String> createRow(String gitReference, String mergedPR, String buildDetails, String testName) {
+        Map<String, String> row = new HashMap<>();
+        row.put("Git Reference", gitReference);
+        row.put("Merged Pull Request", mergedPR);
+        row.put("Build Details", buildDetails);
+        row.put("Test Name", testName);
+        return row;
+    }
+}

tests/gradlecheck/ParseMarkDownTableTest.groovy

@@ -0,0 +1,45 @@
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package gradlecheck
+
+import org.junit.Test
+import static org.junit.Assert.assertEquals
+
+class ParseMarkDownTableTest {
+
+    @Test
+    void testParseMarkdownTableRows() {
+        def markdown = """
+            ## Flaky Test Report for `PitMultiNodeIT`
+
+            Noticed the `PitMultiNodeIT` has some flaky, failing tests that failed during **post-merge actions**.
+
+            ### Details
+
+            | Git Reference | Merged Pull Request | Build Details | Test Name |
+            |---------------|----------------------|---------------|-----------|
+            | 708c37120bea33f258d1656132b9b05642c92720 | [14502](https://github.com/opensearch-project/OpenSearch/pull/14502) | [41571](https://build.ci.opensearch.org/job/gradle-check/41571/testReport/) | `org.opensearch.search.pit.PitMultiNodeIT.testCreatePitWhileNodeDropWithAllowPartialCreationFalse {p0={"search.concurrent_segment_search.enabled":"false"}}` |
+
+            The other pull requests, besides those involved in post-merge actions, that contain failing tests with the `PitMultiNodeIT` class are:
+
+            - [13801](https://github.com/opensearch-project/OpenSearch/pull/13801)
+            - [14362](https://github.com/opensearch-project/OpenSearch/pull/14362)
+
+            For more details on the failed tests refer to [OpenSearch Gradle Check Metrics](https://metrics.opensearch.org/_dashboards/app/dashboards#/view/e5e64d40-ed31-11ee-be99-69d1dbc75083) dashboard.
+        """
+        ParseMarkDownTable parser = new ParseMarkDownTable(markdown)
+        def result = parser.parseMarkdownTableRows()
+        assertEquals("Expected 1 row in the result", 1, result.size())
+        assertEquals("708c37120bea33f258d1656132b9b05642c92720", result[0]['Git Reference'])
+        assertEquals("[14502](https://github.com/opensearch-project/OpenSearch/pull/14502)", result[0]['Merged Pull Request'])
+        assertEquals("[41571](https://build.ci.opensearch.org/job/gradle-check/41571/testReport/)", result[0]['Build Details'])
+        assertEquals("`org.opensearch.search.pit.PitMultiNodeIT.testCreatePitWhileNodeDropWithAllowPartialCreationFalse {p0={\"search.concurrent_segment_search.enabled\":\"false\"}}`", result[0]['Test Name'])
+    }
+}