Add Java Connectors (with EventHubs) to master (#1154)

* adding Azure EventHubs connector skeleton with config * Azure EventHubs Connector MVP (#1033) Adds Azure EventHubs Connector MVP over Kafka API. Commits: * Azure Connector MVP * added prefix to Config * adressing PR comments pt. 1 * adressing PR comments pt. 2 * addressing PR comments pt. 3 * removed unused files * addressing PR comments pt. 4 * small log issue, getting rid of unused singleton classes * Build KCQL from gradle (#1042) * adding Gradle buildscripts for KCQL * LC-128 adding KCQL to Azure EventHubs and additional properties to Azure * removing mongodb module (it's scrapped) * added org.antlr.* package shadowing to avoid conflicts in kafka connect * adressing PR comments, fixing unit tests * Azure Eventhubs bytes passthrough (#1047) * Abstracting data that's being passed through connector to Bytes * cleanup, additional tests * picking the first KCQL (in case there are more) * delegated KCQL properties handling to ProducerProvider * PR comments * PR comments pt.2 * added java-specific .gitignore * missing import * renaming producer to better indicate its purpose * Lc 136 eventhubs get output input topics from kcql (#1058) * getting input/output from KCQL initial impl * removing small issues (we don't need deserializers configs) * codacy check * addressing PR comments pt.1 * cleanup of multiple output topics * remove 'java' part from package name (#1103) * remove 'java' part from package name * comments and remarks pt. 1 * eventub connector to support multiple kcqls (#1108) * support multiple KCQLs * remove 'java' part from package name * merged latest changes, added multi-KCQL support * fixing bug aroud regex check for topic name, added unit tests * build scripts (in order to merge PRs) * additional tests * Added KCQL parsing and validation to Connector class * addressing PR comments pt. 1 * Using connector name as consumer group (#1109) * Using connector name as consumer group * not exposing ignored ConsumerProperties in ConfigDef not exposing ignored ConsumerProperties in ConfigDef (as connector sets it by itself) * file movements * GitHub actions for java connectors (#1147) * preparing modules list in Gradle * addung workflow for build scripts * adding Java release pipeline, small regex changes * JDK to 17 for full-support * pushing version to 6.4.0-SNAPSHOT to keep them in sync with sbt * add dependency check to java build workflow * removed KCQL from java release (as it's built by sbt) * removed unnecessary parameter * fix dependency check * push java test results to subdirectory * run build on push branch * run actions on push * remove build workflow run on every push * fix github warnings for deprecated functions * addressing java-matrix output being reset with scala artifacts * creating separate output from build script 'java_modules' * Feat/java license headers (#1160) * add java headers config * add license headers to source files * add missing HEADER file * this commit should fail (no header in one of the tests) * fixing commit plus including info about check in github actions * addressing PR comments pt. 1 * PR comments pt.2 * added assertj dependancy for unit tests * fix build script * fixed failing test * addressing PR comments pt.3 * adressed PR comments pt.4 * adressed PR comments pt.5 * Refactoring for simplicity (#1170) * Refactoring for simplicity Method Extraction: Extracted repetitive setup code into helper methods like mockKeyValueTypes, mockByteBlockingProducer, mockConsumerRecord, mockEmptyHeaders, and mockRecordsQueue. Parameterized Methods: Made the mockRecordsQueue method varargs to accept variable numbers of input topics for better reusability. Local Variable Removal: Removed unnecessary local variables like emptyHeaderList and directly invoked Collections.emptyIterator() where needed. Inline Initialization: Initialized recordsQueue directly in the test methods instead of setting it up in setUp method, as it's only used within the scope of those methods. Stream Processing: Utilized Java Streams to simplify the creation of consumerRecordList in the mockRecordsQueue method. Simplified Assertions: Removed unnecessary assertions and verifications to reduce clutter and improve readability. * small optimizations (imports, constants, formatting) --------- Co-authored-by: Mati Urban <[email protected]> --------- Co-authored-by: Mati Urban <[email protected]> Co-authored-by: David Sloan <[email protected]>
lensesio · Apr 23, 2024 · 5e77256 · 5e77256
1 parent e916b35
commit 5e77256
Show file tree

Hide file tree

Showing 56 changed files with 4,107 additions and 117 deletions.
diff --git a/.github/workflows/java-build.yml b/.github/workflows/java-build.yml
@@ -0,0 +1,135 @@
+name: CI-java
+on:
+  pull_request:
+  workflow_call:
+    inputs:
+      version:
+        required: true
+        type: string
+    outputs:
+      java_modules:
+        description: "Stream reactor collection of java modules"
+        value: ${{ jobs.initiate-java-modules.outputs.java_matrix }}
+
+jobs:
+
+  initiate-java-modules:
+    timeout-minutes: 5
+    runs-on: ubuntu-latest
+    outputs:
+      java_matrix: ${{ steps.java-mods.outputs.java-matrix }}
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up JDK 17
+        uses: actions/setup-java@v4
+        with:
+          java-version: '17'
+          distribution: 'temurin'
+          cache: 'gradle'
+      - name: Generate modules lists
+        run: cd 'java-connectors' && ./gradlew releaseModuleList
+        env:
+          JVM_OPTS: -Xmx512m
+      - name: Read java modules lists
+        id: java-mods
+        run: |
+          echo "java-matrix=$(cat ./java-connectors/gradle-modules.txt)" >> $GITHUB_OUTPUT
+
+  test:
+    needs:
+      - initiate-java-modules
+    strategy:
+      matrix:
+        module: ${{fromJSON(needs.initiate-java-modules.outputs.java_matrix)}}
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up JDK 17
+        uses: actions/setup-java@v4
+        with:
+          java-version: 17
+          distribution: 'temurin'
+
+      - name: Setup Gradle
+        uses: gradle/actions/setup-gradle@v3
+        with:
+          gradle-version: 8.6
+
+      - name: Check License Headers and Test with Gradle
+        run: cd 'java-connectors' && ./gradlew ${{ matrix.module }}:test
+
+  build-and-cache:
+    needs:
+      - test
+      - initiate-java-modules
+    strategy:
+      matrix:
+        module: ${{fromJSON(needs.initiate-java-modules.outputs.java_matrix)}}
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up JDK 17
+        uses: actions/setup-java@v4
+        with:
+          java-version: 17
+          distribution: 'temurin'
+          cache: gradle
+
+      - name: Setup Gradle
+        uses: gradle/actions/setup-gradle@v3
+        with:
+          gradle-version: 8.6
+
+      - name: Execute Gradle build
+        run: cd 'java-connectors' && ./gradlew ${{ matrix.module }}:shadowJar --scan
+
+      - name: Move to release folder
+        shell: bash
+        run: |
+          JAVA_RELEASE_FOLDER=java-connectors/release
+          JAVA_BUILD_FOLDER=java-connectors/${{ matrix.module }}/build/libs
+          mkdir -p $JAVA_RELEASE_FOLDER
+          cp $JAVA_BUILD_FOLDER/${{ matrix.module }}*.jar LICENSE $JAVA_RELEASE_FOLDER/
+
+      - name: Cache assembly
+        uses: actions/cache/save@v4
+        with:
+          path: ./java-connectors/release/${{ matrix.module }}*.jar
+          key: assembly-java-${{ github.run_id }}
+
+  jar-dependency-check:
+    needs:
+      - build-and-cache
+      - initiate-java-modules
+    timeout-minutes: 30
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        module: ${{fromJSON(needs.initiate-java-modules.outputs.java_matrix)}}
+    steps:
+      - name: Restore assembly
+        uses: actions/cache/restore@v4
+        with:
+          path: ./java-connectors/release/${{ matrix.module }}*.jar
+          key: assembly-java-${{ github.run_id }}
+          fail-on-cache-miss: true
+
+      - name: Get branch names.
+        id: branch_name
+        uses: tj-actions/branch-names@v8
+      - name: JAR Dependency Check
+        uses: dependency-check/Dependency-Check_Action@main
+        with:
+          project: kafka-connect-${{matrix.module}}-deps
+          path: ./java-connectors/release/${{ matrix.module }}*.jar
+          format: 'HTML'
+          args: >-
+            --failOnCVSS 5
+            --suppression https://raw.githubusercontent.com/${{ github.event.pull_request.head.repo.owner.login }}/${{github.event.repository.name}}/${{ steps.branch_name.outputs.tag }}${{ steps.branch_name.outputs.current_branch }}/suppression.xml
+
+      - name: Upload Test results
+        uses: actions/upload-artifact@master
+        with:
+          name: ${{matrix.module}}-depcheck-results
+          path: ${{github.workspace}}/reports
diff --git a/.github/workflows/java-release.yml b/.github/workflows/java-release.yml
@@ -0,0 +1,95 @@
+name: Publish New Java Release
+on:
+  push:
+    tags:
+      - "*"
+  workflow_dispatch:
+
+jobs:
+  validate-tag:
+    runs-on: ubuntu-latest
+    outputs:
+      draft_release: ${{ steps.get_tag.outputs.draft_release }}
+      tag: ${{ steps.get_tag.outputs.tag }}
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Get tag, release mode
+        shell: bash
+        id: get_tag
+        run: |
+          if [[ ${GITHUB_REF##*/} =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]];
+          then
+              draft_release=false
+          elif [[ ${GITHUB_REF##*/} =~ ^[0-9]+\.[0-9]+\.[0-9]+(-(alpha|beta|rc)(\.[0-9]+)?)?(\+[A-Za-z0-9.]+)?$ ]];
+          then
+              draft_release=true
+          else
+              echo "Exiting, github ref needs to be a tag with format x.y.z or x.y.z-(alpha|beta|rc)"
+              exit 1
+          fi
+          echo "draft_release=$draft_release" >> $GITHUB_OUTPUT
+          echo "tag=${GITHUB_REF##*/}" >> $GITHUB_OUTPUT
+
+  build:
+    needs:
+      - validate-tag
+    uses: ./.github/workflows/java-build.yml
+    with:
+      version: ${{ needs.validate-tag.outputs.tag }}
+    secrets: inherit
+
+  create-release:
+    runs-on: ubuntu-latest
+    needs:
+      - validate-tag
+      - build
+    strategy:
+      # Avoid parallel uploads
+      max-parallel: 1
+      # GitHub will NOT cancel all in-progress and queued jobs in the matrix if any job in the matrix fails, which could create inconsistencies.
+      # If any matrix job fails, the job will be marked as failure
+      fail-fast: false
+      matrix:
+        module: ${{fromJSON(needs.build.outputs.java_modules)}}
+    env:
+      DRAFT_RELEASE: '${{ needs.validate-tag.outputs.draft_release }}'
+      TAG: ${{ needs.validate-tag.outputs.tag }}
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+      - name: Set up JDK 17
+        uses: actions/setup-java@v4
+        with:
+          java-version: '17'
+          distribution: 'temurin'
+          cache: 'gradle'
+
+      - name: Uncache assembly
+        uses: actions/cache/restore@v4
+        with:
+          path: |
+            ./java-connectors/release/${{ matrix.module }}*.jar
+          key: assembly-java-${{ github.run_id }}
+          fail-on-cache-miss: true
+
+      - name: Package Connector
+        shell: bash
+        run: |
+          JAVA_RELEASE_FOLDER=java-connectors/release
+          FOLDER=${{ matrix.module }}-${{ env.TAG }}
+          mkdir -p $FOLDER
+          cp $JAVA_RELEASE_FOLDER/${{ matrix.module }}*.jar LICENSE $FOLDER/
+          zip -r "$FOLDER.zip" $FOLDER/
+
+      - name: Upload binaries to release
+        uses: svenstaro/upload-release-action@v2
+        with:
+          file: ${{ matrix.module }}-${{ env.TAG }}.zip
+          asset_name: "${{ matrix.module }}-${{ env.TAG }}.zip"
+          release_name: 'Stream Reactor ${{ env.TAG }}'
+          prerelease: ${{ env.DRAFT_RELEASE }}
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -22,14 +22,14 @@ jobs:
         shell: bash
         id: get_tag
         run: |
-          if [[ ${GITHUB_REF##*/} =~ ^[0-9]\.[0-9]\.[0-9]$ ]];
+          if [[ ${GITHUB_REF##*/} =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]];
           then
               draft_release=false
-          elif [[ ${GITHUB_REF##*/} =~ ^[0-9]\.[0-9]\.[0-9]+(-(alpha|beta|rc)(\.[0-9]+)?)?(\+[A-Za-z0-9.]+)?$ ]];
+          elif [[ ${GITHUB_REF##*/} =~ ^[0-9]+\.[0-9]+\.[0-9]+(-(alpha|beta|rc)(\.[0-9]+)?)?(\+[A-Za-z0-9.]+)?$ ]];
           then
               draft_release=true
           else
-              echo "Exiting, github ref needs to be a tag with format x.y.z or x.y.z+(alpha|beta|rc)"
+              echo "Exiting, github ref needs to be a tag with format x.y.z or x.y.z-(alpha|beta|rc)"
               exit 1
           fi
           echo "draft_release=$draft_release" >> $GITHUB_OUTPUT

diff --git a/.gitignore b/.gitignore
@@ -1,3 +1,6 @@
+#Including java-specific ignores
+#!include:java-connectors/.gitignore
+
 .bsp
 stageman
 cass-test

diff --git a/java-connectors/.gitignore b/java-connectors/.gitignore
@@ -0,0 +1,47 @@
+.gradle
+build/
+!gradle/wrapper/gradle-wrapper.jar
+!**/src/main/**/build/
+!**/src/test/**/build/
+
+### IntelliJ IDEA ###
+.idea/*
+.idea/modules.xml
+.idea/jarRepositories.xml
+.idea/compiler.xml
+.idea/libraries/
+*.iws
+*.iml
+*.ipr
+out/
+!**/src/main/**/out/
+!**/src/test/**/out/
+
+### Eclipse ###
+.apt_generated
+.classpath
+.factorypath
+.project
+.settings
+.springBeans
+.sts4-cache
+bin/
+!**/src/main/**/bin/
+!**/src/test/**/bin/
+
+### NetBeans ###
+/nbproject/private/
+/nbbuild/
+/dist/
+/nbdist/
+/.nb-gradle/
+
+### VS Code ###
+.vscode/
+
+### Mac OS ###
+.DS_Store
+
+### Lenses-specific ###
+release/
+gradle-modules.txt
diff --git a/java-connectors/HEADER.txt b/java-connectors/HEADER.txt
@@ -0,0 +1,13 @@
+Copyright 2017-${year} ${name} Ltd
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.