From 5e77256760421286621ce65b30add823cf4b59fb Mon Sep 17 00:00:00 2001 From: Mati Urban <157909548+GoMati-MU@users.noreply.github.com> Date: Tue, 23 Apr 2024 14:10:07 +0200 Subject: [PATCH] Add Java Connectors (with EventHubs) to master (#1154) * adding Azure EventHubs connector skeleton with config * Azure EventHubs Connector MVP (#1033) Adds Azure EventHubs Connector MVP over Kafka API. Commits: * Azure Connector MVP * added prefix to Config * adressing PR comments pt. 1 * adressing PR comments pt. 2 * addressing PR comments pt. 3 * removed unused files * addressing PR comments pt. 4 * small log issue, getting rid of unused singleton classes * Build KCQL from gradle (#1042) * adding Gradle buildscripts for KCQL * LC-128 adding KCQL to Azure EventHubs and additional properties to Azure * removing mongodb module (it's scrapped) * added org.antlr.* package shadowing to avoid conflicts in kafka connect * adressing PR comments, fixing unit tests * Azure Eventhubs bytes passthrough (#1047) * Abstracting data that's being passed through connector to Bytes * cleanup, additional tests * picking the first KCQL (in case there are more) * delegated KCQL properties handling to ProducerProvider * PR comments * PR comments pt.2 * added java-specific .gitignore * missing import * renaming producer to better indicate its purpose * Lc 136 eventhubs get output input topics from kcql (#1058) * getting input/output from KCQL initial impl * removing small issues (we don't need deserializers configs) * codacy check * addressing PR comments pt.1 * cleanup of multiple output topics * remove 'java' part from package name (#1103) * remove 'java' part from package name * comments and remarks pt. 1 * eventub connector to support multiple kcqls (#1108) * support multiple KCQLs * remove 'java' part from package name * merged latest changes, added multi-KCQL support * fixing bug aroud regex check for topic name, added unit tests * build scripts (in order to merge PRs) * additional tests * Added KCQL parsing and validation to Connector class * addressing PR comments pt. 1 * Using connector name as consumer group (#1109) * Using connector name as consumer group * not exposing ignored ConsumerProperties in ConfigDef not exposing ignored ConsumerProperties in ConfigDef (as connector sets it by itself) * file movements * GitHub actions for java connectors (#1147) * preparing modules list in Gradle * addung workflow for build scripts * adding Java release pipeline, small regex changes * JDK to 17 for full-support * pushing version to 6.4.0-SNAPSHOT to keep them in sync with sbt * add dependency check to java build workflow * removed KCQL from java release (as it's built by sbt) * removed unnecessary parameter * fix dependency check * push java test results to subdirectory * run build on push branch * run actions on push * remove build workflow run on every push * fix github warnings for deprecated functions * addressing java-matrix output being reset with scala artifacts * creating separate output from build script 'java_modules' * Feat/java license headers (#1160) * add java headers config * add license headers to source files * add missing HEADER file * this commit should fail (no header in one of the tests) * fixing commit plus including info about check in github actions * addressing PR comments pt. 1 * PR comments pt.2 * added assertj dependancy for unit tests * fix build script * fixed failing test * addressing PR comments pt.3 * adressed PR comments pt.4 * adressed PR comments pt.5 * Refactoring for simplicity (#1170) * Refactoring for simplicity Method Extraction: Extracted repetitive setup code into helper methods like mockKeyValueTypes, mockByteBlockingProducer, mockConsumerRecord, mockEmptyHeaders, and mockRecordsQueue. Parameterized Methods: Made the mockRecordsQueue method varargs to accept variable numbers of input topics for better reusability. Local Variable Removal: Removed unnecessary local variables like emptyHeaderList and directly invoked Collections.emptyIterator() where needed. Inline Initialization: Initialized recordsQueue directly in the test methods instead of setting it up in setUp method, as it's only used within the scope of those methods. Stream Processing: Utilized Java Streams to simplify the creation of consumerRecordList in the mockRecordsQueue method. Simplified Assertions: Removed unnecessary assertions and verifications to reduce clutter and improve readability. * small optimizations (imports, constants, formatting) --------- Co-authored-by: Mati Urban --------- Co-authored-by: Mati Urban Co-authored-by: David Sloan <33483659+davidsloan@users.noreply.github.com> --- .github/workflows/java-build.yml | 135 +++++++ .github/workflows/java-release.yml | 95 +++++ .github/workflows/release.yml | 6 +- .gitignore | 3 + java-connectors/.gitignore | 47 +++ java-connectors/HEADER.txt | 13 + java-connectors/build.gradle | 177 ++++++++ .../config/checkstyle/checkstyle.xml | 379 ++++++++++++++++++ .../config/checkstyle/suppressions.xml | 11 + .../gradle/wrapper/gradle-wrapper.jar | Bin 0 -> 60756 bytes .../gradle/wrapper/gradle-wrapper.properties | 6 + java-connectors/gradlew | 234 +++++++++++ java-connectors/gradlew.bat | 89 ++++ .../build.gradle | 19 + .../config/AzureEventHubsConfigConstants.java | 50 +++ .../config/AzureEventHubsSourceConfig.java | 122 ++++++ .../eventhubs/config/SourceDataType.java | 69 ++++ .../eventhubs/mapping/SourceRecordMapper.java | 80 ++++ .../AzureConsumerRebalancerListener.java | 80 ++++ .../source/AzureEventHubsSourceConnector.java | 90 +++++ .../source/AzureEventHubsSourceTask.java | 109 +++++ .../source/BlockingQueueProducer.java | 37 ++ .../source/BlockingQueueProducerProvider.java | 104 +++++ .../EventHubsKafkaConsumerController.java | 109 +++++ .../KafkaByteBlockingQueuedProducer.java | 123 ++++++ .../eventhubs/source/ProducerProvider.java | 31 ++ .../source/TopicPartitionOffsetProvider.java | 91 +++++ .../eventhubs/util/KcqlConfigTopicMapper.java | 77 ++++ ...pache.kafka.connect.source.SourceConnector | 1 + .../main/resources/azure-eventhubs-ascii.txt | 14 + .../eventhubs/config/SourceDataTypeTest.java | 60 +++ .../mapping/SourceRecordMapperTest.java | 125 ++++++ .../AzureConsumerRebalancerListenerTest.java | 109 +++++ .../AzureEventHubsSourceConnectorTest.java | 87 ++++ .../source/AzureEventHubsSourceTaskTest.java | 144 +++++++ .../BlockingQueueProducerProviderTest.java | 108 +++++ .../EventHubsKafkaConsumerControllerTest.java | 183 +++++++++ .../KafkaByteBlockingQueuedProducerTest.java | 51 +++ .../TopicPartitionOffsetProviderTest.java | 109 +++++ .../util/KcqlConfigTopicMapperTest.java | 108 +++++ .../kafka-connect-common/build.gradle | 17 + .../common/config/base/BaseConfig.java | 33 ++ .../config/base/intf/ConnectorPrefixed.java | 23 ++ .../exception/ConnectorStartupException.java | 26 ++ .../InputStreamExtractionException.java | 26 ++ .../common/util/AsciiArtPrinter.java | 49 +++ .../common/util/InputStreamHandler.java | 54 +++ .../common/util/JarManifest.java | 142 +++++++ .../src/main/resources/logback.xml | 27 ++ .../common/util/AsciiArtPrinterTest.java | 79 ++++ .../common/util/JarManifestTest.java | 82 ++++ .../src/test/resources/testingAsciiArt.txt | 5 + .../kafka-connect-query-language/.gitignore | 28 -- .../kafka-connect-query-language/build.gradle | 57 +++ .../checkstyle/checkstyle.xml | 86 ---- java-connectors/settings.gradle | 5 + 56 files changed, 4107 insertions(+), 117 deletions(-) create mode 100644 .github/workflows/java-build.yml create mode 100644 .github/workflows/java-release.yml create mode 100644 java-connectors/.gitignore create mode 100644 java-connectors/HEADER.txt create mode 100644 java-connectors/build.gradle create mode 100644 java-connectors/config/checkstyle/checkstyle.xml create mode 100644 java-connectors/config/checkstyle/suppressions.xml create mode 100644 java-connectors/gradle/wrapper/gradle-wrapper.jar create mode 100644 java-connectors/gradle/wrapper/gradle-wrapper.properties create mode 100755 java-connectors/gradlew create mode 100644 java-connectors/gradlew.bat create mode 100644 java-connectors/kafka-connect-azure-eventhubs/build.gradle create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/config/AzureEventHubsConfigConstants.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/config/AzureEventHubsSourceConfig.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/config/SourceDataType.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/mapping/SourceRecordMapper.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureConsumerRebalancerListener.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureEventHubsSourceConnector.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureEventHubsSourceTask.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/BlockingQueueProducer.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/BlockingQueueProducerProvider.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/EventHubsKafkaConsumerController.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/KafkaByteBlockingQueuedProducer.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/ProducerProvider.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/TopicPartitionOffsetProvider.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/util/KcqlConfigTopicMapper.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/main/resources/META-INF/services/org.apache.kafka.connect.source.SourceConnector create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/main/resources/azure-eventhubs-ascii.txt create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/config/SourceDataTypeTest.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/mapping/SourceRecordMapperTest.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureConsumerRebalancerListenerTest.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureEventHubsSourceConnectorTest.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureEventHubsSourceTaskTest.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/BlockingQueueProducerProviderTest.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/EventHubsKafkaConsumerControllerTest.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/KafkaByteBlockingQueuedProducerTest.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/TopicPartitionOffsetProviderTest.java create mode 100644 java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/util/KcqlConfigTopicMapperTest.java create mode 100644 java-connectors/kafka-connect-common/build.gradle create mode 100644 java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/config/base/BaseConfig.java create mode 100644 java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/config/base/intf/ConnectorPrefixed.java create mode 100644 java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/exception/ConnectorStartupException.java create mode 100644 java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/exception/InputStreamExtractionException.java create mode 100644 java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/util/AsciiArtPrinter.java create mode 100644 java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/util/InputStreamHandler.java create mode 100644 java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/util/JarManifest.java create mode 100644 java-connectors/kafka-connect-common/src/main/resources/logback.xml create mode 100644 java-connectors/kafka-connect-common/src/test/java/io/lenses/streamreactor/common/util/AsciiArtPrinterTest.java create mode 100644 java-connectors/kafka-connect-common/src/test/java/io/lenses/streamreactor/common/util/JarManifestTest.java create mode 100644 java-connectors/kafka-connect-common/src/test/resources/testingAsciiArt.txt delete mode 100644 java-connectors/kafka-connect-query-language/.gitignore create mode 100644 java-connectors/kafka-connect-query-language/build.gradle delete mode 100644 java-connectors/kafka-connect-query-language/checkstyle/checkstyle.xml create mode 100644 java-connectors/settings.gradle diff --git a/.github/workflows/java-build.yml b/.github/workflows/java-build.yml new file mode 100644 index 000000000..04bedeb28 --- /dev/null +++ b/.github/workflows/java-build.yml @@ -0,0 +1,135 @@ +name: CI-java +on: + pull_request: + workflow_call: + inputs: + version: + required: true + type: string + outputs: + java_modules: + description: "Stream reactor collection of java modules" + value: ${{ jobs.initiate-java-modules.outputs.java_matrix }} + +jobs: + + initiate-java-modules: + timeout-minutes: 5 + runs-on: ubuntu-latest + outputs: + java_matrix: ${{ steps.java-mods.outputs.java-matrix }} + steps: + - uses: actions/checkout@v4 + - name: Set up JDK 17 + uses: actions/setup-java@v4 + with: + java-version: '17' + distribution: 'temurin' + cache: 'gradle' + - name: Generate modules lists + run: cd 'java-connectors' && ./gradlew releaseModuleList + env: + JVM_OPTS: -Xmx512m + - name: Read java modules lists + id: java-mods + run: | + echo "java-matrix=$(cat ./java-connectors/gradle-modules.txt)" >> $GITHUB_OUTPUT + + test: + needs: + - initiate-java-modules + strategy: + matrix: + module: ${{fromJSON(needs.initiate-java-modules.outputs.java_matrix)}} + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Set up JDK 17 + uses: actions/setup-java@v4 + with: + java-version: 17 + distribution: 'temurin' + + - name: Setup Gradle + uses: gradle/actions/setup-gradle@v3 + with: + gradle-version: 8.6 + + - name: Check License Headers and Test with Gradle + run: cd 'java-connectors' && ./gradlew ${{ matrix.module }}:test + + build-and-cache: + needs: + - test + - initiate-java-modules + strategy: + matrix: + module: ${{fromJSON(needs.initiate-java-modules.outputs.java_matrix)}} + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Set up JDK 17 + uses: actions/setup-java@v4 + with: + java-version: 17 + distribution: 'temurin' + cache: gradle + + - name: Setup Gradle + uses: gradle/actions/setup-gradle@v3 + with: + gradle-version: 8.6 + + - name: Execute Gradle build + run: cd 'java-connectors' && ./gradlew ${{ matrix.module }}:shadowJar --scan + + - name: Move to release folder + shell: bash + run: | + JAVA_RELEASE_FOLDER=java-connectors/release + JAVA_BUILD_FOLDER=java-connectors/${{ matrix.module }}/build/libs + mkdir -p $JAVA_RELEASE_FOLDER + cp $JAVA_BUILD_FOLDER/${{ matrix.module }}*.jar LICENSE $JAVA_RELEASE_FOLDER/ + + - name: Cache assembly + uses: actions/cache/save@v4 + with: + path: ./java-connectors/release/${{ matrix.module }}*.jar + key: assembly-java-${{ github.run_id }} + + jar-dependency-check: + needs: + - build-and-cache + - initiate-java-modules + timeout-minutes: 30 + runs-on: ubuntu-latest + strategy: + matrix: + module: ${{fromJSON(needs.initiate-java-modules.outputs.java_matrix)}} + steps: + - name: Restore assembly + uses: actions/cache/restore@v4 + with: + path: ./java-connectors/release/${{ matrix.module }}*.jar + key: assembly-java-${{ github.run_id }} + fail-on-cache-miss: true + + - name: Get branch names. + id: branch_name + uses: tj-actions/branch-names@v8 + - name: JAR Dependency Check + uses: dependency-check/Dependency-Check_Action@main + with: + project: kafka-connect-${{matrix.module}}-deps + path: ./java-connectors/release/${{ matrix.module }}*.jar + format: 'HTML' + args: >- + --failOnCVSS 5 + --suppression https://raw.githubusercontent.com/${{ github.event.pull_request.head.repo.owner.login }}/${{github.event.repository.name}}/${{ steps.branch_name.outputs.tag }}${{ steps.branch_name.outputs.current_branch }}/suppression.xml + + - name: Upload Test results + uses: actions/upload-artifact@master + with: + name: ${{matrix.module}}-depcheck-results + path: ${{github.workspace}}/reports \ No newline at end of file diff --git a/.github/workflows/java-release.yml b/.github/workflows/java-release.yml new file mode 100644 index 000000000..4738213cf --- /dev/null +++ b/.github/workflows/java-release.yml @@ -0,0 +1,95 @@ +name: Publish New Java Release +on: + push: + tags: + - "*" + workflow_dispatch: + +jobs: + validate-tag: + runs-on: ubuntu-latest + outputs: + draft_release: ${{ steps.get_tag.outputs.draft_release }} + tag: ${{ steps.get_tag.outputs.tag }} + steps: + - name: Checkout repository + uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Get tag, release mode + shell: bash + id: get_tag + run: | + if [[ ${GITHUB_REF##*/} =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; + then + draft_release=false + elif [[ ${GITHUB_REF##*/} =~ ^[0-9]+\.[0-9]+\.[0-9]+(-(alpha|beta|rc)(\.[0-9]+)?)?(\+[A-Za-z0-9.]+)?$ ]]; + then + draft_release=true + else + echo "Exiting, github ref needs to be a tag with format x.y.z or x.y.z-(alpha|beta|rc)" + exit 1 + fi + echo "draft_release=$draft_release" >> $GITHUB_OUTPUT + echo "tag=${GITHUB_REF##*/}" >> $GITHUB_OUTPUT + + build: + needs: + - validate-tag + uses: ./.github/workflows/java-build.yml + with: + version: ${{ needs.validate-tag.outputs.tag }} + secrets: inherit + + create-release: + runs-on: ubuntu-latest + needs: + - validate-tag + - build + strategy: + # Avoid parallel uploads + max-parallel: 1 + # GitHub will NOT cancel all in-progress and queued jobs in the matrix if any job in the matrix fails, which could create inconsistencies. + # If any matrix job fails, the job will be marked as failure + fail-fast: false + matrix: + module: ${{fromJSON(needs.build.outputs.java_modules)}} + env: + DRAFT_RELEASE: '${{ needs.validate-tag.outputs.draft_release }}' + TAG: ${{ needs.validate-tag.outputs.tag }} + + steps: + - name: Checkout repository + uses: actions/checkout@v4 + - name: Set up JDK 17 + uses: actions/setup-java@v4 + with: + java-version: '17' + distribution: 'temurin' + cache: 'gradle' + + - name: Uncache assembly + uses: actions/cache/restore@v4 + with: + path: | + ./java-connectors/release/${{ matrix.module }}*.jar + key: assembly-java-${{ github.run_id }} + fail-on-cache-miss: true + + - name: Package Connector + shell: bash + run: | + JAVA_RELEASE_FOLDER=java-connectors/release + FOLDER=${{ matrix.module }}-${{ env.TAG }} + mkdir -p $FOLDER + cp $JAVA_RELEASE_FOLDER/${{ matrix.module }}*.jar LICENSE $FOLDER/ + zip -r "$FOLDER.zip" $FOLDER/ + + - name: Upload binaries to release + uses: svenstaro/upload-release-action@v2 + with: + file: ${{ matrix.module }}-${{ env.TAG }}.zip + asset_name: "${{ matrix.module }}-${{ env.TAG }}.zip" + release_name: 'Stream Reactor ${{ env.TAG }}' + prerelease: ${{ env.DRAFT_RELEASE }} \ No newline at end of file diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 956b47966..a6fb59e27 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -22,14 +22,14 @@ jobs: shell: bash id: get_tag run: | - if [[ ${GITHUB_REF##*/} =~ ^[0-9]\.[0-9]\.[0-9]$ ]]; + if [[ ${GITHUB_REF##*/} =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then draft_release=false - elif [[ ${GITHUB_REF##*/} =~ ^[0-9]\.[0-9]\.[0-9]+(-(alpha|beta|rc)(\.[0-9]+)?)?(\+[A-Za-z0-9.]+)?$ ]]; + elif [[ ${GITHUB_REF##*/} =~ ^[0-9]+\.[0-9]+\.[0-9]+(-(alpha|beta|rc)(\.[0-9]+)?)?(\+[A-Za-z0-9.]+)?$ ]]; then draft_release=true else - echo "Exiting, github ref needs to be a tag with format x.y.z or x.y.z+(alpha|beta|rc)" + echo "Exiting, github ref needs to be a tag with format x.y.z or x.y.z-(alpha|beta|rc)" exit 1 fi echo "draft_release=$draft_release" >> $GITHUB_OUTPUT diff --git a/.gitignore b/.gitignore index d5b55b6dc..980db392b 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,6 @@ +#Including java-specific ignores +#!include:java-connectors/.gitignore + .bsp stageman cass-test diff --git a/java-connectors/.gitignore b/java-connectors/.gitignore new file mode 100644 index 000000000..13047a0b9 --- /dev/null +++ b/java-connectors/.gitignore @@ -0,0 +1,47 @@ +.gradle +build/ +!gradle/wrapper/gradle-wrapper.jar +!**/src/main/**/build/ +!**/src/test/**/build/ + +### IntelliJ IDEA ### +.idea/* +.idea/modules.xml +.idea/jarRepositories.xml +.idea/compiler.xml +.idea/libraries/ +*.iws +*.iml +*.ipr +out/ +!**/src/main/**/out/ +!**/src/test/**/out/ + +### Eclipse ### +.apt_generated +.classpath +.factorypath +.project +.settings +.springBeans +.sts4-cache +bin/ +!**/src/main/**/bin/ +!**/src/test/**/bin/ + +### NetBeans ### +/nbproject/private/ +/nbbuild/ +/dist/ +/nbdist/ +/.nb-gradle/ + +### VS Code ### +.vscode/ + +### Mac OS ### +.DS_Store + +### Lenses-specific ### +release/ +gradle-modules.txt \ No newline at end of file diff --git a/java-connectors/HEADER.txt b/java-connectors/HEADER.txt new file mode 100644 index 000000000..8828f9cd2 --- /dev/null +++ b/java-connectors/HEADER.txt @@ -0,0 +1,13 @@ +Copyright 2017-${year} ${name} Ltd + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. \ No newline at end of file diff --git a/java-connectors/build.gradle b/java-connectors/build.gradle new file mode 100644 index 000000000..c7c74b793 --- /dev/null +++ b/java-connectors/build.gradle @@ -0,0 +1,177 @@ +plugins { + id 'com.github.johnrengelman.shadow' version '8.1.1' + id 'org.cadixdev.licenser' version '0.6.1' + id 'java' + id 'java-library' +} + +allprojects { + + group = "io.lenses.streamreactor" + version = "6.4.0-SNAPSHOT" + description = "stream-reactor" + + apply plugin: 'java' + apply plugin: 'java-library' + apply plugin: 'com.github.johnrengelman.shadow' + apply plugin: 'org.cadixdev.licenser' + + java { + setSourceCompatibility(JavaVersion.VERSION_11) + setTargetCompatibility(JavaVersion.VERSION_11) + } + + ext { + //DEPENDENCY VERSIONS + lombokVersion = '1.18.30' + kafkaVersion = '3.7.0' + logbackVersion = '1.4.14' + jUnitVersion = '5.9.1' + mockitoJupiterVersion = '5.10.0' + apacheToConfluentVersionAxis = ["2.8.1": "6.2.2", "3.3.0": "7.3.1"] + + //Other Manifest Info + mainClassName = '' + gitCommitHash = ("git rev-parse HEAD").execute().text.trim() + gitTag = ("git describe --abbrev=0 --tags").execute().text.trim() + gitRepo = ("git remote get-url origin").execute().text.trim() + + //for jar building + rootRelease = "${project.rootDir}/release/" + versionDir = "${rootRelease}/${project.description}-${project.version}" + confDir = "${versionDir}/conf" + libsDir = "${versionDir}/libs" + } + + repositories { + mavenCentral() + maven { + url "https://packages.confluent.io/maven/" + } + } + + dependencies { + //logback + implementation group: 'ch.qos.logback', name: 'logback-classic', version: logbackVersion + + //lombok + compileOnly group: 'org.projectlombok', name: 'lombok', version: '1.18.30' + annotationProcessor group: 'org.projectlombok', name: 'lombok', version: lombokVersion + + //tests + testImplementation group: 'org.mockito', name: 'mockito-core', version: mockitoJupiterVersion + testImplementation group: 'org.junit.jupiter', name: 'junit-jupiter', version: mockitoJupiterVersion + testImplementation group: 'org.assertj', name: 'assertj-core', version: '3.25.3' + + } + + test { + useJUnitPlatform() + + maxHeapSize = '1G' + + testLogging { + events "passed" + } + } + + license { + include("**/**.java", "**/**Test.java") + exclude("**/kcql/antlr4/**.java") //antlr generated files + header = project.file("${project.rootDir}/HEADER.txt") + newLine = false + + style { + java = 'BLOCK_COMMENT' + } + + properties { + name = 'Lenses.io' + year = LocalDate.now().year + } + } + + jar { + manifest { + attributes("StreamReactor-Version": project.version, + "Kafka-Version": kafkaVersion, + "Created-By": "Lenses", + "Created-At": new Date().format("YYYYMMDDHHmm"), + "Git-Repo": gitRepo, + "Git-Commit-Hash": gitCommitHash, + "Git-Tag": gitTag, + "StreamReactor-Docs": "https://docs.lenses.io/connectors/" + ) + } + } + + shadowJar { + + manifest { + attributes("StreamReactor-Version": project.version, + "Kafka-Version": kafkaVersion, + "Created-By": "Lenses", + "Created-At": new Date().format("YYYYMMDDHHmm"), + "Git-Repo": gitRepo, + "Git-Commit-Hash": gitCommitHash, + "Git-Tag": gitTag, + "StreamReactor-Docs": "https://docs.lenses.io/connectors/" + ) + } + configurations = [project.configurations.compileClasspath] + //archiveBaseName = "${project.name}-${project.version}-${kafkaVersion}-all" + zip64 true + + mergeServiceFiles { + exclude "META-INF/*.SF" + exclude "META-INF/*.DSA" + exclude "META-INF/*.RSA" + } + + //shadowing antlr packages in order to avoid conflict when using kafka connect + relocate('org.antlr', 'lshaded.antlr') + + dependencies { +// // UNCOMMENT BELOW IF NEED CLEAN JAR +// exclude(dependency { +// it.moduleGroup != 'io.lenses.streamreactor' +// }) +// exclude(dependency('org.apache.logging.log4j:log4j-core:2.11.1')) +// exclude(dependency("org.apache.avro:.*")) +// exclude(dependency("org.apache.kafka:.*")) +// exclude(dependency("io.confluent:.*")) +// exclude(dependency("org.apache.kafka:.*")) +// exclude(dependency("org.apache.zookeeper:.*")) +// exclude(dependency("com.google.guava:guava:28.1-android")) + } + + } + compileJava.dependsOn("checkLicenses") + + task fatJar(dependsOn: [test, jar, shadowJar]) + + task collectFatJar(type: Copy, dependsOn: [fatJar]) { + from("${buildDir}/libs").include("kafka-connect-*-all.jar") + .exclude("*-common-*").into(libsDir) + } +} + +task prepareRelease(dependsOn: [collectFatJar]) { + dependsOn subprojects.collectFatJar +} + +task releaseModuleList() { + def nonReleaseModules = ["java-reactor", "kafka-connect-cloud-common", + "kafka-connect-common", "kafka-connect-query-language"] + + def modulesFile = new File("gradle-modules.txt") + modulesFile.delete() + modulesFile.createNewFile() + + def modulesBuilder = new StringBuilder("[") + allprojects.name.stream() + .filter {moduleName -> !nonReleaseModules.contains(moduleName)} + .forEach {moduleName -> modulesBuilder.append("\"" + moduleName + "\",") } + modulesBuilder.deleteCharAt(modulesBuilder.lastIndexOf(",")).append("]") + modulesFile.append(modulesBuilder) +} diff --git a/java-connectors/config/checkstyle/checkstyle.xml b/java-connectors/config/checkstyle/checkstyle.xml new file mode 100644 index 000000000..c731fe024 --- /dev/null +++ b/java-connectors/config/checkstyle/checkstyle.xml @@ -0,0 +1,379 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/java-connectors/config/checkstyle/suppressions.xml b/java-connectors/config/checkstyle/suppressions.xml new file mode 100644 index 000000000..3d20b94b9 --- /dev/null +++ b/java-connectors/config/checkstyle/suppressions.xml @@ -0,0 +1,11 @@ + + + + + + + + + diff --git a/java-connectors/gradle/wrapper/gradle-wrapper.jar b/java-connectors/gradle/wrapper/gradle-wrapper.jar new file mode 100644 index 0000000000000000000000000000000000000000..249e5832f090a2944b7473328c07c9755baa3196 GIT binary patch literal 60756 zcmb5WV{~QRw(p$^Dz@00IL3?^hro$gg*4VI_WAaTyVM5Foj~O|-84 z$;06hMwt*rV;^8iB z1~&0XWpYJmG?Ts^K9PC62H*`G}xom%S%yq|xvG~FIfP=9*f zZoDRJBm*Y0aId=qJ?7dyb)6)JGWGwe)MHeNSzhi)Ko6J<-m@v=a%NsP537lHe0R* z`If4$aaBA#S=w!2z&m>{lpTy^Lm^mg*3?M&7HFv}7K6x*cukLIGX;bQG|QWdn{%_6 zHnwBKr84#B7Z+AnBXa16a?or^R?+>$4`}{*a_>IhbjvyTtWkHw)|ay)ahWUd-qq$~ zMbh6roVsj;_qnC-R{G+Cy6bApVOinSU-;(DxUEl!i2)1EeQ9`hrfqj(nKI7?Z>Xur zoJz-a`PxkYit1HEbv|jy%~DO^13J-ut986EEG=66S}D3!L}Efp;Bez~7tNq{QsUMm zh9~(HYg1pA*=37C0}n4g&bFbQ+?-h-W}onYeE{q;cIy%eZK9wZjSwGvT+&Cgv z?~{9p(;bY_1+k|wkt_|N!@J~aoY@|U_RGoWX<;p{Nu*D*&_phw`8jYkMNpRTWx1H* z>J-Mi_!`M468#5Aix$$u1M@rJEIOc?k^QBc?T(#=n&*5eS#u*Y)?L8Ha$9wRWdH^3D4|Ps)Y?m0q~SiKiSfEkJ!=^`lJ(%W3o|CZ zSrZL-Xxc{OrmsQD&s~zPfNJOpSZUl%V8tdG%ei}lQkM+z@-4etFPR>GOH9+Y_F<3=~SXln9Kb-o~f>2a6Xz@AS3cn^;c_>lUwlK(n>z?A>NbC z`Ud8^aQy>wy=$)w;JZzA)_*Y$Z5hU=KAG&htLw1Uh00yE!|Nu{EZkch zY9O6x7Y??>!7pUNME*d!=R#s)ghr|R#41l!c?~=3CS8&zr6*aA7n9*)*PWBV2w+&I zpW1-9fr3j{VTcls1>ua}F*bbju_Xq%^v;-W~paSqlf zolj*dt`BBjHI)H9{zrkBo=B%>8}4jeBO~kWqO!~Thi!I1H(in=n^fS%nuL=X2+s!p}HfTU#NBGiwEBF^^tKU zbhhv+0dE-sbK$>J#t-J!B$TMgN@Wh5wTtK2BG}4BGfsZOoRUS#G8Cxv|6EI*n&Xxq zt{&OxCC+BNqz$9b0WM7_PyBJEVObHFh%%`~!@MNZlo*oXDCwDcFwT~Rls!aApL<)^ zbBftGKKBRhB!{?fX@l2_y~%ygNFfF(XJzHh#?`WlSL{1lKT*gJM zs>bd^H9NCxqxn(IOky5k-wALFowQr(gw%|`0991u#9jXQh?4l|l>pd6a&rx|v=fPJ z1mutj{YzpJ_gsClbWFk(G}bSlFi-6@mwoQh-XeD*j@~huW4(8ub%^I|azA)h2t#yG z7e_V_<4jlM3D(I+qX}yEtqj)cpzN*oCdYHa!nm%0t^wHm)EmFP*|FMw!tb@&`G-u~ zK)=Sf6z+BiTAI}}i{*_Ac$ffr*Wrv$F7_0gJkjx;@)XjYSh`RjAgrCck`x!zP>Ifu z&%he4P|S)H*(9oB4uvH67^0}I-_ye_!w)u3v2+EY>eD3#8QR24<;7?*hj8k~rS)~7 zSXs5ww)T(0eHSp$hEIBnW|Iun<_i`}VE0Nc$|-R}wlSIs5pV{g_Dar(Zz<4X3`W?K z6&CAIl4U(Qk-tTcK{|zYF6QG5ArrEB!;5s?tW7 zrE3hcFY&k)+)e{+YOJ0X2uDE_hd2{|m_dC}kgEKqiE9Q^A-+>2UonB+L@v3$9?AYw zVQv?X*pK;X4Ovc6Ev5Gbg{{Eu*7{N3#0@9oMI~}KnObQE#Y{&3mM4`w%wN+xrKYgD zB-ay0Q}m{QI;iY`s1Z^NqIkjrTlf`B)B#MajZ#9u41oRBC1oM1vq0i|F59> z#StM@bHt|#`2)cpl_rWB($DNJ3Lap}QM-+A$3pe}NyP(@+i1>o^fe-oxX#Bt`mcQc zb?pD4W%#ep|3%CHAYnr*^M6Czg>~L4?l16H1OozM{P*en298b+`i4$|w$|4AHbzqB zHpYUsHZET$Z0ztC;U+0*+amF!@PI%^oUIZy{`L{%O^i{Xk}X0&nl)n~tVEpcAJSJ} zverw15zP1P-O8h9nd!&hj$zuwjg?DoxYIw{jWM zW5_pj+wFy8Tsa9g<7Qa21WaV&;ejoYflRKcz?#fSH_)@*QVlN2l4(QNk| z4aPnv&mrS&0|6NHq05XQw$J^RR9T{3SOcMKCXIR1iSf+xJ0E_Wv?jEc*I#ZPzyJN2 zUG0UOXHl+PikM*&g$U@g+KbG-RY>uaIl&DEtw_Q=FYq?etc!;hEC_}UX{eyh%dw2V zTTSlap&5>PY{6I#(6`j-9`D&I#|YPP8a;(sOzgeKDWsLa!i-$frD>zr-oid!Hf&yS z!i^cr&7tN}OOGmX2)`8k?Tn!!4=tz~3hCTq_9CdiV!NIblUDxHh(FJ$zs)B2(t5@u z-`^RA1ShrLCkg0)OhfoM;4Z{&oZmAec$qV@ zGQ(7(!CBk<5;Ar%DLJ0p0!ResC#U<+3i<|vib1?{5gCebG7$F7URKZXuX-2WgF>YJ^i zMhHDBsh9PDU8dlZ$yJKtc6JA#y!y$57%sE>4Nt+wF1lfNIWyA`=hF=9Gj%sRwi@vd z%2eVV3y&dvAgyuJ=eNJR+*080dbO_t@BFJO<@&#yqTK&+xc|FRR;p;KVk@J3$S{p` zGaMj6isho#%m)?pOG^G0mzOAw0z?!AEMsv=0T>WWcE>??WS=fII$t$(^PDPMU(P>o z_*0s^W#|x)%tx8jIgZY~A2yG;US0m2ZOQt6yJqW@XNY_>_R7(Nxb8Ged6BdYW6{prd!|zuX$@Q2o6Ona8zzYC1u!+2!Y$Jc9a;wy+pXt}o6~Bu1oF1c zp7Y|SBTNi@=I(K%A60PMjM#sfH$y*c{xUgeSpi#HB`?|`!Tb&-qJ3;vxS!TIzuTZs-&%#bAkAyw9m4PJgvey zM5?up*b}eDEY+#@tKec)-c(#QF0P?MRlD1+7%Yk*jW;)`f;0a-ZJ6CQA?E%>i2Dt7T9?s|9ZF|KP4;CNWvaVKZ+Qeut;Jith_y{v*Ny6Co6!8MZx;Wgo z=qAi%&S;8J{iyD&>3CLCQdTX*$+Rx1AwA*D_J^0>suTgBMBb=*hefV+Ars#mmr+YsI3#!F@Xc1t4F-gB@6aoyT+5O(qMz*zG<9Qq*f0w^V!03rpr*-WLH}; zfM{xSPJeu6D(%8HU%0GEa%waFHE$G?FH^kMS-&I3)ycx|iv{T6Wx}9$$D&6{%1N_8 z_CLw)_9+O4&u94##vI9b-HHm_95m)fa??q07`DniVjAy`t7;)4NpeyAY(aAk(+T_O z1om+b5K2g_B&b2DCTK<>SE$Ode1DopAi)xaJjU>**AJK3hZrnhEQ9E`2=|HHe<^tv z63e(bn#fMWuz>4erc47}!J>U58%<&N<6AOAewyzNTqi7hJc|X{782&cM zHZYclNbBwU6673=!ClmxMfkC$(CykGR@10F!zN1Se83LR&a~$Ht&>~43OX22mt7tcZUpa;9@q}KDX3O&Ugp6< zLZLfIMO5;pTee1vNyVC$FGxzK2f>0Z-6hM82zKg44nWo|n}$Zk6&;5ry3`(JFEX$q zK&KivAe${e^5ZGc3a9hOt|!UOE&OocpVryE$Y4sPcs4rJ>>Kbi2_subQ9($2VN(3o zb~tEzMsHaBmBtaHAyES+d3A(qURgiskSSwUc9CfJ@99&MKp2sooSYZu+-0t0+L*!I zYagjOlPgx|lep9tiU%ts&McF6b0VE57%E0Ho%2oi?=Ks+5%aj#au^OBwNwhec zta6QAeQI^V!dF1C)>RHAmB`HnxyqWx?td@4sd15zPd*Fc9hpDXP23kbBenBxGeD$k z;%0VBQEJ-C)&dTAw_yW@k0u?IUk*NrkJ)(XEeI z9Y>6Vel>#s_v@=@0<{4A{pl=9cQ&Iah0iD0H`q)7NeCIRz8zx;! z^OO;1+IqoQNak&pV`qKW+K0^Hqp!~gSohcyS)?^P`JNZXw@gc6{A3OLZ?@1Uc^I2v z+X!^R*HCm3{7JPq{8*Tn>5;B|X7n4QQ0Bs79uTU%nbqOJh`nX(BVj!#f;#J+WZxx4 z_yM&1Y`2XzhfqkIMO7tB3raJKQS+H5F%o83bM+hxbQ zeeJm=Dvix$2j|b4?mDacb67v-1^lTp${z=jc1=j~QD>7c*@+1?py>%Kj%Ejp7Y-!? z8iYRUlGVrQPandAaxFfks53@2EC#0)%mrnmGRn&>=$H$S8q|kE_iWko4`^vCS2aWg z#!`RHUGyOt*k?bBYu3*j3u0gB#v(3tsije zgIuNNWNtrOkx@Pzs;A9un+2LX!zw+p3_NX^Sh09HZAf>m8l@O*rXy_82aWT$Q>iyy zqO7Of)D=wcSn!0+467&!Hl))eff=$aneB?R!YykdKW@k^_uR!+Q1tR)+IJb`-6=jj zymzA>Sv4>Z&g&WWu#|~GcP7qP&m*w-S$)7Xr;(duqCTe7p8H3k5>Y-n8438+%^9~K z3r^LIT_K{i7DgEJjIocw_6d0!<;wKT`X;&vv+&msmhAAnIe!OTdybPctzcEzBy88_ zWO{6i4YT%e4^WQZB)KHCvA(0tS zHu_Bg+6Ko%a9~$EjRB90`P(2~6uI@SFibxct{H#o&y40MdiXblu@VFXbhz>Nko;7R z70Ntmm-FePqhb%9gL+7U8@(ch|JfH5Fm)5${8|`Lef>LttM_iww6LW2X61ldBmG0z zax3y)njFe>j*T{i0s8D4=L>X^j0)({R5lMGVS#7(2C9@AxL&C-lZQx~czI7Iv+{%1 z2hEG>RzX4S8x3v#9sgGAnPzptM)g&LB}@%E>fy0vGSa(&q0ch|=ncKjNrK z`jA~jObJhrJ^ri|-)J^HUyeZXz~XkBp$VhcTEcTdc#a2EUOGVX?@mYx#Vy*!qO$Jv zQ4rgOJ~M*o-_Wptam=~krnmG*p^j!JAqoQ%+YsDFW7Cc9M%YPiBOrVcD^RY>m9Pd< zu}#9M?K{+;UIO!D9qOpq9yxUquQRmQNMo0pT`@$pVt=rMvyX)ph(-CCJLvUJy71DI zBk7oc7)-%ngdj~s@76Yse3L^gV0 z2==qfp&Q~L(+%RHP0n}+xH#k(hPRx(!AdBM$JCfJ5*C=K3ts>P?@@SZ_+{U2qFZb>4kZ{Go37{# zSQc+-dq*a-Vy4?taS&{Ht|MLRiS)Sn14JOONyXqPNnpq&2y~)6wEG0oNy>qvod$FF z`9o&?&6uZjhZ4_*5qWVrEfu(>_n2Xi2{@Gz9MZ8!YmjYvIMasE9yVQL10NBrTCczq zcTY1q^PF2l!Eraguf{+PtHV3=2A?Cu&NN&a8V(y;q(^_mFc6)%Yfn&X&~Pq zU1?qCj^LF(EQB1F`8NxNjyV%fde}dEa(Hx=r7$~ts2dzDwyi6ByBAIx$NllB4%K=O z$AHz1<2bTUb>(MCVPpK(E9wlLElo(aSd(Os)^Raum`d(g9Vd_+Bf&V;l=@mM=cC>) z)9b0enb)u_7V!!E_bl>u5nf&Rl|2r=2F3rHMdb7y9E}}F82^$Rf+P8%dKnOeKh1vs zhH^P*4Ydr^$)$h@4KVzxrHyy#cKmWEa9P5DJ|- zG;!Qi35Tp7XNj60=$!S6U#!(${6hyh7d4q=pF{`0t|N^|L^d8pD{O9@tF~W;#Je*P z&ah%W!KOIN;SyAEhAeTafJ4uEL`(RtnovM+cb(O#>xQnk?dzAjG^~4$dFn^<@-Na3 z395;wBnS{t*H;Jef2eE!2}u5Ns{AHj>WYZDgQJt8v%x?9{MXqJsGP|l%OiZqQ1aB! z%E=*Ig`(!tHh>}4_z5IMpg{49UvD*Pp9!pxt_gdAW%sIf3k6CTycOT1McPl=_#0?8 zVjz8Hj*Vy9c5-krd-{BQ{6Xy|P$6LJvMuX$* zA+@I_66_ET5l2&gk9n4$1M3LN8(yEViRx&mtd#LD}AqEs?RW=xKC(OCWH;~>(X6h!uDxXIPH06xh z*`F4cVlbDP`A)-fzf>MuScYsmq&1LUMGaQ3bRm6i7OsJ|%uhTDT zlvZA1M}nz*SalJWNT|`dBm1$xlaA>CCiQ zK`xD-RuEn>-`Z?M{1%@wewf#8?F|(@1e0+T4>nmlSRrNK5f)BJ2H*$q(H>zGD0>eL zQ!tl_Wk)k*e6v^m*{~A;@6+JGeWU-q9>?+L_#UNT%G?4&BnOgvm9@o7l?ov~XL+et zbGT)|G7)KAeqb=wHSPk+J1bdg7N3$vp(ekjI1D9V$G5Cj!=R2w=3*4!z*J-r-cyeb zd(i2KmX!|Lhey!snRw z?#$Gu%S^SQEKt&kep)up#j&9}e+3=JJBS(s>MH+|=R(`8xK{mmndWo_r`-w1#SeRD&YtAJ#GiVI*TkQZ}&aq<+bU2+coU3!jCI6E+Ad_xFW*ghnZ$q zAoF*i&3n1j#?B8x;kjSJD${1jdRB;)R*)Ao!9bd|C7{;iqDo|T&>KSh6*hCD!rwv= zyK#F@2+cv3=|S1Kef(E6Niv8kyLVLX&e=U;{0x{$tDfShqkjUME>f8d(5nzSkY6@! z^-0>DM)wa&%m#UF1F?zR`8Y3X#tA!*7Q$P3lZJ%*KNlrk_uaPkxw~ zxZ1qlE;Zo;nb@!SMazSjM>;34ROOoygo%SF);LL>rRonWwR>bmSd1XD^~sGSu$Gg# zFZ`|yKU0%!v07dz^v(tY%;So(e`o{ZYTX`hm;@b0%8|H>VW`*cr8R%3n|ehw2`(9B+V72`>SY}9^8oh$En80mZK9T4abVG*to;E z1_S6bgDOW?!Oy1LwYy=w3q~KKdbNtyH#d24PFjX)KYMY93{3-mPP-H>@M-_>N~DDu zENh~reh?JBAK=TFN-SfDfT^=+{w4ea2KNWXq2Y<;?(gf(FgVp8Zp-oEjKzB%2Iqj;48GmY3h=bcdYJ}~&4tS`Q1sb=^emaW$IC$|R+r-8V- zf0$gGE(CS_n4s>oicVk)MfvVg#I>iDvf~Ov8bk}sSxluG!6#^Z_zhB&U^`eIi1@j( z^CK$z^stBHtaDDHxn+R;3u+>Lil^}fj?7eaGB z&5nl^STqcaBxI@v>%zG|j))G(rVa4aY=B@^2{TFkW~YP!8!9TG#(-nOf^^X-%m9{Z zCC?iC`G-^RcBSCuk=Z`(FaUUe?hf3{0C>>$?Vs z`2Uud9M+T&KB6o4o9kvdi^Q=Bw!asPdxbe#W-Oaa#_NP(qpyF@bVxv5D5))srkU#m zj_KA+#7sqDn*Ipf!F5Byco4HOSd!Ui$l94|IbW%Ny(s1>f4|Mv^#NfB31N~kya9!k zWCGL-$0ZQztBate^fd>R!hXY_N9ZjYp3V~4_V z#eB)Kjr8yW=+oG)BuNdZG?jaZlw+l_ma8aET(s+-x+=F-t#Qoiuu1i`^x8Sj>b^U} zs^z<()YMFP7CmjUC@M=&lA5W7t&cxTlzJAts*%PBDAPuqcV5o7HEnqjif_7xGt)F% zGx2b4w{@!tE)$p=l3&?Bf#`+!-RLOleeRk3 z7#pF|w@6_sBmn1nECqdunmG^}pr5(ZJQVvAt$6p3H(16~;vO>?sTE`Y+mq5YP&PBo zvq!7#W$Gewy`;%6o^!Dtjz~x)T}Bdk*BS#=EY=ODD&B=V6TD2z^hj1m5^d6s)D*wk zu$z~D7QuZ2b?5`p)E8e2_L38v3WE{V`bVk;6fl#o2`) z99JsWhh?$oVRn@$S#)uK&8DL8>An0&S<%V8hnGD7Z^;Y(%6;^9!7kDQ5bjR_V+~wp zfx4m3z6CWmmZ<8gDGUyg3>t8wgJ5NkkiEm^(sedCicP^&3D%}6LtIUq>mXCAt{9eF zNXL$kGcoUTf_Lhm`t;hD-SE)m=iBnxRU(NyL}f6~1uH)`K!hmYZjLI%H}AmEF5RZt z06$wn63GHnApHXZZJ}s^s)j9(BM6e*7IBK6Bq(!)d~zR#rbxK9NVIlgquoMq z=eGZ9NR!SEqP6=9UQg#@!rtbbSBUM#ynF);zKX+|!Zm}*{H z+j=d?aZ2!?@EL7C~%B?6ouCKLnO$uWn;Y6Xz zX8dSwj732u(o*U3F$F=7xwxm>E-B+SVZH;O-4XPuPkLSt_?S0)lb7EEg)Mglk0#eS z9@jl(OnH4juMxY+*r03VDfPx_IM!Lmc(5hOI;`?d37f>jPP$?9jQQIQU@i4vuG6MagEoJrQ=RD7xt@8E;c zeGV*+Pt+t$@pt!|McETOE$9k=_C!70uhwRS9X#b%ZK z%q(TIUXSS^F0`4Cx?Rk07C6wI4!UVPeI~-fxY6`YH$kABdOuiRtl73MqG|~AzZ@iL&^s?24iS;RK_pdlWkhcF z@Wv-Om(Aealfg)D^adlXh9Nvf~Uf@y;g3Y)i(YP zEXDnb1V}1pJT5ZWyw=1i+0fni9yINurD=EqH^ciOwLUGi)C%Da)tyt=zq2P7pV5-G zR7!oq28-Fgn5pW|nlu^b!S1Z#r7!Wtr{5J5PQ>pd+2P7RSD?>(U7-|Y z7ZQ5lhYIl_IF<9?T9^IPK<(Hp;l5bl5tF9>X-zG14_7PfsA>6<$~A338iYRT{a@r_ zuXBaT=`T5x3=s&3=RYx6NgG>No4?5KFBVjE(swfcivcIpPQFx5l+O;fiGsOrl5teR z_Cm+;PW}O0Dwe_(4Z@XZ)O0W-v2X><&L*<~*q3dg;bQW3g7)a#3KiQP>+qj|qo*Hk z?57>f2?f@`=Fj^nkDKeRkN2d$Z@2eNKpHo}ksj-$`QKb6n?*$^*%Fb3_Kbf1(*W9K>{L$mud2WHJ=j0^=g30Xhg8$#g^?36`p1fm;;1@0Lrx+8t`?vN0ZorM zSW?rhjCE8$C|@p^sXdx z|NOHHg+fL;HIlqyLp~SSdIF`TnSHehNCU9t89yr@)FY<~hu+X`tjg(aSVae$wDG*C zq$nY(Y494R)hD!i1|IIyP*&PD_c2FPgeY)&mX1qujB1VHPG9`yFQpLFVQ0>EKS@Bp zAfP5`C(sWGLI?AC{XEjLKR4FVNw(4+9b?kba95ukgR1H?w<8F7)G+6&(zUhIE5Ef% z=fFkL3QKA~M@h{nzjRq!Y_t!%U66#L8!(2-GgFxkD1=JRRqk=n%G(yHKn%^&$dW>; zSjAcjETMz1%205se$iH_)ZCpfg_LwvnsZQAUCS#^FExp8O4CrJb6>JquNV@qPq~3A zZ<6dOU#6|8+fcgiA#~MDmcpIEaUO02L5#T$HV0$EMD94HT_eXLZ2Zi&(! z&5E>%&|FZ`)CN10tM%tLSPD*~r#--K(H-CZqIOb99_;m|D5wdgJ<1iOJz@h2Zkq?} z%8_KXb&hf=2Wza(Wgc;3v3TN*;HTU*q2?#z&tLn_U0Nt!y>Oo>+2T)He6%XuP;fgn z-G!#h$Y2`9>Jtf}hbVrm6D70|ERzLAU>3zoWhJmjWfgM^))T+2u$~5>HF9jQDkrXR z=IzX36)V75PrFjkQ%TO+iqKGCQ-DDXbaE;C#}!-CoWQx&v*vHfyI>$HNRbpvm<`O( zlx9NBWD6_e&J%Ous4yp~s6)Ghni!I6)0W;9(9$y1wWu`$gs<$9Mcf$L*piP zPR0Av*2%ul`W;?-1_-5Zy0~}?`e@Y5A&0H!^ApyVTT}BiOm4GeFo$_oPlDEyeGBbh z1h3q&Dx~GmUS|3@4V36&$2uO8!Yp&^pD7J5&TN{?xphf*-js1fP?B|`>p_K>lh{ij zP(?H%e}AIP?_i^f&Li=FDSQ`2_NWxL+BB=nQr=$ zHojMlXNGauvvwPU>ZLq!`bX-5F4jBJ&So{kE5+ms9UEYD{66!|k~3vsP+mE}x!>%P za98bAU0!h0&ka4EoiDvBM#CP#dRNdXJcb*(%=<(g+M@<)DZ!@v1V>;54En?igcHR2 zhubQMq}VSOK)onqHfczM7YA@s=9*ow;k;8)&?J3@0JiGcP! zP#00KZ1t)GyZeRJ=f0^gc+58lc4Qh*S7RqPIC6GugG1gXe$LIQMRCo8cHf^qXgAa2 z`}t>u2Cq1CbSEpLr~E=c7~=Qkc9-vLE%(v9N*&HF`(d~(0`iukl5aQ9u4rUvc8%m) zr2GwZN4!s;{SB87lJB;veebPmqE}tSpT>+`t?<457Q9iV$th%i__Z1kOMAswFldD6 ztbOvO337S5o#ZZgN2G99_AVqPv!?Gmt3pzgD+Hp3QPQ`9qJ(g=kjvD+fUSS3upJn! zqoG7acIKEFRX~S}3|{EWT$kdz#zrDlJU(rPkxjws_iyLKU8+v|*oS_W*-guAb&Pj1 z35Z`3z<&Jb@2Mwz=KXucNYdY#SNO$tcVFr9KdKm|%^e-TXzs6M`PBper%ajkrIyUe zp$vVxVs9*>Vp4_1NC~Zg)WOCPmOxI1V34QlG4!aSFOH{QqSVq1^1)- z0P!Z?tT&E-ll(pwf0?=F=yOzik=@nh1Clxr9}Vij89z)ePDSCYAqw?lVI?v?+&*zH z)p$CScFI8rrwId~`}9YWPFu0cW1Sf@vRELs&cbntRU6QfPK-SO*mqu|u~}8AJ!Q$z znzu}50O=YbjwKCuSVBs6&CZR#0FTu)3{}qJJYX(>QPr4$RqWiwX3NT~;>cLn*_&1H zaKpIW)JVJ>b{uo2oq>oQt3y=zJjb%fU@wLqM{SyaC6x2snMx-}ivfU<1- znu1Lh;i$3Tf$Kh5Uk))G!D1UhE8pvx&nO~w^fG)BC&L!_hQk%^p`Kp@F{cz>80W&T ziOK=Sq3fdRu*V0=S53rcIfWFazI}Twj63CG(jOB;$*b`*#B9uEnBM`hDk*EwSRdwP8?5T?xGUKs=5N83XsR*)a4|ijz|c{4tIU+4j^A5C<#5 z*$c_d=5ml~%pGxw#?*q9N7aRwPux5EyqHVkdJO=5J>84!X6P>DS8PTTz>7C#FO?k#edkntG+fJk8ZMn?pmJSO@`x-QHq;7^h6GEXLXo1TCNhH z8ZDH{*NLAjo3WM`xeb=X{((uv3H(8&r8fJJg_uSs_%hOH%JDD?hu*2NvWGYD+j)&` zz#_1%O1wF^o5ryt?O0n;`lHbzp0wQ?rcbW(F1+h7_EZZ9{>rePvLAPVZ_R|n@;b$;UchU=0j<6k8G9QuQf@76oiE*4 zXOLQ&n3$NR#p4<5NJMVC*S);5x2)eRbaAM%VxWu9ohlT;pGEk7;002enCbQ>2r-us z3#bpXP9g|mE`65VrN`+3mC)M(eMj~~eOf)do<@l+fMiTR)XO}422*1SL{wyY(%oMpBgJagtiDf zz>O6(m;};>Hi=t8o{DVC@YigqS(Qh+ix3Rwa9aliH}a}IlOCW1@?%h_bRbq-W{KHF z%Vo?-j@{Xi@=~Lz5uZP27==UGE15|g^0gzD|3x)SCEXrx`*MP^FDLl%pOi~~Il;dc z^hrwp9sYeT7iZ)-ajKy@{a`kr0-5*_!XfBpXwEcFGJ;%kV$0Nx;apKrur zJN2J~CAv{Zjj%FolyurtW8RaFmpn&zKJWL>(0;;+q(%(Hx!GMW4AcfP0YJ*Vz!F4g z!ZhMyj$BdXL@MlF%KeInmPCt~9&A!;cRw)W!Hi@0DY(GD_f?jeV{=s=cJ6e}JktJw zQORnxxj3mBxfrH=x{`_^Z1ddDh}L#V7i}$njUFRVwOX?qOTKjfPMBO4y(WiU<)epb zvB9L=%jW#*SL|Nd_G?E*_h1^M-$PG6Pc_&QqF0O-FIOpa4)PAEPsyvB)GKasmBoEt z?_Q2~QCYGH+hW31x-B=@5_AN870vY#KB~3a*&{I=f);3Kv7q4Q7s)0)gVYx2#Iz9g(F2;=+Iy4 z6KI^8GJ6D@%tpS^8boU}zpi=+(5GfIR)35PzrbuXeL1Y1N%JK7PG|^2k3qIqHfX;G zQ}~JZ-UWx|60P5?d1e;AHx!_;#PG%d=^X(AR%i`l0jSpYOpXoKFW~7ip7|xvN;2^? zsYC9fanpO7rO=V7+KXqVc;Q5z%Bj})xHVrgoR04sA2 zl~DAwv=!(()DvH*=lyhIlU^hBkA0$e*7&fJpB0|oB7)rqGK#5##2T`@_I^|O2x4GO z;xh6ROcV<9>?e0)MI(y++$-ksV;G;Xe`lh76T#Htuia+(UrIXrf9?

L(tZ$0BqX1>24?V$S+&kLZ`AodQ4_)P#Q3*4xg8}lMV-FLwC*cN$< zt65Rf%7z41u^i=P*qO8>JqXPrinQFapR7qHAtp~&RZ85$>ob|Js;GS^y;S{XnGiBc zGa4IGvDl?x%gY`vNhv8wgZnP#UYI-w*^4YCZnxkF85@ldepk$&$#3EAhrJY0U)lR{F6sM3SONV^+$;Zx8BD&Eku3K zKNLZyBni3)pGzU0;n(X@1fX8wYGKYMpLmCu{N5-}epPDxClPFK#A@02WM3!myN%bkF z|GJ4GZ}3sL{3{qXemy+#Uk{4>Kf8v11;f8I&c76+B&AQ8udd<8gU7+BeWC`akUU~U zgXoxie>MS@rBoyY8O8Tc&8id!w+_ooxcr!1?#rc$-|SBBtH6S?)1e#P#S?jFZ8u-Bs&k`yLqW|{j+%c#A4AQ>+tj$Y z^CZajspu$F%73E68Lw5q7IVREED9r1Ijsg#@DzH>wKseye>hjsk^{n0g?3+gs@7`i zHx+-!sjLx^fS;fY!ERBU+Q zVJ!e0hJH%P)z!y%1^ZyG0>PN@5W~SV%f>}c?$H8r;Sy-ui>aruVTY=bHe}$e zi&Q4&XK!qT7-XjCrDaufT@>ieQ&4G(SShUob0Q>Gznep9fR783jGuUynAqc6$pYX; z7*O@@JW>O6lKIk0G00xsm|=*UVTQBB`u1f=6wGAj%nHK_;Aqmfa!eAykDmi-@u%6~ z;*c!pS1@V8r@IX9j&rW&d*}wpNs96O2Ute>%yt{yv>k!6zfT6pru{F1M3P z2WN1JDYqoTB#(`kE{H676QOoX`cnqHl1Yaru)>8Ky~VU{)r#{&s86Vz5X)v15ULHA zAZDb{99+s~qI6;-dQ5DBjHJP@GYTwn;Dv&9kE<0R!d z8tf1oq$kO`_sV(NHOSbMwr=To4r^X$`sBW4$gWUov|WY?xccQJN}1DOL|GEaD_!@& z15p?Pj+>7d`@LvNIu9*^hPN)pwcv|akvYYq)ks%`G>!+!pW{-iXPZsRp8 z35LR;DhseQKWYSD`%gO&k$Dj6_6q#vjWA}rZcWtQr=Xn*)kJ9kacA=esi*I<)1>w^ zO_+E>QvjP)qiSZg9M|GNeLtO2D7xT6vsj`88sd!94j^AqxFLi}@w9!Y*?nwWARE0P znuI_7A-saQ+%?MFA$gttMV-NAR^#tjl_e{R$N8t2NbOlX373>e7Ox=l=;y#;M7asp zRCz*CLnrm$esvSb5{T<$6CjY zmZ(i{Rs_<#pWW>(HPaaYj`%YqBra=Ey3R21O7vUbzOkJJO?V`4-D*u4$Me0Bx$K(lYo`JO}gnC zx`V}a7m-hLU9Xvb@K2ymioF)vj12<*^oAqRuG_4u%(ah?+go%$kOpfb`T96P+L$4> zQ#S+sA%VbH&mD1k5Ak7^^dZoC>`1L%i>ZXmooA!%GI)b+$D&ziKrb)a=-ds9xk#~& z7)3iem6I|r5+ZrTRe_W861x8JpD`DDIYZNm{$baw+$)X^Jtjnl0xlBgdnNY}x%5za zkQ8E6T<^$sKBPtL4(1zi_Rd(tVth*3Xs!ulflX+70?gb&jRTnI8l+*Aj9{|d%qLZ+ z>~V9Z;)`8-lds*Zgs~z1?Fg?Po7|FDl(Ce<*c^2=lFQ~ahwh6rqSjtM5+$GT>3WZW zj;u~w9xwAhOc<kF}~`CJ68 z?(S5vNJa;kriPlim33{N5`C{9?NWhzsna_~^|K2k4xz1`xcui*LXL-1#Y}Hi9`Oo!zQ>x-kgAX4LrPz63uZ+?uG*84@PKq-KgQlMNRwz=6Yes) zY}>YN+qP}nwr$(CZQFjUOI=-6J$2^XGvC~EZ+vrqWaOXB$k?%Suf5k=4>AveC1aJ! ziaW4IS%F$_Babi)kA8Y&u4F7E%99OPtm=vzw$$ zEz#9rvn`Iot_z-r3MtV>k)YvErZ<^Oa${`2>MYYODSr6?QZu+be-~MBjwPGdMvGd!b!elsdi4% z`37W*8+OGulab8YM?`KjJ8e+jM(tqLKSS@=jimq3)Ea2EB%88L8CaM+aG7;27b?5` z4zuUWBr)f)k2o&xg{iZ$IQkJ+SK>lpq4GEacu~eOW4yNFLU!Kgc{w4&D$4ecm0f}~ zTTzquRW@`f0}|IILl`!1P+;69g^upiPA6F{)U8)muWHzexRenBU$E^9X-uIY2%&1w z_=#5*(nmxJ9zF%styBwivi)?#KMG96-H@hD-H_&EZiRNsfk7mjBq{L%!E;Sqn!mVX*}kXhwH6eh;b42eD!*~upVG@ z#smUqz$ICm!Y8wY53gJeS|Iuard0=;k5i5Z_hSIs6tr)R4n*r*rE`>38Pw&lkv{_r!jNN=;#?WbMj|l>cU(9trCq; z%nN~r^y7!kH^GPOf3R}?dDhO=v^3BeP5hF|%4GNQYBSwz;x({21i4OQY->1G=KFyu z&6d`f2tT9Yl_Z8YACZaJ#v#-(gcyeqXMhYGXb=t>)M@fFa8tHp2x;ODX=Ap@a5I=U z0G80^$N0G4=U(>W%mrrThl0DjyQ-_I>+1Tdd_AuB3qpYAqY54upwa3}owa|x5iQ^1 zEf|iTZxKNGRpI>34EwkIQ2zHDEZ=(J@lRaOH>F|2Z%V_t56Km$PUYu^xA5#5Uj4I4RGqHD56xT%H{+P8Ag>e_3pN$4m8n>i%OyJFPNWaEnJ4McUZPa1QmOh?t8~n& z&RulPCors8wUaqMHECG=IhB(-tU2XvHP6#NrLVyKG%Ee*mQ5Ps%wW?mcnriTVRc4J`2YVM>$ixSF2Xi+Wn(RUZnV?mJ?GRdw%lhZ+t&3s7g!~g{%m&i<6 z5{ib-<==DYG93I(yhyv4jp*y3#*WNuDUf6`vTM%c&hiayf(%=x@4$kJ!W4MtYcE#1 zHM?3xw63;L%x3drtd?jot!8u3qeqctceX3m;tWetK+>~q7Be$h>n6riK(5@ujLgRS zvOym)k+VAtyV^mF)$29Y`nw&ijdg~jYpkx%*^ z8dz`C*g=I?;clyi5|!27e2AuSa$&%UyR(J3W!A=ZgHF9OuKA34I-1U~pyD!KuRkjA zbkN!?MfQOeN>DUPBxoy5IX}@vw`EEB->q!)8fRl_mqUVuRu|C@KD-;yl=yKc=ZT0% zB$fMwcC|HE*0f8+PVlWHi>M`zfsA(NQFET?LrM^pPcw`cK+Mo0%8*x8@65=CS_^$cG{GZQ#xv($7J z??R$P)nPLodI;P!IC3eEYEHh7TV@opr#*)6A-;EU2XuogHvC;;k1aI8asq7ovoP!* z?x%UoPrZjj<&&aWpsbr>J$Er-7!E(BmOyEv!-mbGQGeJm-U2J>74>o5x`1l;)+P&~ z>}f^=Rx(ZQ2bm+YE0u=ZYrAV@apyt=v1wb?R@`i_g64YyAwcOUl=C!i>=Lzb$`tjv zOO-P#A+)t-JbbotGMT}arNhJmmGl-lyUpMn=2UacVZxmiG!s!6H39@~&uVokS zG=5qWhfW-WOI9g4!R$n7!|ViL!|v3G?GN6HR0Pt_L5*>D#FEj5wM1DScz4Jv@Sxnl zB@MPPmdI{(2D?;*wd>3#tjAirmUnQoZrVv`xM3hARuJksF(Q)wd4P$88fGYOT1p6U z`AHSN!`St}}UMBT9o7i|G`r$ zrB=s$qV3d6$W9@?L!pl0lf%)xs%1ko^=QY$ty-57=55PvP(^6E7cc zGJ*>m2=;fOj?F~yBf@K@9qwX0hA803Xw+b0m}+#a(>RyR8}*Y<4b+kpp|OS+!whP( zH`v{%s>jsQI9rd$*vm)EkwOm#W_-rLTHcZRek)>AtF+~<(did)*oR1|&~1|e36d-d zgtm5cv1O0oqgWC%Et@P4Vhm}Ndl(Y#C^MD03g#PH-TFy+7!Osv1z^UWS9@%JhswEq~6kSr2DITo59+; ze=ZC}i2Q?CJ~Iyu?vn|=9iKV>4j8KbxhE4&!@SQ^dVa-gK@YfS9xT(0kpW*EDjYUkoj! zE49{7H&E}k%5(>sM4uGY)Q*&3>{aitqdNnRJkbOmD5Mp5rv-hxzOn80QsG=HJ_atI-EaP69cacR)Uvh{G5dTpYG7d zbtmRMq@Sexey)||UpnZ?;g_KMZq4IDCy5}@u!5&B^-=6yyY{}e4Hh3ee!ZWtL*s?G zxG(A!<9o!CL+q?u_utltPMk+hn?N2@?}xU0KlYg?Jco{Yf@|mSGC<(Zj^yHCvhmyx z?OxOYoxbptDK()tsJ42VzXdINAMWL$0Gcw?G(g8TMB)Khw_|v9`_ql#pRd2i*?CZl z7k1b!jQB=9-V@h%;Cnl7EKi;Y^&NhU0mWEcj8B|3L30Ku#-9389Q+(Yet0r$F=+3p z6AKOMAIi|OHyzlHZtOm73}|ntKtFaXF2Fy|M!gOh^L4^62kGUoWS1i{9gsds_GWBc zLw|TaLP64z3z9?=R2|T6Xh2W4_F*$cq>MtXMOy&=IPIJ`;!Tw?PqvI2b*U1)25^<2 zU_ZPoxg_V0tngA0J+mm?3;OYw{i2Zb4x}NedZug!>EoN3DC{1i)Z{Z4m*(y{ov2%- zk(w>+scOO}MN!exSc`TN)!B=NUX`zThWO~M*ohqq;J2hx9h9}|s#?@eR!=F{QTrq~ zTcY|>azkCe$|Q0XFUdpFT=lTcyW##i;-e{}ORB4D?t@SfqGo_cS z->?^rh$<&n9DL!CF+h?LMZRi)qju!meugvxX*&jfD!^1XB3?E?HnwHP8$;uX{Rvp# zh|)hM>XDv$ZGg=$1{+_bA~u-vXqlw6NH=nkpyWE0u}LQjF-3NhATL@9rRxMnpO%f7 z)EhZf{PF|mKIMFxnC?*78(}{Y)}iztV12}_OXffJ;ta!fcFIVjdchyHxH=t%ci`Xd zX2AUB?%?poD6Zv*&BA!6c5S#|xn~DK01#XvjT!w!;&`lDXSJT4_j$}!qSPrb37vc{ z9^NfC%QvPu@vlxaZ;mIbn-VHA6miwi8qJ~V;pTZkKqqOii<1Cs}0i?uUIss;hM4dKq^1O35y?Yp=l4i zf{M!@QHH~rJ&X~8uATV><23zZUbs-J^3}$IvV_ANLS08>k`Td7aU_S1sLsfi*C-m1 z-e#S%UGs4E!;CeBT@9}aaI)qR-6NU@kvS#0r`g&UWg?fC7|b^_HyCE!8}nyh^~o@< zpm7PDFs9yxp+byMS(JWm$NeL?DNrMCNE!I^ko-*csB+dsf4GAq{=6sfyf4wb>?v1v zmb`F*bN1KUx-`ra1+TJ37bXNP%`-Fd`vVQFTwWpX@;s(%nDQa#oWhgk#mYlY*!d>( zE&!|ySF!mIyfING+#%RDY3IBH_fW$}6~1%!G`suHub1kP@&DoAd5~7J55;5_noPI6eLf{t;@9Kf<{aO0`1WNKd?<)C-|?C?)3s z>wEq@8=I$Wc~Mt$o;g++5qR+(6wt9GI~pyrDJ%c?gPZe)owvy^J2S=+M^ z&WhIE`g;;J^xQLVeCtf7b%Dg#Z2gq9hp_%g)-%_`y*zb; zn9`f`mUPN-Ts&fFo(aNTsXPA|J!TJ{0hZp0^;MYHLOcD=r_~~^ymS8KLCSeU3;^QzJNqS z5{5rEAv#l(X?bvwxpU;2%pQftF`YFgrD1jt2^~Mt^~G>T*}A$yZc@(k9orlCGv&|1 zWWvVgiJsCAtamuAYT~nzs?TQFt<1LSEx!@e0~@yd6$b5!Zm(FpBl;(Cn>2vF?k zOm#TTjFwd2D-CyA!mqR^?#Uwm{NBemP>(pHmM}9;;8`c&+_o3#E5m)JzfwN?(f-a4 zyd%xZc^oQx3XT?vcCqCX&Qrk~nu;fxs@JUoyVoi5fqpi&bUhQ2y!Ok2pzsFR(M(|U zw3E+kH_zmTRQ9dUMZWRE%Zakiwc+lgv7Z%|YO9YxAy`y28`Aw;WU6HXBgU7fl@dnt z-fFBV)}H-gqP!1;V@Je$WcbYre|dRdp{xt!7sL3Eoa%IA`5CAA%;Wq8PktwPdULo! z8!sB}Qt8#jH9Sh}QiUtEPZ6H0b*7qEKGJ%ITZ|vH)5Q^2m<7o3#Z>AKc%z7_u`rXA zqrCy{-{8;9>dfllLu$^M5L z-hXs))h*qz%~ActwkIA(qOVBZl2v4lwbM>9l70Y`+T*elINFqt#>OaVWoja8RMsep z6Or3f=oBnA3vDbn*+HNZP?8LsH2MY)x%c13@(XfuGR}R?Nu<|07{$+Lc3$Uv^I!MQ z>6qWgd-=aG2Y^24g4{Bw9ueOR)(9h`scImD=86dD+MnSN4$6 z^U*o_mE-6Rk~Dp!ANp#5RE9n*LG(Vg`1)g6!(XtDzsov$Dvz|Gv1WU68J$CkshQhS zCrc|cdkW~UK}5NeaWj^F4MSgFM+@fJd{|LLM)}_O<{rj z+?*Lm?owq?IzC%U%9EBga~h-cJbIu=#C}XuWN>OLrc%M@Gu~kFEYUi4EC6l#PR2JS zQUkGKrrS#6H7}2l0F@S11DP`@pih0WRkRJl#F;u{c&ZC{^$Z+_*lB)r)-bPgRFE;* zl)@hK4`tEP=P=il02x7-C7p%l=B`vkYjw?YhdJU9!P!jcmY$OtC^12w?vy3<<=tlY zUwHJ_0lgWN9vf>1%WACBD{UT)1qHQSE2%z|JHvP{#INr13jM}oYv_5#xsnv9`)UAO zuwgyV4YZ;O)eSc3(mka6=aRohi!HH@I#xq7kng?Acdg7S4vDJb6cI5fw?2z%3yR+| zU5v@Hm}vy;${cBp&@D=HQ9j7NcFaOYL zj-wV=eYF{|XTkFNM2uz&T8uH~;)^Zo!=KP)EVyH6s9l1~4m}N%XzPpduPg|h-&lL` zAXspR0YMOKd2yO)eMFFJ4?sQ&!`dF&!|niH*!^*Ml##o0M(0*uK9&yzekFi$+mP9s z>W9d%Jb)PtVi&-Ha!o~Iyh@KRuKpQ@)I~L*d`{O8!kRObjO7=n+Gp36fe!66neh+7 zW*l^0tTKjLLzr`x4`_8&on?mjW-PzheTNox8Hg7Nt@*SbE-%kP2hWYmHu#Fn@Q^J(SsPUz*|EgOoZ6byg3ew88UGdZ>9B2Tq=jF72ZaR=4u%1A6Vm{O#?@dD!(#tmR;eP(Fu z{$0O%=Vmua7=Gjr8nY%>ul?w=FJ76O2js&17W_iq2*tb!i{pt#`qZB#im9Rl>?t?0c zicIC}et_4d+CpVPx)i4~$u6N-QX3H77ez z?ZdvXifFk|*F8~L(W$OWM~r`pSk5}#F?j_5u$Obu9lDWIknO^AGu+Blk7!9Sb;NjS zncZA?qtASdNtzQ>z7N871IsPAk^CC?iIL}+{K|F@BuG2>qQ;_RUYV#>hHO(HUPpk@ z(bn~4|F_jiZi}Sad;_7`#4}EmD<1EiIxa48QjUuR?rC}^HRocq`OQPM@aHVKP9E#q zy%6bmHygCpIddPjE}q_DPC`VH_2m;Eey&ZH)E6xGeStOK7H)#+9y!%-Hm|QF6w#A( zIC0Yw%9j$s-#odxG~C*^MZ?M<+&WJ+@?B_QPUyTg9DJGtQN#NIC&-XddRsf3n^AL6 zT@P|H;PvN;ZpL0iv$bRb7|J{0o!Hq+S>_NrH4@coZtBJu#g8#CbR7|#?6uxi8d+$g z87apN>EciJZ`%Zv2**_uiET9Vk{pny&My;+WfGDw4EVL#B!Wiw&M|A8f1A@ z(yFQS6jfbH{b8Z-S7D2?Ixl`j0{+ZnpT=;KzVMLW{B$`N?Gw^Fl0H6lT61%T2AU**!sX0u?|I(yoy&Xveg7XBL&+>n6jd1##6d>TxE*Vj=8lWiG$4=u{1UbAa5QD>5_ z;Te^42v7K6Mmu4IWT6Rnm>oxrl~b<~^e3vbj-GCdHLIB_>59}Ya+~OF68NiH=?}2o zP(X7EN=quQn&)fK>M&kqF|<_*H`}c zk=+x)GU>{Af#vx&s?`UKUsz})g^Pc&?Ka@t5$n$bqf6{r1>#mWx6Ep>9|A}VmWRnowVo`OyCr^fHsf# zQjQ3Ttp7y#iQY8l`zEUW)(@gGQdt(~rkxlkefskT(t%@i8=|p1Y9Dc5bc+z#n$s13 zGJk|V0+&Ekh(F};PJzQKKo+FG@KV8a<$gmNSD;7rd_nRdc%?9)p!|B-@P~kxQG}~B zi|{0}@}zKC(rlFUYp*dO1RuvPC^DQOkX4<+EwvBAC{IZQdYxoq1Za!MW7%p7gGr=j zzWnAq%)^O2$eItftC#TTSArUyL$U54-O7e|)4_7%Q^2tZ^0-d&3J1}qCzR4dWX!)4 zzIEKjgnYgMus^>6uw4Jm8ga6>GBtMjpNRJ6CP~W=37~||gMo_p@GA@#-3)+cVYnU> zE5=Y4kzl+EbEh%dhQokB{gqNDqx%5*qBusWV%!iprn$S!;oN_6E3?0+umADVs4ako z?P+t?m?};gev9JXQ#Q&KBpzkHPde_CGu-y z<{}RRAx=xlv#mVi+Ibrgx~ujW$h{?zPfhz)Kp7kmYS&_|97b&H&1;J-mzrBWAvY} zh8-I8hl_RK2+nnf&}!W0P+>5?#?7>npshe<1~&l_xqKd0_>dl_^RMRq@-Myz&|TKZBj1=Q()) zF{dBjv5)h=&Z)Aevx}+i|7=R9rG^Di!sa)sZCl&ctX4&LScQ-kMncgO(9o6W6)yd< z@Rk!vkja*X_N3H=BavGoR0@u0<}m-7|2v!0+2h~S2Q&a=lTH91OJsvms2MT~ zY=c@LO5i`mLpBd(vh|)I&^A3TQLtr>w=zoyzTd=^f@TPu&+*2MtqE$Avf>l>}V|3-8Fp2hzo3y<)hr_|NO(&oSD z!vEjTWBxbKTiShVl-U{n*B3#)3a8$`{~Pk}J@elZ=>Pqp|MQ}jrGv7KrNcjW%TN_< zZz8kG{#}XoeWf7qY?D)L)8?Q-b@Na&>i=)(@uNo zr;cH98T3$Iau8Hn*@vXi{A@YehxDE2zX~o+RY`)6-X{8~hMpc#C`|8y> zU8Mnv5A0dNCf{Ims*|l-^ z(MRp{qoGohB34|ggDI*p!Aw|MFyJ|v+<+E3brfrI)|+l3W~CQLPbnF@G0)P~Ly!1TJLp}xh8uW`Q+RB-v`MRYZ9Gam3cM%{ zb4Cb*f)0deR~wtNb*8w-LlIF>kc7DAv>T0D(a3@l`k4TFnrO+g9XH7;nYOHxjc4lq zMmaW6qpgAgy)MckYMhl?>sq;-1E)-1llUneeA!ya9KM$)DaNGu57Z5aE>=VST$#vb zFo=uRHr$0M{-ha>h(D_boS4zId;3B|Tpqo|?B?Z@I?G(?&Iei+-{9L_A9=h=Qfn-U z1wIUnQe9!z%_j$F_{rf&`ZFSott09gY~qrf@g3O=Y>vzAnXCyL!@(BqWa)Zqt!#_k zfZHuwS52|&&)aK;CHq9V-t9qt0au{$#6c*R#e5n3rje0hic7c7m{kW$p(_`wB=Gw7 z4k`1Hi;Mc@yA7dp@r~?@rfw)TkjAW++|pkfOG}0N|2guek}j8Zen(!+@7?qt_7ndX zB=BG6WJ31#F3#Vk3=aQr8T)3`{=p9nBHlKzE0I@v`{vJ}h8pd6vby&VgFhzH|q;=aonunAXL6G2y(X^CtAhWr*jI zGjpY@raZDQkg*aMq}Ni6cRF z{oWv}5`nhSAv>usX}m^GHt`f(t8@zHc?K|y5Zi=4G*UG1Sza{$Dpj%X8 zzEXaKT5N6F5j4J|w#qlZP!zS7BT)9b+!ZSJdToqJts1c!)fwih4d31vfb{}W)EgcA zH2pZ^8_k$9+WD2n`6q5XbOy8>3pcYH9 z07eUB+p}YD@AH!}p!iKv><2QF-Y^&xx^PAc1F13A{nUeCDg&{hnix#FiO!fe(^&%Qcux!h znu*S!s$&nnkeotYsDthh1dq(iQrE|#f_=xVgfiiL&-5eAcC-> z5L0l|DVEM$#ulf{bj+Y~7iD)j<~O8CYM8GW)dQGq)!mck)FqoL^X zwNdZb3->hFrbHFm?hLvut-*uK?zXn3q1z|UX{RZ;-WiLoOjnle!xs+W0-8D)kjU#R z+S|A^HkRg$Ij%N4v~k`jyHffKaC~=wg=9)V5h=|kLQ@;^W!o2^K+xG&2n`XCd>OY5Ydi= zgHH=lgy++erK8&+YeTl7VNyVm9-GfONlSlVb3)V9NW5tT!cJ8d7X)!b-$fb!s76{t z@d=Vg-5K_sqHA@Zx-L_}wVnc@L@GL9_K~Zl(h5@AR#FAiKad8~KeWCo@mgXIQ#~u{ zgYFwNz}2b6Vu@CP0XoqJ+dm8px(5W5-Jpis97F`+KM)TuP*X8H@zwiVKDKGVp59pI zifNHZr|B+PG|7|Y<*tqap0CvG7tbR1R>jn70t1X`XJixiMVcHf%Ez*=xm1(CrTSDt z0cle!+{8*Ja&EOZ4@$qhBuKQ$U95Q%rc7tg$VRhk?3=pE&n+T3upZg^ZJc9~c2es% zh7>+|mrmA-p&v}|OtxqmHIBgUxL~^0+cpfkSK2mhh+4b=^F1Xgd2)}U*Yp+H?ls#z zrLxWg_hm}AfK2XYWr!rzW4g;+^^&bW%LmbtRai9f3PjU${r@n`JThy-cphbcwn)rq9{A$Ht`lmYKxOacy z6v2R(?gHhD5@&kB-Eg?4!hAoD7~(h>(R!s1c1Hx#s9vGPePUR|of32bS`J5U5w{F) z>0<^ktO2UHg<0{oxkdOQ;}coZDQph8p6ruj*_?uqURCMTac;>T#v+l1Tc~%^k-Vd@ zkc5y35jVNc49vZpZx;gG$h{%yslDI%Lqga1&&;mN{Ush1c7p>7e-(zp}6E7f-XmJb4nhk zb8zS+{IVbL$QVF8pf8}~kQ|dHJAEATmmnrb_wLG}-yHe>W|A&Y|;muy-d^t^<&)g5SJfaTH@P1%euONny=mxo+C z4N&w#biWY41r8k~468tvuYVh&XN&d#%QtIf9;iVXfWY)#j=l`&B~lqDT@28+Y!0E+MkfC}}H*#(WKKdJJq=O$vNYCb(ZG@p{fJgu;h z21oHQ(14?LeT>n5)s;uD@5&ohU!@wX8w*lB6i@GEH0pM>YTG+RAIWZD;4#F1&F%Jp zXZUml2sH0!lYJT?&sA!qwez6cXzJEd(1ZC~kT5kZSp7(@=H2$Azb_*W&6aA|9iwCL zdX7Q=42;@dspHDwYE?miGX#L^3xD&%BI&fN9^;`v4OjQXPBaBmOF1;#C)8XA(WFlH zycro;DS2?(G&6wkr6rqC>rqDv3nfGw3hmN_9Al>TgvmGsL8_hXx09};l9Ow@)F5@y z#VH5WigLDwZE4nh^7&@g{1FV^UZ%_LJ-s<{HN*2R$OPg@R~Z`c-ET*2}XB@9xvAjrK&hS=f|R8Gr9 zr|0TGOsI7RD+4+2{ZiwdVD@2zmg~g@^D--YL;6UYGSM8i$NbQr4!c7T9rg!8;TM0E zT#@?&S=t>GQm)*ua|?TLT2ktj#`|R<_*FAkOu2Pz$wEc%-=Y9V*$&dg+wIei3b*O8 z2|m$!jJG!J!ZGbbIa!(Af~oSyZV+~M1qGvelMzPNE_%5?c2>;MeeG2^N?JDKjFYCy z7SbPWH-$cWF9~fX%9~v99L!G(wi!PFp>rB!9xj7=Cv|F+7CsGNwY0Q_J%FID%C^CBZQfJ9K(HK%k31j~e#&?hQ zNuD6gRkVckU)v+53-fc} z7ZCzYN-5RG4H7;>>Hg?LU9&5_aua?A0)0dpew1#MMlu)LHe(M;OHjHIUl7|%%)YPo z0cBk;AOY00%Fe6heoN*$(b<)Cd#^8Iu;-2v@>cE-OB$icUF9EEoaC&q8z9}jMTT2I z8`9;jT%z0;dy4!8U;GW{i`)3!c6&oWY`J3669C!tM<5nQFFrFRglU8f)5Op$GtR-3 zn!+SPCw|04sv?%YZ(a7#L?vsdr7ss@WKAw&A*}-1S|9~cL%uA+E~>N6QklFE>8W|% zyX-qAUGTY1hQ-+um`2|&ji0cY*(qN!zp{YpDO-r>jPk*yuVSay<)cUt`t@&FPF_&$ zcHwu1(SQ`I-l8~vYyUxm@D1UEdFJ$f5Sw^HPH7b!9 zzYT3gKMF((N(v0#4f_jPfVZ=ApN^jQJe-X$`A?X+vWjLn_%31KXE*}5_}d8 zw_B1+a#6T1?>M{ronLbHIlEsMf93muJ7AH5h%;i99<~JX^;EAgEB1uHralD*!aJ@F zV2ruuFe9i2Q1C?^^kmVy921eb=tLDD43@-AgL^rQ3IO9%+vi_&R2^dpr}x{bCVPej z7G0-0o64uyWNtr*loIvslyo0%)KSDDKjfThe0hcqs)(C-MH1>bNGBDRTW~scy_{w} zp^aq8Qb!h9Lwielq%C1b8=?Z=&U)ST&PHbS)8Xzjh2DF?d{iAv)Eh)wsUnf>UtXN( zL7=$%YrZ#|^c{MYmhn!zV#t*(jdmYdCpwqpZ{v&L8KIuKn`@IIZfp!uo}c;7J57N` zAxyZ-uA4=Gzl~Ovycz%MW9ZL7N+nRo&1cfNn9(1H5eM;V_4Z_qVann7F>5f>%{rf= zPBZFaV@_Sobl?Fy&KXyzFDV*FIdhS5`Uc~S^Gjo)aiTHgn#<0C=9o-a-}@}xDor;D zZyZ|fvf;+=3MZd>SR1F^F`RJEZo+|MdyJYQAEauKu%WDol~ayrGU3zzbHKsnHKZ*z zFiwUkL@DZ>!*x05ql&EBq@_Vqv83&?@~q5?lVmffQZ+V-=qL+!u4Xs2Z2zdCQ3U7B&QR9_Iggy} z(om{Y9eU;IPe`+p1ifLx-XWh?wI)xU9ik+m#g&pGdB5Bi<`PR*?92lE0+TkRuXI)z z5LP!N2+tTc%cB6B1F-!fj#}>S!vnpgVU~3!*U1ej^)vjUH4s-bd^%B=ItQqDCGbrEzNQi(dJ`J}-U=2{7-d zK8k^Rlq2N#0G?9&1?HSle2vlkj^KWSBYTwx`2?9TU_DX#J+f+qLiZCqY1TXHFxXZqYMuD@RU$TgcnCC{_(vwZ-*uX)~go#%PK z@}2Km_5aQ~(<3cXeJN6|F8X_1@L%@xTzs}$_*E|a^_URF_qcF;Pfhoe?FTFwvjm1o z8onf@OY@jC2tVcMaZS;|T!Ks(wOgPpRzRnFS-^RZ4E!9dsnj9sFt609a|jJbb1Dt@ z<=Gal2jDEupxUSwWu6zp<<&RnAA;d&4gKVG0iu6g(DsST(4)z6R)zDpfaQ}v{5ARt zyhwvMtF%b-YazR5XLz+oh=mn;y-Mf2a8>7?2v8qX;19y?b>Z5laGHvzH;Nu9S`B8} zI)qN$GbXIQ1VL3lnof^6TS~rvPVg4V?Dl2Bb*K2z4E{5vy<(@@K_cN@U>R!>aUIRnb zL*)=787*cs#zb31zBC49x$`=fkQbMAef)L2$dR{)6BAz!t5U_B#1zZG`^neKSS22oJ#5B=gl%U=WeqL9REF2g zZnfCb0?quf?Ztj$VXvDSWoK`0L=Zxem2q}!XWLoT-kYMOx)!7fcgT35uC~0pySEme z`{wGWTkGr7>+Kb^n;W?BZH6ZP(9tQX%-7zF>vc2}LuWDI(9kh1G#7B99r4x6;_-V+k&c{nPUrR zAXJGRiMe~aup{0qzmLNjS_BC4cB#sXjckx{%_c&^xy{M61xEb>KW_AG5VFXUOjAG4 z^>Qlm9A#1N{4snY=(AmWzatb!ngqiqPbBZ7>Uhb3)dTkSGcL#&SH>iMO-IJBPua`u zo)LWZ>=NZLr758j{%(|uQuZ)pXq_4c!!>s|aDM9#`~1bzK3J1^^D#<2bNCccH7~-X}Ggi!pIIF>uFx%aPARGQsnC8ZQc8lrQ5o~smqOg>Ti^GNme94*w z)JZy{_{#$jxGQ&`M z!OMvZMHR>8*^>eS%o*6hJwn!l8VOOjZQJvh)@tnHVW&*GYPuxqXw}%M!(f-SQf`=L z5;=5w2;%82VMH6Xi&-K3W)o&K^+vJCepWZ-rW%+Dc6X3(){z$@4zjYxQ|}8UIojeC zYZpQ1dU{fy=oTr<4VX?$q)LP}IUmpiez^O&N3E_qPpchGTi5ZM6-2ScWlQq%V&R2Euz zO|Q0Hx>lY1Q1cW5xHv5!0OGU~PVEqSuy#fD72d#O`N!C;o=m+YioGu-wH2k6!t<~K zSr`E=W9)!g==~x9VV~-8{4ZN9{~-A9zJpRe%NGg$+MDuI-dH|b@BD)~>pPCGUNNzY zMDg||0@XGQgw`YCt5C&A{_+J}mvV9Wg{6V%2n#YSRN{AP#PY?1FF1#|vO_%e+#`|2*~wGAJaeRX6=IzFNeWhz6gJc8+(03Ph4y6ELAm=AkN7TOgMUEw*N{= z_)EIDQx5q22oUR+_b*tazu9+pX|n1c*IB-}{DqIj z-?E|ks{o3AGRNb;+iKcHkZvYJvFsW&83RAPs1Oh@IWy%l#5x2oUP6ZCtv+b|q>jsf zZ_9XO;V!>n`UxH1LvH8)L4?8raIvasEhkpQoJ`%!5rBs!0Tu(s_D{`4opB;57)pkX z4$A^8CsD3U5*!|bHIEqsn~{q+Ddj$ME@Gq4JXtgVz&7l{Ok!@?EA{B3P~NAqb9)4? zkQo30A^EbHfQ@87G5&EQTd`frrwL)&Yw?%-W@uy^Gn23%j?Y!Iea2xw<-f;esq zf%w5WN@E1}zyXtYv}}`U^B>W`>XPmdLj%4{P298|SisrE;7HvXX;A}Ffi8B#3Lr;1 zHt6zVb`8{#+e$*k?w8|O{Uh|&AG}|DG1PFo1i?Y*cQm$ZwtGcVgMwtBUDa{~L1KT-{jET4w60>{KZ27vXrHJ;fW{6| z=|Y4!&UX020wU1>1iRgB@Q#m~1^Z^9CG1LqDhYBrnx%IEdIty z!46iOoKlKs)c}newDG)rWUikD%j`)p z_w9Ph&e40=(2eBy;T!}*1p1f1SAUDP9iWy^u^Ubdj21Kn{46;GR+hwLO=4D11@c~V zI8x&(D({K~Df2E)Nx_yQvYfh4;MbMJ@Z}=Dt3_>iim~QZ*hZIlEs0mEb z_54+&*?wMD`2#vsQRN3KvoT>hWofI_Vf(^C1ff-Ike@h@saEf7g}<9T`W;HAne-Nd z>RR+&SP35w)xKn8^U$7))PsM!jKwYZ*RzEcG-OlTrX3}9a{q%#Un5E5W{{hp>w~;` zGky+3(vJvQyGwBo`tCpmo0mo((?nM8vf9aXrrY1Ve}~TuVkB(zeds^jEfI}xGBCM2 zL1|#tycSaWCurP+0MiActG3LCas@_@tao@(R1ANlwB$4K53egNE_;!&(%@Qo$>h`^1S_!hN6 z)vZtG$8fN!|BXBJ=SI>e(LAU(y(i*PHvgQ2llulxS8>qsimv7yL}0q_E5WiAz7)(f zC(ahFvG8&HN9+6^jGyLHM~$)7auppeWh_^zKk&C_MQ~8;N??OlyH~azgz5fe^>~7F zl3HnPN3z-kN)I$4@`CLCMQx3sG~V8hPS^}XDXZrQA>}mQPw%7&!sd(Pp^P=tgp-s^ zjl}1-KRPNWXgV_K^HkP__SR`S-|OF0bR-N5>I%ODj&1JUeAQ3$9i;B~$S6}*^tK?= z**%aCiH7y?xdY?{LgVP}S0HOh%0%LI$wRx;$T|~Y8R)Vdwa}kGWv8?SJVm^>r6+%I z#lj1aR94{@MP;t-scEYQWc#xFA30^}?|BeX*W#9OL;Q9#WqaaM546j5j29((^_8Nu z4uq}ESLr~r*O7E7$D{!k9W>`!SLoyA53i9QwRB{!pHe8um|aDE`Cg0O*{jmor)^t)3`>V>SWN-2VJcFmj^1?~tT=JrP`fVh*t zXHarp=8HEcR#vFe+1a%XXuK+)oFs`GDD}#Z+TJ}Ri`FvKO@ek2ayn}yaOi%(8p%2$ zpEu)v0Jym@f}U|-;}CbR=9{#<^z28PzkkTNvyKvJDZe+^VS2bES3N@Jq!-*}{oQlz z@8bgC_KnDnT4}d#&Cpr!%Yb?E!brx0!eVOw~;lLwUoz#Np%d$o%9scc3&zPm`%G((Le|6o1 zM(VhOw)!f84zG^)tZ1?Egv)d8cdNi+T${=5kV+j;Wf%2{3g@FHp^Gf*qO0q!u$=m9 zCaY`4mRqJ;FTH5`a$affE5dJrk~k`HTP_7nGTY@B9o9vvnbytaID;^b=Tzp7Q#DmD zC(XEN)Ktn39z5|G!wsVNnHi) z%^q94!lL|hF`IijA^9NR0F$@h7k5R^ljOW(;Td9grRN0Mb)l_l7##{2nPQ@?;VjXv zaLZG}yuf$r$<79rVPpXg?6iiieX|r#&`p#Con2i%S8*8F}(E) zI5E6c3tG*<;m~6>!&H!GJ6zEuhH7mkAzovdhLy;)q z{H2*8I^Pb}xC4s^6Y}6bJvMu=8>g&I)7!N!5QG$xseeU#CC?ZM-TbjsHwHgDGrsD= z{%f;@Sod+Ch66Ko2WF~;Ty)v>&x^aovCbCbD7>qF*!?BXmOV3(s|nxsb*Lx_2lpB7 zokUnzrk;P=T-&kUHO}td+Zdj!3n&NR?K~cRU zAXU!DCp?51{J4w^`cV#ye}(`SQhGQkkMu}O3M*BWt4UsC^jCFUy;wTINYmhD$AT;4 z?Xd{HaJjP`raZ39qAm;%beDbrLpbRf(mkKbANan7XsL>_pE2oo^$TgdidjRP!5-`% zv0d!|iKN$c0(T|L0C~XD0aS8t{*&#LnhE;1Kb<9&=c2B+9JeLvJr*AyyRh%@jHej=AetOMSlz^=!kxX>>B{2B1uIrQyfd8KjJ+DBy!h)~*(!|&L4^Q_07SQ~E zcemVP`{9CwFvPFu7pyVGCLhH?LhEVb2{7U+Z_>o25#+3<|8%1T^5dh}*4(kfJGry} zm%r#hU+__Z;;*4fMrX=Bkc@7|v^*B;HAl0((IBPPii%X9+u3DDF6%bI&6?Eu$8&aWVqHIM7mK6?Uvq$1|(-T|)IV<>e?!(rY zqkmO1MRaLeTR=)io(0GVtQT@s6rN%C6;nS3@eu;P#ry4q;^O@1ZKCJyp_Jo)Ty^QW z+vweTx_DLm{P-XSBj~Sl<%_b^$=}odJ!S2wAcxenmzFGX1t&Qp8Vxz2VT`uQsQYtdn&_0xVivIcxZ_hnrRtwq4cZSj1c-SG9 z7vHBCA=fd0O1<4*=lu$6pn~_pVKyL@ztw1swbZi0B?spLo56ZKu5;7ZeUml1Ws1?u zqMf1p{5myAzeX$lAi{jIUqo1g4!zWLMm9cfWcnw`k6*BR^?$2(&yW?>w;G$EmTA@a z6?y#K$C~ZT8+v{87n5Dm&H6Pb_EQ@V0IWmG9cG=O;(;5aMWWrIPzz4Q`mhK;qQp~a z+BbQrEQ+w{SeiuG-~Po5f=^EvlouB@_|4xQXH@A~KgpFHrwu%dwuCR)=B&C(y6J4J zvoGk9;lLs9%iA-IJGU#RgnZZR+@{5lYl8(e1h6&>Vc_mvg0d@);X zji4T|n#lB!>pfL|8tQYkw?U2bD`W{na&;*|znjmalA&f;*U++_aBYerq;&C8Kw7mI z7tsG*?7*5j&dU)Lje;^{D_h`%(dK|pB*A*1(Jj)w^mZ9HB|vGLkF1GEFhu&rH=r=8 zMxO42e{Si6$m+Zj`_mXb&w5Q(i|Yxyg?juUrY}78uo@~3v84|8dfgbPd0iQJRdMj< zncCNGdMEcsxu#o#B5+XD{tsg*;j-eF8`mp~K8O1J!Z0+>0=7O=4M}E?)H)ENE;P*F z$Ox?ril_^p0g7xhDUf(q652l|562VFlC8^r8?lQv;TMvn+*8I}&+hIQYh2 z1}uQQaag&!-+DZ@|C+C$bN6W;S-Z@)d1|en+XGvjbOxCa-qAF*LA=6s(Jg+g;82f$ z(Vb)8I)AH@cdjGFAR5Rqd0wiNCu!xtqWbcTx&5kslzTb^7A78~Xzw1($UV6S^VWiP zFd{Rimd-0CZC_Bu(WxBFW7+k{cOW7DxBBkJdJ;VsJ4Z@lERQr%3eVv&$%)b%<~ zCl^Y4NgO}js@u{|o~KTgH}>!* z_iDNqX2(As7T0xivMH|3SC1ivm8Q}6Ffcd7owUKN5lHAtzMM4<0v+ykUT!QiowO;`@%JGv+K$bBx@*S7C8GJVqQ_K>12}M`f_Ys=S zKFh}HM9#6Izb$Y{wYzItTy+l5U2oL%boCJn?R3?jP@n$zSIwlmyGq30Cw4QBO|14` zW5c);AN*J3&eMFAk$SR~2k|&+&Bc$e>s%c{`?d~85S-UWjA>DS5+;UKZ}5oVa5O(N zqqc@>)nee)+4MUjH?FGv%hm2{IlIF-QX}ym-7ok4Z9{V+ZHVZQl$A*x!(q%<2~iVv znUa+BX35&lCb#9VE-~Y^W_f;Xhl%vgjwdjzMy$FsSIj&ok}L+X`4>J=9BkN&nu^E*gbhj3(+D>C4E z@Fwq_=N)^bKFSHTzZk?-gNU$@l}r}dwGyh_fNi=9b|n}J>&;G!lzilbWF4B}BBq4f zYIOl?b)PSh#XTPp4IS5ZR_2C!E)Z`zH0OW%4;&~z7UAyA-X|sh9@~>cQW^COA9hV4 zXcA6qUo9P{bW1_2`eo6%hgbN%(G-F1xTvq!sc?4wN6Q4`e9Hku zFwvlAcRY?6h^Fj$R8zCNEDq8`=uZB8D-xn)tA<^bFFy}4$vA}Xq0jAsv1&5!h!yRA zU()KLJya5MQ`q&LKdH#fwq&(bNFS{sKlEh_{N%{XCGO+po#(+WCLmKW6&5iOHny>g z3*VFN?mx!16V5{zyuMWDVP8U*|BGT$(%IO|)?EF|OI*sq&RovH!N%=>i_c?K*A>>k zyg1+~++zY4Q)J;VWN0axhoIKx;l&G$gvj(#go^pZskEVj8^}is3Jw26LzYYVos0HX zRPvmK$dVxM8(Tc?pHFe0Z3uq){{#OK3i-ra#@+;*=ui8)y6hsRv z4Fxx1c1+fr!VI{L3DFMwXKrfl#Q8hfP@ajgEau&QMCxd{g#!T^;ATXW)nUg&$-n25 zruy3V!!;{?OTobo|0GAxe`Acn3GV@W=&n;~&9 zQM>NWW~R@OYORkJAo+eq1!4vzmf9K%plR4(tB@TR&FSbDoRgJ8qVcH#;7lQub*nq&?Z>7WM=oeEVjkaG zT#f)=o!M2DO5hLR+op>t0CixJCIeXH*+z{-XS|%jx)y(j&}Wo|3!l7{o)HU3m7LYyhv*xF&tq z%IN7N;D4raue&&hm0xM=`qv`+TK@;_xAcGKuK(2|75~ar2Yw)geNLSmVxV@x89bQu zpViVKKnlkwjS&&c|-X6`~xdnh}Ps)Hs z4VbUL^{XNLf7_|Oi>tA%?SG5zax}esF*FH3d(JH^Gvr7Rp*n=t7frH!U;!y1gJB^i zY_M$KL_}mW&XKaDEi9K-wZR|q*L32&m+2n_8lq$xRznJ7p8}V>w+d@?uB!eS3#u<} zIaqi!b!w}a2;_BfUUhGMy#4dPx>)_>yZ`ai?Rk`}d0>~ce-PfY-b?Csd(28yX22L% zI7XI>OjIHYTk_@Xk;Gu^F52^Gn6E1&+?4MxDS2G_#PQ&yXPXP^<-p|2nLTb@AAQEY zI*UQ9Pmm{Kat}wuazpjSyXCdnrD&|C1c5DIb1TnzF}f4KIV6D)CJ!?&l&{T)e4U%3HTSYqsQ zo@zWB1o}ceQSV)<4G<)jM|@@YpL+XHuWsr5AYh^Q{K=wSV99D~4RRU52FufmMBMmd z_H}L#qe(}|I9ZyPRD6kT>Ivj&2Y?qVZq<4bG_co_DP`sE*_Xw8D;+7QR$Uq(rr+u> z8bHUWbV19i#)@@G4bCco@Xb<8u~wVDz9S`#k@ciJtlu@uP1U0X?yov8v9U3VOig2t zL9?n$P3=1U_Emi$#slR>N5wH-=J&T=EdUHA}_Z zZIl3nvMP*AZS9{cDqFanrA~S5BqxtNm9tlu;^`)3X&V4tMAkJ4gEIPl= zoV!Gyx0N{3DpD@)pv^iS*dl2FwANu;1;%EDl}JQ7MbxLMAp>)UwNwe{=V}O-5C*>F zu?Ny+F64jZn<+fKjF01}8h5H_3pey|;%bI;SFg$w8;IC<8l|3#Lz2;mNNik6sVTG3 z+Su^rIE#40C4a-587$U~%KedEEw1%r6wdvoMwpmlXH$xPnNQN#f%Z7|p)nC>WsuO= z4zyqapLS<8(UJ~Qi9d|dQijb_xhA2)v>la)<1md5s^R1N&PiuA$^k|A<+2C?OiHbj z>Bn$~t)>Y(Zb`8hW7q9xQ=s>Rv81V+UiuZJc<23HplI88isqRCId89fb`Kt|CxVIg znWcwprwXnotO>3s&Oypkte^9yJjlUVVxSe%_xlzmje|mYOVPH^vjA=?6xd0vaj0Oz zwJ4OJNiFdnHJX3rw&inskjryukl`*fRQ#SMod5J|KroJRsVXa5_$q7whSQ{gOi*s0 z1LeCy|JBWRsDPn7jCb4s(p|JZiZ8+*ExC@Vj)MF|*Vp{B(ziccSn`G1Br9bV(v!C2 z6#?eqpJBc9o@lJ#^p-`-=`4i&wFe>2)nlPK1p9yPFzJCzBQbpkcR>={YtamIw)3nt z(QEF;+)4`>8^_LU)_Q3 zC5_7lgi_6y>U%m)m@}Ku4C}=l^J=<<7c;99ec3p{aR+v=diuJR7uZi%aQv$oP?dn?@6Yu_+*^>T0ptf(oobdL;6)N-I!TO`zg^Xbv3#L0I~sn@WGk-^SmPh5>W+LB<+1PU}AKa?FCWF|qMNELOgdxR{ zbqE7@jVe+FklzdcD$!(A$&}}H*HQFTJ+AOrJYnhh}Yvta(B zQ_bW4Rr;R~&6PAKwgLWXS{Bnln(vUI+~g#kl{r+_zbngT`Y3`^Qf=!PxN4IYX#iW4 zucW7@LLJA9Zh3(rj~&SyN_pjO8H&)|(v%!BnMWySBJV=eSkB3YSTCyIeJ{i;(oc%_hk{$_l;v>nWSB)oVeg+blh=HB5JSlG_r7@P z3q;aFoZjD_qS@zygYqCn=;Zxjo!?NK!%J$ z52lOP`8G3feEj+HTp@Tnn9X~nG=;tS+z}u{mQX_J0kxtr)O30YD%oo)L@wy`jpQYM z@M>Me=95k1p*FW~rHiV1CIfVc{K8r|#Kt(ApkXKsDG$_>76UGNhHExFCw#Ky9*B-z zNq2ga*xax!HMf_|Vp-86r{;~YgQKqu7%szk8$hpvi_2I`OVbG1doP(`gn}=W<8%Gn z%81#&WjkH4GV;4u43EtSW>K_Ta3Zj!XF?;SO3V#q=<=>Tc^@?A`i;&`-cYj|;^ zEo#Jl5zSr~_V-4}y8pnufXLa80vZY4z2ko7fj>DR)#z=wWuS1$$W!L?(y}YC+yQ|G z@L&`2upy3f>~*IquAjkVNU>}c10(fq#HdbK$~Q3l6|=@-eBbo>B9(6xV`*)sae58*f zym~RRVx;xoCG3`JV`xo z!lFw)=t2Hy)e!IFs?0~7osWk(d%^wxq&>_XD4+U#y&-VF%4z?XH^i4w`TxpF{`XhZ z%G}iEzf!T(l>g;W9<~K+)$g!{UvhW{E0Lis(S^%I8OF&%kr!gJ&fMOpM=&=Aj@wuL zBX?*6i51Qb$uhkwkFYkaD_UDE+)rh1c;(&Y=B$3)J&iJfQSx!1NGgPtK!$c9OtJuu zX(pV$bfuJpRR|K(dp@^j}i&HeJOh@|7lWo8^$*o~Xqo z5Sb+!EtJ&e@6F+h&+_1ETbg7LfP5GZjvIUIN3ibCOldAv z)>YdO|NH$x7AC8dr=<2ekiY1%fN*r~e5h6Yaw<{XIErujKV~tiyrvV_DV0AzEknC- zR^xKM3i<1UkvqBj3C{wDvytOd+YtDSGu!gEMg+!&|8BQrT*|p)(dwQLEy+ zMtMzij3zo40)CA!BKZF~yWg?#lWhqD3@qR)gh~D{uZaJO;{OWV8XZ_)J@r3=)T|kt zUS1pXr6-`!Z}w2QR7nP%d?ecf90;K_7C3d!UZ`N(TZoWNN^Q~RjVhQG{Y<%E1PpV^4 z-m-K+$A~-+VDABs^Q@U*)YvhY4Znn2^w>732H?NRK(5QSS$V@D7yz2BVX4)f5A04~$WbxGOam22>t&uD)JB8-~yiQW6ik;FGblY_I>SvB_z2?PS z*Qm&qbKI{H1V@YGWzpx`!v)WeLT02};JJo*#f$a*FH?IIad-^(;9XC#YTWN6;Z6+S zm4O1KH=#V@FJw7Pha0!9Vb%ZIM$)a`VRMoiN&C|$YA3~ZC*8ayZRY^fyuP6$n%2IU z$#XceYZeqLTXw(m$_z|33I$B4k~NZO>pP6)H_}R{E$i%USGy{l{-jOE;%CloYPEU+ zRFxOn4;7lIOh!7abb23YKD+_-?O z0FP9otcAh+oSj;=f#$&*ExUHpd&e#bSF%#8*&ItcL2H$Sa)?pt0Xtf+t)z$_u^wZi z44oE}r4kIZGy3!Mc8q$B&6JqtnHZ>Znn!Zh@6rgIu|yU+zG8q`q9%B18|T|oN3zMq z`l&D;U!OL~%>vo&q0>Y==~zLiCZk4v%s_7!9DxQ~id1LLE93gf*gg&2$|hB#j8;?3 z5v4S;oM6rT{Y;I+#FdmNw z){d%tNM<<#GN%n9ox7B=3#;u7unZ~tLB_vRZ52a&2=IM)2VkXm=L+Iqq~uk#Dug|x z>S84e+A7EiOY5lj*!q?6HDkNh~0g;0Jy(al!ZHHDtur9T$y-~)94HelX1NHjXWIM7UAe}$?jiz z9?P4`I0JM=G5K{3_%2jPLC^_Mlw?-kYYgb7`qGa3@dn|^1fRMwiyM@Ch z;CB&o7&&?c5e>h`IM;Wnha0QKnEp=$hA8TJgR-07N~U5(>9vJzeoFsSRBkDq=x(YgEMpb=l4TDD`2 zwVJpWGTA_u7}?ecW7s6%rUs&NXD3+n;jB86`X?8(l3MBo6)PdakI6V6a}22{)8ilT zM~T*mU}__xSy|6XSrJ^%lDAR3Lft%+yxC|ZUvSO_nqMX!_ul3;R#*{~4DA=h$bP)%8Yv9X zyp><|e8=_ttI}ZAwOd#dlnSjck#6%273{E$kJuCGu=I@O)&6ID{nWF5@gLb16sj|&Sb~+du4e4O_%_o`Ix4NRrAsyr1_}MuP94s>de8cH-OUkVPk3+K z&jW)It9QiU-ti~AuJkL`XMca8Oh4$SyJ=`-5WU<{cIh+XVH#e4d&zive_UHC!pN>W z3TB;Mn5i)9Qn)#6@lo4QpI3jFYc0~+jS)4AFz8fVC;lD^+idw^S~Qhq>Tg(!3$yLD zzktzoFrU@6s4wwCMz}edpF5i5Q1IMmEJQHzp(LAt)pgN3&O!&d?3W@6U4)I^2V{;- z6A(?zd93hS*uQmnh4T)nHnE{wVhh(=MMD(h(P4+^p83Om6t<*cUW>l(qJzr%5vp@K zN27ka(L{JX=1~e2^)F^i=TYj&;<7jyUUR2Bek^A8+3Up*&Xwc{)1nRR5CT8vG>ExV zHnF3UqXJOAno_?bnhCX-&kwI~Ti8t4`n0%Up>!U`ZvK^w2+0Cs-b9%w%4`$+To|k= zKtgc&l}P`*8IS>8DOe?EB84^kx4BQp3<7P{Pq}&p%xF_81pg!l2|u=&I{AuUgmF5n zJQCTLv}%}xbFGYtKfbba{CBo)lWW%Z>i(_NvLhoQZ*5-@2l&x>e+I~0Nld3UI9tdL zRzu8}i;X!h8LHVvN?C+|M81e>Jr38%&*9LYQec9Ax>?NN+9(_>XSRv&6hlCYB`>Qm z1&ygi{Y()OU4@D_jd_-7vDILR{>o|7-k)Sjdxkjgvi{@S>6GqiF|o`*Otr;P)kLHN zZkpts;0zw_6;?f(@4S1FN=m!4^mv~W+lJA`&7RH%2$)49z0A+8@0BCHtj|yH--AEL z0tW6G%X-+J+5a{5*WKaM0QDznf;V?L5&uQw+yegDNDP`hA;0XPYc6e0;Xv6|i|^F2WB)Z$LR|HR4 zTQsRAby9(^Z@yATyOgcfQw7cKyr^3Tz7lc7+JEwwzA7)|2x+PtEb>nD(tpxJQm)Kn zW9K_*r!L%~N*vS8<5T=iv|o!zTe9k_2jC_j*7ik^M_ zaf%k{WX{-;0*`t`G!&`eW;gChVXnJ-Rn)To8vW-?>>a%QU1v`ZC=U)f8iA@%JG0mZ zDqH;~mgBnrCP~1II<=V9;EBL)J+xzCoiRBaeH&J6rL!{4zIY8tZka?_FBeQeNO3q6 zyG_alW54Ba&wQf{&F1v-r1R6ID)PTsqjIBc+5MHkcW5Fnvi~{-FjKe)t1bl}Y;z@< z=!%zvpRua>>t_x}^}z0<7MI!H2v6|XAyR9!t50q-A)xk0nflgF4*OQlCGK==4S|wc zRMsSscNhRzHMBU8TdcHN!q^I}x0iXJ%uehac|Zs_B$p@CnF)HeXPpB_Za}F{<@6-4 zl%kml@}kHQ(ypD8FsPJ2=14xXJE|b20RUIgs!2|R3>LUMGF6X*B_I|$`Qg=;zm7C z{mEDy9dTmPbued7mlO@phdmAmJ7p@GR1bjCkMw6*G7#4+`k>fk1czdJUB!e@Q(~6# zwo%@p@V5RL0ABU2LH7Asq^quDUho@H>eTZH9f*no9fY0T zD_-9px3e}A!>>kv5wk91%C9R1J_Nh!*&Kk$J3KNxC}c_@zlgpJZ+5L)Nw|^p=2ue}CJtm;uj*Iqr)K})kA$xtNUEvX;4!Px*^&9T_`IN{D z{6~QY=Nau6EzpvufB^hflc#XIsSq0Y9(nf$d~6ZwK}fal92)fr%T3=q{0mP-EyP_G z)UR5h@IX}3Qll2b0oCAcBF>b*@Etu*aTLPU<%C>KoOrk=x?pN!#f_Og-w+;xbFgjQ zXp`et%lDBBh~OcFnMKMUoox0YwBNy`N0q~bSPh@+enQ=4RUw1) zpovN`QoV>vZ#5LvC;cl|6jPr}O5tu!Ipoyib8iXqy}TeJ;4+_7r<1kV0v5?Kv>fYp zg>9L`;XwXa&W7-jf|9~uP2iyF5`5AJ`Q~p4eBU$MCC00`rcSF>`&0fbd^_eqR+}mK z4n*PMMa&FOcc)vTUR zlDUAn-mh`ahi_`f`=39JYTNVjsTa_Y3b1GOIi)6dY)D}xeshB0T8Eov5%UhWd1)u}kjEQ|LDo{tqKKrYIfVz~@dp!! zMOnah@vp)%_-jDTUG09l+;{CkDCH|Q{NqX*uHa1YxFShy*1+;J`gywKaz|2Q{lG8x zP?KBur`}r`!WLKXY_K;C8$EWG>jY3UIh{+BLv0=2)KH%P}6xE2kg)%(-uA6lC?u8}{K(#P*c zE9C8t*u%j2r_{;Rpe1A{9nNXU;b_N0vNgyK!EZVut~}+R2rcbsHilqsOviYh-pYX= zHw@53nlmwYI5W5KP>&`dBZe0Jn?nAdC^HY1wlR6$u^PbpB#AS&5L6zqrXN&7*N2Q` z+Rae1EwS)H=aVSIkr8Ek^1jy2iS2o7mqm~Mr&g5=jjt7VxwglQ^`h#Mx+x2v|9ZAwE$i_9918MjJxTMr?n!bZ6n$}y11u8I9COTU`Z$Fi z!AeAQLMw^gp_{+0QTEJrhL424pVDp%wpku~XRlD3iv{vQ!lAf!_jyqd_h}+Tr1XG| z`*FT*NbPqvHCUsYAkFnM`@l4u_QH&bszpUK#M~XLJt{%?00GXY?u_{gj3Hvs!=N(I z(=AuWPijyoU!r?aFTsa8pLB&cx}$*%;K$e*XqF{~*rA-qn)h^!(-;e}O#B$|S~c+U zN4vyOK0vmtx$5K!?g*+J@G1NmlEI=pyZXZ69tAv=@`t%ag_Hk{LP~OH9iE)I= zaJ69b4kuCkV0V zo(M0#>phpQ_)@j;h%m{-a*LGi(72TP)ws2w*@4|C-3+;=5DmC4s7Lp95%n%@Ko zfdr3-a7m*dys9iIci$A=4NPJ`HfJ;hujLgU)ZRuJI`n;Pw|yksu!#LQnJ#dJysgNb z@@qwR^wrk(jbq4H?d!lNyy72~Dnn87KxsgQ!)|*m(DRM+eC$wh7KnS-mho3|KE)7h zK3k;qZ;K1Lj6uEXLYUYi)1FN}F@-xJ z@@3Hb84sl|j{4$3J}aTY@cbX@pzB_qM~APljrjju6P0tY{C@ zpUCOz_NFmALMv1*blCcwUD3?U6tYs+N%cmJ98D%3)%)Xu^uvzF zS5O!sc#X6?EwsYkvPo6A%O8&y8sCCQH<%f2togVwW&{M;PR!a(ZT_A+jVAbf{@5kL zB@Z(hb$3U{T_}SKA_CoQVU-;j>2J=L#lZ~aQCFg-d<9rzs$_gO&d5N6eFSc z1ml8)P*FSi+k@!^M9nDWR5e@ATD8oxtDu=36Iv2!;dZzidIS(PCtEuXAtlBb1;H%Z zwnC^Ek*D)EX4#Q>R$$WA2sxC_t(!!6Tr?C#@{3}n{<^o;9id1RA&-Pig1e-2B1XpG zliNjgmd3c&%A}s>qf{_j#!Z`fu0xIwm4L0)OF=u(OEmp;bLCIaZX$&J_^Z%4Sq4GZ zPn6sV_#+6pJmDN_lx@1;Zw6Md_p0w9h6mHtzpuIEwNn>OnuRSC2=>fP^Hqgc)xu^4 z<3!s`cORHJh#?!nKI`Et7{3C27+EuH)Gw1f)aoP|B3y?fuVfvpYYmmukx0ya-)TQX zR{ggy5cNf4X|g)nl#jC9p>7|09_S7>1D2GTRBUTW zAkQ=JMRogZqG#v;^=11O6@rPPwvJkr{bW-Qg8`q8GoD#K`&Y+S#%&B>SGRL>;ZunM@49!}Uy zN|bBCJ%sO;@3wl0>0gbl3L@1^O60ONObz8ZI7nder>(udj-jt`;yj^nTQ$L9`OU9W zX4alF#$|GiR47%x@s&LV>2Sz2R6?;2R~5k6V>)nz!o_*1Y!$p>BC5&?hJg_MiE6UBy>RkVZj`9UWbRkN-Hk!S`=BS3t3uyX6)7SF#)71*}`~Ogz z1rap5H6~dhBJ83;q-Y<5V35C2&F^JI-it(=5D#v!fAi9p#UwV~2tZQI+W(Dv?1t9? zfh*xpxxO{-(VGB>!Q&0%^YW_F!@aZS#ucP|YaD#>wd1Fv&Z*SR&mc;asi}1G) z_H>`!akh-Zxq9#io(7%;a$)w+{QH)Y$?UK1Dt^4)up!Szcxnu}kn$0afcfJL#IL+S z5gF_Y30j;{lNrG6m~$Ay?)*V9fZuU@3=kd40=LhazjFrau>(Y>SJNtOz>8x_X-BlA zIpl{i>OarVGj1v(4?^1`R}aQB&WCRQzS~;7R{tDZG=HhgrW@B`W|#cdyj%YBky)P= zpxuOZkW>S6%q7U{VsB#G(^FMsH5QuGXhb(sY+!-R8Bmv6Sx3WzSW<1MPPN1!&PurYky(@`bP9tz z52}LH9Q?+FF5jR6-;|+GVdRA!qtd;}*-h&iIw3Tq3qF9sDIb1FFxGbo&fbG5n8$3F zyY&PWL{ys^dTO}oZ#@sIX^BKW*bon=;te9j5k+T%wJ zNJtoN1~YVj4~YRrlZl)b&kJqp+Z`DqT!la$x&&IxgOQw#yZd-nBP3!7FijBXD|IsU8Zl^ zc6?MKpJQ+7ka|tZQLfchD$PD|;K(9FiLE|eUZX#EZxhG!S-63C$jWX1Yd!6-Yxi-u zjULIr|0-Q%D9jz}IF~S%>0(jOqZ(Ln<$9PxiySr&2Oic7vb<8q=46)Ln%Z|<*z5&> z3f~Zw@m;vR(bESB<=Jqkxn(=#hQw42l(7)h`vMQQTttz9XW6^|^8EK7qhju4r_c*b zJIi`)MB$w@9epwdIfnEBR+?~);yd6C(LeMC& zn&&N*?-g&BBJcV;8&UoZi4Lmxcj16ojlxR~zMrf=O_^i1wGb9X-0@6_rpjPYemIin zmJb+;lHe;Yp=8G)Q(L1bzH*}I>}uAqhj4;g)PlvD9_e_ScR{Ipq|$8NvAvLD8MYr}xl=bU~)f%B3E>r3Bu9_t|ThF3C5~BdOve zEbk^r&r#PT&?^V1cb{72yEWH}TXEE}w>t!cY~rA+hNOTK8FAtIEoszp!qqptS&;r$ zaYV-NX96-h$6aR@1xz6_E0^N49mU)-v#bwtGJm)ibygzJ8!7|WIrcb`$XH~^!a#s& z{Db-0IOTFq#9!^j!n_F}#Z_nX{YzBK8XLPVmc&X`fT7!@$U-@2KM9soGbmOSAmqV z{nr$L^MBo_u^Joyf0E^=eo{Rt0{{e$IFA(#*kP@SQd6lWT2-#>` zP1)7_@IO!9lk>Zt?#CU?cuhiLF&)+XEM9B)cS(gvQT!X3`wL*{fArTS;Ak`J<84du zALKPz4}3nlG8Fo^MH0L|oK2-4xIY!~Oux~1sw!+It)&D3p;+N8AgqKI`ld6v71wy8I!eP0o~=RVcFQR2Gr(eP_JbSytoQ$Yt}l*4r@A8Me94y z8cTDWhqlq^qoAhbOzGBXv^Wa4vUz$(7B!mX`T=x_ueKRRDfg&Uc-e1+z4x$jyW_Pm zp?U;-R#xt^Z8Ev~`m`iL4*c#65Nn)q#=Y0l1AuD&+{|8-Gsij3LUZXpM0Bx0u7WWm zH|%yE@-#XEph2}-$-thl+S;__ciBxSSzHveP%~v}5I%u!z_l_KoW{KRx2=eB33umE zIYFtu^5=wGU`Jab8#}cnYry@9p5UE#U|VVvx_4l49JQ;jQdp(uw=$^A$EA$LM%vmE zvdEOaIcp5qX8wX{mYf0;#51~imYYPn4=k&#DsKTxo{_Mg*;S495?OBY?#gv=edYC* z^O@-sd-qa+U24xvcbL0@C7_6o!$`)sVr-jSJE4XQUQ$?L7}2(}Eixqv;L8AdJAVqc zq}RPgpnDb@E_;?6K58r3h4-!4rT4Ab#rLHLX?eMOfluJk=3i1@Gt1i#iA=O`M0@x! z(HtJP9BMHXEzuD93m|B&woj0g6T?f#^)>J>|I4C5?Gam>n9!8CT%~aT;=oco5d6U8 zMXl(=W;$ND_8+DD*?|5bJ!;8ebESXMUKBAf7YBwNVJibGaJ*(2G`F%wx)grqVPjudiaq^Kl&g$8A2 zWMxMr@_$c}d+;_B`#kUX-t|4VKH&_f^^EP0&=DPLW)H)UzBG%%Tra*5 z%$kyZe3I&S#gfie^z5)!twG={3Cuh)FdeA!Kj<-9** zvT*5%Tb`|QbE!iW-XcOuy39>D3oe6x{>&<#E$o8Ac|j)wq#kQzz|ATd=Z0K!p2$QE zPu?jL8Lb^y3_CQE{*}sTDe!2!dtlFjq&YLY@2#4>XS`}v#PLrpvc4*@q^O{mmnr5D zmyJq~t?8>FWU5vZdE(%4cuZuao0GNjp3~Dt*SLaxI#g_u>hu@k&9Ho*#CZP~lFJHj z(e!SYlLigyc?&5-YxlE{uuk$9b&l6d`uIlpg_z15dPo*iU&|Khx2*A5Fp;8iK_bdP z?T6|^7@lcx2j0T@x>X7|kuuBSB7<^zeY~R~4McconTxA2flHC0_jFxmSTv-~?zVT| zG_|yDqa9lkF*B6_{j=T>=M8r<0s;@z#h)3BQ4NLl@`Xr__o7;~M&dL3J8fP&zLfDfy z);ckcTev{@OUlZ`bCo(-3? z1u1xD`PKgSg?RqeVVsF<1SLF;XYA@Bsa&cY!I48ZJn1V<3d!?s=St?TLo zC0cNr`qD*M#s6f~X>SCNVkva^9A2ZP>CoJ9bvgXe_c}WdX-)pHM5m7O zrHt#g$F0AO+nGA;7dSJ?)|Mo~cf{z2L)Rz!`fpi73Zv)H=a5K)*$5sf_IZypi($P5 zsPwUc4~P-J1@^3C6-r9{V-u0Z&Sl7vNfmuMY4yy*cL>_)BmQF!8Om9Dej%cHxbIzA zhtV0d{=%cr?;bpBPjt@4w=#<>k5ee=TiWAXM2~tUGfm z$s&!Dm0R^V$}fOR*B^kGaipi~rx~A2cS0;t&khV1a4u38*XRUP~f za!rZMtay8bsLt6yFYl@>-y^31(*P!L^^s@mslZy(SMsv9bVoX`O#yBgEcjCmGpyc* zeH$Dw6vB5P*;jor+JOX@;6K#+xc)Z9B8M=x2a@Wx-{snPGpRmOC$zpsqW*JCh@M2Y z#K+M(>=#d^>Of9C`))h<=Bsy)6zaMJ&x-t%&+UcpLjV`jo4R2025 zXaG8EA!0lQa)|dx-@{O)qP6`$rhCkoQqZ`^SW8g-kOwrwsK8 z3ms*AIcyj}-1x&A&vSq{r=QMyp3CHdWH35!sad#!Sm>^|-|afB+Q;|Iq@LFgqIp#Z zD1%H+3I?6RGnk&IFo|u+E0dCxXz4yI^1i!QTu7uvIEH>i3rR{srcST`LIRwdV1P;W z+%AN1NIf@xxvVLiSX`8ILA8MzNqE&7>%jMzGt9wm78bo9<;h*W84i29^w!>V>{N+S zd`5Zmz^G;f=icvoOZfK5#1ctx*~UwD=ab4DGQXehQ!XYnak*dee%YN$_ZPL%KZuz$ zD;$PpT;HM^$KwtQm@7uvT`i6>Hae1CoRVM2)NL<2-k2PiX=eAx+-6j#JI?M}(tuBW zkF%jjLR)O`gI2fcPBxF^HeI|DWwQWHVR!;;{BXXHskxh8F@BMDn`oEi-NHt;CLymW z=KSv5)3dyzec0T5B*`g-MQ<;gz=nIWKUi9ko<|4I(-E0k$QncH>E4l z**1w&#={&zv4Tvhgz#c29`m|;lU-jmaXFMC11 z*dlXDMEOG>VoLMc>!rApwOu2prKSi*!w%`yzGmS+k(zm*CsLK*wv{S_0WX^8A-rKy zbk^Gf_92^7iB_uUF)EE+ET4d|X|>d&mdN?x@vxKAQk`O+r4Qdu>XGy(a(19g;=jU} zFX{O*_NG>!$@jh!U369Lnc+D~qch3uT+_Amyi}*k#LAAwh}k8IPK5a-WZ81ufD>l> z$4cF}GSz>ce`3FAic}6W4Z7m9KGO?(eWqi@L|5Hq0@L|&2flN1PVl}XgQ2q*_n2s3 zt5KtowNkTYB5b;SVuoXA@i5irXO)A&%7?V`1@HGCB&)Wgk+l|^XXChq;u(nyPB}b3 zY>m5jkxpZgi)zfbgv&ec4Zqdvm+D<?Im*mXweS9H+V>)zF#Zp3)bhl$PbISY{5=_z!8&*Jv~NYtI-g!>fDs zmvL5O^U%!^VaKA9gvKw|5?-jk>~%CVGvctKmP$kpnpfN{D8@X*Aazi$txfa%vd-|E z>kYmV66W!lNekJPom29LdZ%(I+ZLZYTXzTg*to~m?7vp%{V<~>H+2}PQ?PPAq`36R z<%wR8v6UkS>Wt#hzGk#44W<%9S=nBfB);6clKwnxY}T*w21Qc3_?IJ@4gYzC7s;WP zVQNI(M=S=JT#xsZy7G`cR(BP9*je0bfeN8JN5~zY(DDs0t{LpHOIbN);?T-69Pf3R zSNe*&p2%AwXHL>__g+xd4Hlc_vu<25H?(`nafS%)3UPP7_4;gk-9ckt8SJRTv5v0M z_Hww`qPudL?ajIR&X*;$y-`<)6dxx1U~5eGS13CB!lX;3w7n&lDDiArbAhSycd}+b zya_3p@A`$kQy;|NJZ~s44Hqo7Hwt}X86NK=(ey>lgWTtGL6k@Gy;PbO!M%1~Wcn2k zUFP|*5d>t-X*RU8g%>|(wwj*~#l4z^Aatf^DWd1Wj#Q*AY0D^V@sC`M zjJc6qXu0I7Y*2;;gGu!plAFzG=J;1%eIOdn zQA>J&e05UN*7I5@yRhK|lbBSfJ+5Uq;!&HV@xfPZrgD}kE*1DSq^=%{o%|LChhl#0 zlMb<^a6ixzpd{kNZr|3jTGeEzuo}-eLT-)Q$#b{!vKx8Tg}swCni>{#%vDY$Ww$84 zew3c9BBovqb}_&BRo#^!G(1Eg((BScRZ}C)Oz?y`T5wOrv);)b^4XR8 zhJo7+<^7)qB>I;46!GySzdneZ>n_E1oWZY;kf94#)s)kWjuJN1c+wbVoNQcmnv}{> zN0pF+Sl3E}UQ$}slSZeLJrwT>Sr}#V(dVaezCQl2|4LN`7L7v&siYR|r7M(*JYfR$ zst3=YaDw$FSc{g}KHO&QiKxuhEzF{f%RJLKe3p*7=oo`WNP)M(9X1zIQPP0XHhY3c znrP{$4#Ol$A0s|4S7Gx2L23dv*Gv2o;h((XVn+9+$qvm}s%zi6nI-_s6?mG! zj{DV;qesJb&owKeEK?=J>UcAlYckA7Sl+I&IN=yasrZOkejir*kE@SN`fk<8Fgx*$ zy&fE6?}G)d_N`){P~U@1jRVA|2*69)KSe_}!~?+`Yb{Y=O~_+@!j<&oVQQMnhoIRU zA0CyF1OFfkK44n*JD~!2!SCPM;PRSk%1XL=0&rz00wxPs&-_eapJy#$h!eqY%nS0{ z!aGg58JIJPF3_ci%n)QSVpa2H`vIe$RD43;#IRfDV&Ibit z+?>HW4{2wOfC6Fw)}4x}i1maDxcE1qi@BS*qcxD2gE@h3#4cgU*D-&3z7D|tVZWt= z-Cy2+*Cm@P4GN_TPUtaVyVesbVDazF@)j8VJ4>XZv!f%}&eO1SvIgr}4`A*3#vat< z_MoByL(qW6L7SFZ#|Gc1fFN)L2PxY+{B8tJp+pxRyz*87)vXR}*=&ahXjBlQKguuf zX6x<<6fQulE^C*KH8~W%ptpaC0l?b=_{~*U4?5Vt;dgM4t_{&UZ1C2j?b>b+5}{IF_CUyvz-@QZPMlJ)r_tS$9kH%RPv#2_nMb zRLj5;chJ72*U`Z@Dqt4$@_+k$%|8m(HqLG!qT4P^DdfvGf&){gKnGCX#H0!;W=AGP zbA&Z`-__a)VTS}kKFjWGk z%|>yE?t*EJ!qeQ%dPk$;xIQ+P0;()PCBDgjJm6Buj{f^awNoVx+9<|lg3%-$G(*f) zll6oOkN|yamn1uyl2*N-lnqRI1cvs_JxLTeahEK=THV$Sz*gQhKNb*p0fNoda#-&F zB-qJgW^g}!TtM|0bS2QZekW7_tKu%GcJ!4?lObt0z_$mZ4rbQ0o=^curCs3bJK6sq z9fu-aW-l#>z~ca(B;4yv;2RZ?tGYAU)^)Kz{L|4oPj zdOf_?de|#yS)p2v8-N||+XL=O*%3+y)oI(HbM)Ds?q8~HPzIP(vs*G`iddbWq}! z(2!VjP&{Z1w+%eUq^ '} + case $link in #( + /*) app_path=$link ;; #( + *) app_path=$APP_HOME$link ;; + esac +done + +APP_HOME=$( cd "${APP_HOME:-./}" && pwd -P ) || exit + +APP_NAME="Gradle" +APP_BASE_NAME=${0##*/} + +# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. +DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"' + +# Use the maximum available, or set MAX_FD != -1 to use that value. +MAX_FD=maximum + +warn () { + echo "$*" +} >&2 + +die () { + echo + echo "$*" + echo + exit 1 +} >&2 + +# OS specific support (must be 'true' or 'false'). +cygwin=false +msys=false +darwin=false +nonstop=false +case "$( uname )" in #( + CYGWIN* ) cygwin=true ;; #( + Darwin* ) darwin=true ;; #( + MSYS* | MINGW* ) msys=true ;; #( + NONSTOP* ) nonstop=true ;; +esac + +CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar + + +# Determine the Java command to use to start the JVM. +if [ -n "$JAVA_HOME" ] ; then + if [ -x "$JAVA_HOME/jre/sh/java" ] ; then + # IBM's JDK on AIX uses strange locations for the executables + JAVACMD=$JAVA_HOME/jre/sh/java + else + JAVACMD=$JAVA_HOME/bin/java + fi + if [ ! -x "$JAVACMD" ] ; then + die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME + +Please set the JAVA_HOME variable in your environment to match the +location of your Java installation." + fi +else + JAVACMD=java + which java >/dev/null 2>&1 || die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. + +Please set the JAVA_HOME variable in your environment to match the +location of your Java installation." +fi + +# Increase the maximum file descriptors if we can. +if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then + case $MAX_FD in #( + max*) + MAX_FD=$( ulimit -H -n ) || + warn "Could not query maximum file descriptor limit" + esac + case $MAX_FD in #( + '' | soft) :;; #( + *) + ulimit -n "$MAX_FD" || + warn "Could not set maximum file descriptor limit to $MAX_FD" + esac +fi + +# Collect all arguments for the java command, stacking in reverse order: +# * args from the command line +# * the main class name +# * -classpath +# * -D...appname settings +# * --module-path (only if needed) +# * DEFAULT_JVM_OPTS, JAVA_OPTS, and GRADLE_OPTS environment variables. + +# For Cygwin or MSYS, switch paths to Windows format before running java +if "$cygwin" || "$msys" ; then + APP_HOME=$( cygpath --path --mixed "$APP_HOME" ) + CLASSPATH=$( cygpath --path --mixed "$CLASSPATH" ) + + JAVACMD=$( cygpath --unix "$JAVACMD" ) + + # Now convert the arguments - kludge to limit ourselves to /bin/sh + for arg do + if + case $arg in #( + -*) false ;; # don't mess with options #( + /?*) t=${arg#/} t=/${t%%/*} # looks like a POSIX filepath + [ -e "$t" ] ;; #( + *) false ;; + esac + then + arg=$( cygpath --path --ignore --mixed "$arg" ) + fi + # Roll the args list around exactly as many times as the number of + # args, so each arg winds up back in the position where it started, but + # possibly modified. + # + # NB: a `for` loop captures its iteration list before it begins, so + # changing the positional parameters here affects neither the number of + # iterations, nor the values presented in `arg`. + shift # remove old arg + set -- "$@" "$arg" # push replacement arg + done +fi + +# Collect all arguments for the java command; +# * $DEFAULT_JVM_OPTS, $JAVA_OPTS, and $GRADLE_OPTS can contain fragments of +# shell script including quotes and variable substitutions, so put them in +# double quotes to make sure that they get re-expanded; and +# * put everything else in single quotes, so that it's not re-expanded. + +set -- \ + "-Dorg.gradle.appname=$APP_BASE_NAME" \ + -classpath "$CLASSPATH" \ + org.gradle.wrapper.GradleWrapperMain \ + "$@" + +# Use "xargs" to parse quoted args. +# +# With -n1 it outputs one arg per line, with the quotes and backslashes removed. +# +# In Bash we could simply go: +# +# readarray ARGS < <( xargs -n1 <<<"$var" ) && +# set -- "${ARGS[@]}" "$@" +# +# but POSIX shell has neither arrays nor command substitution, so instead we +# post-process each arg (as a line of input to sed) to backslash-escape any +# character that might be a shell metacharacter, then use eval to reverse +# that process (while maintaining the separation between arguments), and wrap +# the whole thing up as a single "set" statement. +# +# This will of course break if any of these variables contains a newline or +# an unmatched quote. +# + +eval "set -- $( + printf '%s\n' "$DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS" | + xargs -n1 | + sed ' s~[^-[:alnum:]+,./:=@_]~\\&~g; ' | + tr '\n' ' ' + )" '"$@"' + +exec "$JAVACMD" "$@" diff --git a/java-connectors/gradlew.bat b/java-connectors/gradlew.bat new file mode 100644 index 000000000..ac1b06f93 --- /dev/null +++ b/java-connectors/gradlew.bat @@ -0,0 +1,89 @@ +@rem +@rem Copyright 2015 the original author or authors. +@rem +@rem Licensed under the Apache License, Version 2.0 (the "License"); +@rem you may not use this file except in compliance with the License. +@rem You may obtain a copy of the License at +@rem +@rem https://www.apache.org/licenses/LICENSE-2.0 +@rem +@rem Unless required by applicable law or agreed to in writing, software +@rem distributed under the License is distributed on an "AS IS" BASIS, +@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +@rem See the License for the specific language governing permissions and +@rem limitations under the License. +@rem + +@if "%DEBUG%" == "" @echo off +@rem ########################################################################## +@rem +@rem Gradle startup script for Windows +@rem +@rem ########################################################################## + +@rem Set local scope for the variables with windows NT shell +if "%OS%"=="Windows_NT" setlocal + +set DIRNAME=%~dp0 +if "%DIRNAME%" == "" set DIRNAME=. +set APP_BASE_NAME=%~n0 +set APP_HOME=%DIRNAME% + +@rem Resolve any "." and ".." in APP_HOME to make it shorter. +for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi + +@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. +set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m" + +@rem Find java.exe +if defined JAVA_HOME goto findJavaFromJavaHome + +set JAVA_EXE=java.exe +%JAVA_EXE% -version >NUL 2>&1 +if "%ERRORLEVEL%" == "0" goto execute + +echo. +echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. +echo. +echo Please set the JAVA_HOME variable in your environment to match the +echo location of your Java installation. + +goto fail + +:findJavaFromJavaHome +set JAVA_HOME=%JAVA_HOME:"=% +set JAVA_EXE=%JAVA_HOME%/bin/java.exe + +if exist "%JAVA_EXE%" goto execute + +echo. +echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% +echo. +echo Please set the JAVA_HOME variable in your environment to match the +echo location of your Java installation. + +goto fail + +:execute +@rem Setup the command line + +set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar + + +@rem Execute Gradle +"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %* + +:end +@rem End local scope for the variables with windows NT shell +if "%ERRORLEVEL%"=="0" goto mainEnd + +:fail +rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of +rem the _cmd.exe /c_ return code! +if not "" == "%GRADLE_EXIT_CONSOLE%" exit 1 +exit /b 1 + +:mainEnd +if "%OS%"=="Windows_NT" endlocal + +:omega diff --git a/java-connectors/kafka-connect-azure-eventhubs/build.gradle b/java-connectors/kafka-connect-azure-eventhubs/build.gradle new file mode 100644 index 000000000..60ea80b6c --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/build.gradle @@ -0,0 +1,19 @@ +project(':kafka-connect-azure-eventhubs') { + + + test { + maxParallelForks = 1 + } + + dependencies { + implementation project(':kafka-connect-common') + implementation project(':kafka-connect-query-language') + +// //azure-specific dependencies in case we want to change from kafka protocol +// implementation group: 'com.azure', name: 'azure-identity', version: '1.11.2' +// implementation group: 'com.azure', name: 'azure-messaging-eventhubs', version: '5.18.0' +// implementation group: 'com.azure', name: 'azure-storage-blob', version: '12.25.1' +// implementation group: 'com.azure', name: 'azure-messaging-eventhubs-checkpointstore-blob', version: '1.19.0' + + } +} \ No newline at end of file diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/config/AzureEventHubsConfigConstants.java b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/config/AzureEventHubsConfigConstants.java new file mode 100644 index 000000000..35fab9bb6 --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/config/AzureEventHubsConfigConstants.java @@ -0,0 +1,50 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.config; + +import io.lenses.streamreactor.connect.azure.eventhubs.source.AzureEventHubsSourceConnector; + +/** + * Class represents Config Constants for AzureEventHubsSourceConnector Config Definition. + */ +public class AzureEventHubsConfigConstants { + + + private static final String DOT = "."; + public static final String OPTIONAL_EMPTY_DEFAULT = ""; + public static final String CONNECTOR_PREFIX = "connect.eventhubs"; + public static final String SOURCE_CONNECTOR_PREFIX = CONNECTOR_PREFIX + DOT + "source"; + + public static final String CONNECTOR_NAME = "name"; + public static final String CONNECTOR_NAME_DOC = "Connector's name"; + public static final String CONNECTOR_NAME_DEFAULT = AzureEventHubsSourceConnector.class.getSimpleName(); + + public static final String CONNECTOR_WITH_CONSUMER_PREFIX = + SOURCE_CONNECTOR_PREFIX + DOT + "connection.settings" + DOT; + public static final String CONSUMER_OFFSET = SOURCE_CONNECTOR_PREFIX + DOT + "default.offset"; + public static final String CONSUMER_OFFSET_DOC = + "Specifies whether by default we should consumer from earliest (default) or latest offset."; + public static final String CONSUMER_OFFSET_DEFAULT = "earliest"; + public static final String CONSUMER_CLOSE_TIMEOUT = SOURCE_CONNECTOR_PREFIX + DOT + "close.timeout"; + public static final String CONSUMER_CLOSE_TIMEOUT_DOC = + "Specifies timeout for consumer closing."; + public static final String CONSUMER_CLOSE_TIMEOUT_DEFAULT = "30"; + + public static final String KCQL_CONFIG = CONNECTOR_PREFIX + DOT + "kcql"; + public static final String KCQL_DOC = + "KCQL expression describing field selection and data routing to the target."; + +} diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/config/AzureEventHubsSourceConfig.java b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/config/AzureEventHubsSourceConfig.java new file mode 100644 index 000000000..fba2ccd28 --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/config/AzureEventHubsSourceConfig.java @@ -0,0 +1,122 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.config; + +import io.lenses.streamreactor.common.config.base.BaseConfig; +import io.lenses.streamreactor.common.config.base.intf.ConnectorPrefixed; +import java.util.Map; +import java.util.Set; +import java.util.function.UnaryOperator; +import lombok.Getter; +import org.apache.kafka.clients.consumer.ConsumerConfig; +import org.apache.kafka.common.config.ConfigDef; +import org.apache.kafka.common.config.ConfigDef.Importance; +import org.apache.kafka.common.config.ConfigDef.Type; + +/** + * Class represents Config Definition for AzureEventHubsSourceConnector. It additionally adds + * configs from org.apache.kafka.clients.consumer.ConsumerConfig but adds standard Connector + * prefixes to them. + */ +public class AzureEventHubsSourceConfig extends BaseConfig implements ConnectorPrefixed { + + public static final String CONNECTION_GROUP = "Connection"; + + private static final UnaryOperator CONFIG_NAME_PREFIX_APPENDER = name -> + AzureEventHubsConfigConstants.CONNECTOR_WITH_CONSUMER_PREFIX + name; + + private static final Set EXCLUDED_CONSUMER_PROPERTIES = + Set.of(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, + ConsumerConfig.GROUP_ID_CONFIG, ConsumerConfig.CLIENT_ID_CONFIG, ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG); + + + @Getter + static ConfigDef configDefinition; + + static { + ConfigDef kafkaConsumerConfigToExpose = getKafkaConsumerConfigToExpose(); + configDefinition = new ConfigDef(kafkaConsumerConfigToExpose) + .define(AzureEventHubsConfigConstants.CONNECTOR_NAME, + Type.STRING, + AzureEventHubsConfigConstants.CONNECTOR_NAME_DEFAULT, + Importance.HIGH, + AzureEventHubsConfigConstants.CONNECTOR_NAME_DOC, + CONNECTION_GROUP, + 1, + ConfigDef.Width.LONG, + AzureEventHubsConfigConstants.CONNECTOR_NAME + ).define(AzureEventHubsConfigConstants.CONSUMER_CLOSE_TIMEOUT, + Type.INT, + AzureEventHubsConfigConstants.CONSUMER_CLOSE_TIMEOUT_DEFAULT, + Importance.MEDIUM, + AzureEventHubsConfigConstants.CONSUMER_CLOSE_TIMEOUT_DOC, + CONNECTION_GROUP, + 3, + ConfigDef.Width.LONG, + AzureEventHubsConfigConstants.CONSUMER_CLOSE_TIMEOUT + ) + .define(AzureEventHubsConfigConstants.CONSUMER_OFFSET, + Type.STRING, + AzureEventHubsConfigConstants.CONSUMER_OFFSET_DEFAULT, + Importance.MEDIUM, + AzureEventHubsConfigConstants.CONSUMER_OFFSET_DOC, + CONNECTION_GROUP, + 4, + ConfigDef.Width.LONG, + AzureEventHubsConfigConstants.CONSUMER_OFFSET + ).define(AzureEventHubsConfigConstants.KCQL_CONFIG, + Type.STRING, + Importance.HIGH, + AzureEventHubsConfigConstants.KCQL_DOC, + "Mappings", + 1, + ConfigDef.Width.LONG, + AzureEventHubsConfigConstants.KCQL_CONFIG + ); + } + + public AzureEventHubsSourceConfig(Map properties) { + super(AzureEventHubsConfigConstants.CONNECTOR_PREFIX, getConfigDefinition(), properties); + } + + /** + * Provides prefixed KafkaConsumerConfig key. + * + * @param kafkaConsumerConfigKey from org.apache.kafka.clients.consumer.ConsumerConfig + * @return prefixed key. + */ + public static String getPrefixedKafkaConsumerConfigKey(String kafkaConsumerConfigKey) { + return CONFIG_NAME_PREFIX_APPENDER.apply(kafkaConsumerConfigKey); + } + + @Override + public String connectorPrefix() { + return connectorPrefix; + } + + private static ConfigDef getKafkaConsumerConfigToExpose() { + ConfigDef kafkaConsumerConfigToExpose = new ConfigDef(); + ConsumerConfig.configDef().configKeys().values().stream() + .filter(configKey -> !EXCLUDED_CONSUMER_PROPERTIES.contains(configKey.name)) + .forEach(configKey -> kafkaConsumerConfigToExpose.define( + CONFIG_NAME_PREFIX_APPENDER.apply(configKey.name), + configKey.type, configKey.defaultValue, + configKey.importance, configKey.documentation, configKey.group, + configKey.orderInGroup, configKey.width, configKey.displayName)); + + return kafkaConsumerConfigToExpose; + } +} diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/config/SourceDataType.java b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/config/SourceDataType.java new file mode 100644 index 000000000..92722e393 --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/config/SourceDataType.java @@ -0,0 +1,69 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.config; + +import java.util.Arrays; +import java.util.Map; +import java.util.function.Function; +import java.util.stream.Collectors; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import org.apache.kafka.common.serialization.ByteArrayDeserializer; +import org.apache.kafka.common.serialization.Deserializer; +import org.apache.kafka.connect.data.Schema; + +/** + * Class to indicate what kind of data is being received from Kafka Consumer. + */ +@Getter +public enum SourceDataType { + + BYTES(ByteArrayDeserializer.class, Schema.OPTIONAL_BYTES_SCHEMA); + + private final Class deserializerClass; + private final Schema schema; + private static final Map NAME_TO_DATA_SERIALIZER_TYPE; + + static { + NAME_TO_DATA_SERIALIZER_TYPE = + Arrays.stream(values()).collect(Collectors.toMap(Enum::name, Function.identity())); + } + + SourceDataType(Class deserializerClass, Schema schema) { + this.deserializerClass = deserializerClass; + this.schema = schema; + } + + public static SourceDataType fromName(String name) { + return NAME_TO_DATA_SERIALIZER_TYPE.get(name.toUpperCase()); + } + + /** + * Class indicates what data types are being transferred by Task. + */ + @Getter + @EqualsAndHashCode + public static class KeyValueTypes { + private final SourceDataType keyType; + private final SourceDataType valueType; + public static final KeyValueTypes DEFAULT_TYPES = new KeyValueTypes(BYTES, BYTES); + + public KeyValueTypes(SourceDataType keyType, SourceDataType valueType) { + this.keyType = keyType; + this.valueType = valueType; + } + } +} diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/mapping/SourceRecordMapper.java b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/mapping/SourceRecordMapper.java new file mode 100644 index 000000000..0c5f0436c --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/mapping/SourceRecordMapper.java @@ -0,0 +1,80 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.mapping; + +import java.util.Map; +import org.apache.kafka.clients.consumer.ConsumerRecord; +import org.apache.kafka.common.header.Header; +import org.apache.kafka.connect.data.Schema; +import org.apache.kafka.connect.data.SchemaAndValue; +import org.apache.kafka.connect.header.ConnectHeaders; +import org.apache.kafka.connect.source.SourceRecord; + +/** + * Class with utility method to convert to SourceRecord. + */ +public class SourceRecordMapper { + + /** + * Method to make SourceRecord out of ConsumerRecord including optional byte headers from original + * message. + * + * @param consumerRecord original consumer record + * @param partitionKey AzureTopicPartitionKey to indicate topic and partition + * @param offsetMap AzureOffsetMarker to indicate offset + * @param outputTopic Output topic for record + * @param keySchema Schema of the key + * @param valueSchema Schema of the value + * @return SourceRecord with headers + */ + public static SourceRecord mapSourceRecordIncludingHeaders( + ConsumerRecord consumerRecord, + Map partitionKey, Map offsetMap, + String outputTopic, Schema keySchema, Schema valueSchema) { + Iterable

headers = consumerRecord.headers(); + ConnectHeaders connectHeaders = new ConnectHeaders(); + for (Header header : headers) { + connectHeaders.add(header.key(), + new SchemaAndValue(Schema.OPTIONAL_BYTES_SCHEMA, header.value())); + } + return new SourceRecord(partitionKey, offsetMap, + outputTopic, null, keySchema, consumerRecord.key(), + valueSchema, consumerRecord.value(), consumerRecord.timestamp(), + connectHeaders); + } + + /** + * Method to make SourceRecord out of ConsumerRecord including optional byte headers + * from original message. + * + * @param consumerRecord original consumer record + * @param partitionKey partitionKey to indicate topic and partition + * @param offsetMap AzureOffsetMarker to indicate offset + * @param outputTopic Output topic for record + * @param keySchema Schema of the key + * @param valueSchema Schema of the value + * @return SourceRecord without headers + */ + public static SourceRecord mapSourceRecordWithoutHeaders( + ConsumerRecord consumerRecord, + Map partitionKey, Map offsetMap, + String outputTopic, Schema keySchema, Schema valueSchema) { + return new SourceRecord(partitionKey, offsetMap, + outputTopic, null, keySchema, consumerRecord.key(), + valueSchema, consumerRecord.value(), consumerRecord.timestamp(), null); + } + +} diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureConsumerRebalancerListener.java b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureConsumerRebalancerListener.java new file mode 100644 index 000000000..2b800e759 --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureConsumerRebalancerListener.java @@ -0,0 +1,80 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.source; + +import io.lenses.streamreactor.connect.azure.eventhubs.source.TopicPartitionOffsetProvider.AzureOffsetMarker; +import io.lenses.streamreactor.connect.azure.eventhubs.source.TopicPartitionOffsetProvider.AzureTopicPartitionKey; +import java.util.ArrayList; +import java.util.Collection; +import java.util.List; +import java.util.Optional; +import lombok.extern.slf4j.Slf4j; +import org.apache.kafka.clients.consumer.Consumer; +import org.apache.kafka.clients.consumer.ConsumerRebalanceListener; +import org.apache.kafka.common.TopicPartition; + +/** + * This class is an implementation of {@link ConsumerRebalanceListener} that can be used to provide + * OnlyOnce support and seek consumers into relevant offsets if needed. + */ +@Slf4j +public class AzureConsumerRebalancerListener implements ConsumerRebalanceListener { + + private final boolean shouldSeekToLatest; + private final TopicPartitionOffsetProvider topicPartitionOffsetProvider; + private final Consumer kafkaConsumer; + + /** + * Constructs {@link AzureConsumerRebalancerListener} for particular Kafka Consumer. + * + * @param topicPartitionOffsetProvider provider of committed offsets + * @param kafkaConsumer Kafka Consumer + * @param shouldSeekToLatest informs whether we should seek to latest or earliest if no offsets found + */ + public AzureConsumerRebalancerListener( + TopicPartitionOffsetProvider topicPartitionOffsetProvider, + Consumer kafkaConsumer, boolean shouldSeekToLatest) { + this.topicPartitionOffsetProvider = topicPartitionOffsetProvider; + this.kafkaConsumer = kafkaConsumer; + this.shouldSeekToLatest = shouldSeekToLatest; + } + + @Override + public void onPartitionsRevoked(Collection partitions) { + // implementation not needed, offsets already committed + } + + @Override + public void onPartitionsAssigned(Collection partitions) { + List partitionsWithoutOffsets = new ArrayList<>(); + partitions.forEach(partition -> { + AzureTopicPartitionKey partitionKey = new AzureTopicPartitionKey( + partition.topic(), partition.partition()); + Optional partitionOffset = topicPartitionOffsetProvider.getOffset(partitionKey); + partitionOffset.ifPresentOrElse( + offset -> kafkaConsumer.seek(partition, offset.getOffsetValue()), + () -> partitionsWithoutOffsets.add(partition)); + }); + if (!partitionsWithoutOffsets.isEmpty()) { + if (shouldSeekToLatest) { + kafkaConsumer.seekToEnd(partitionsWithoutOffsets); + } else { + kafkaConsumer.seekToBeginning(partitionsWithoutOffsets); + } + } + } + +} diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureEventHubsSourceConnector.java b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureEventHubsSourceConnector.java new file mode 100644 index 000000000..d0d019f65 --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureEventHubsSourceConnector.java @@ -0,0 +1,90 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.source; + +import static io.lenses.streamreactor.common.util.AsciiArtPrinter.printAsciiHeader; + +import io.lenses.streamreactor.common.util.JarManifest; +import io.lenses.streamreactor.connect.azure.eventhubs.config.AzureEventHubsConfigConstants; +import io.lenses.streamreactor.connect.azure.eventhubs.config.AzureEventHubsSourceConfig; +import io.lenses.streamreactor.connect.azure.eventhubs.util.KcqlConfigTopicMapper; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.stream.IntStream; +import lombok.extern.slf4j.Slf4j; +import org.apache.kafka.common.config.ConfigDef; +import org.apache.kafka.connect.connector.Task; +import org.apache.kafka.connect.source.ExactlyOnceSupport; +import org.apache.kafka.connect.source.SourceConnector; + +/** + * Implementation of {@link SourceConnector} for Microsoft Azure EventHubs. + */ +@Slf4j +public class AzureEventHubsSourceConnector extends SourceConnector { + + private final JarManifest jarManifest = + new JarManifest(getClass().getProtectionDomain().getCodeSource().getLocation()); + private Map configProperties; + + @Override + public void start(Map props) { + configProperties = props; + parseAndValidateConfigs(props); + printAsciiHeader(jarManifest, "/azure-eventhubs-ascii.txt"); + } + + @Override + public Class taskClass() { + return AzureEventHubsSourceTask.class; + } + + @Override + public List> taskConfigs(int maxTasks) { + log.info("Setting task configurations for {} workers.", maxTasks); + List> taskConfigs = new ArrayList<>(maxTasks); + + IntStream.range(0, maxTasks).forEach(task -> taskConfigs.add(configProperties)); + return taskConfigs; + } + + @Override + public ExactlyOnceSupport exactlyOnceSupport(Map connectorConfig) { + return ExactlyOnceSupport.SUPPORTED; + } + + @Override + public void stop() { + // connector-specific implementation not needed + } + + @Override + public ConfigDef config() { + return AzureEventHubsSourceConfig.getConfigDefinition(); + } + + @Override + public String version() { + return jarManifest.getVersion(); + } + + private static void parseAndValidateConfigs(Map props) { + AzureEventHubsSourceConfig azureEventHubsSourceConfig = new AzureEventHubsSourceConfig(props); + String kcqlMappings = azureEventHubsSourceConfig.getString(AzureEventHubsConfigConstants.KCQL_CONFIG); + KcqlConfigTopicMapper.mapInputToOutputsFromConfig(kcqlMappings); + } +} \ No newline at end of file diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureEventHubsSourceTask.java b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureEventHubsSourceTask.java new file mode 100644 index 000000000..8c63aa08d --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureEventHubsSourceTask.java @@ -0,0 +1,109 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.source; + +import static java.util.Optional.ofNullable; + +import io.lenses.streamreactor.common.util.JarManifest; +import io.lenses.streamreactor.connect.azure.eventhubs.config.AzureEventHubsConfigConstants; +import io.lenses.streamreactor.connect.azure.eventhubs.config.AzureEventHubsSourceConfig; +import io.lenses.streamreactor.connect.azure.eventhubs.util.KcqlConfigTopicMapper; +import java.time.Duration; +import java.time.temporal.ChronoUnit; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ArrayBlockingQueue; +import lombok.extern.slf4j.Slf4j; +import org.apache.kafka.clients.consumer.ConsumerRecords; +import org.apache.kafka.connect.source.SourceRecord; +import org.apache.kafka.connect.source.SourceTask; +import org.apache.kafka.connect.storage.OffsetStorageReader; + +/** + * Implementation of {@link SourceTask} for Microsoft Azure EventHubs. + */ +@Slf4j +public class AzureEventHubsSourceTask extends SourceTask { + + private static final Duration ONE_SECOND_DURATION = Duration.of(1, ChronoUnit.SECONDS); + private Duration closeTimeout; + private static final int RECORDS_QUEUE_DEFAULT_SIZE = 10; + private final JarManifest jarManifest; + private EventHubsKafkaConsumerController eventHubsKafkaConsumerController; + private BlockingQueueProducerProvider blockingQueueProducerProvider; + + public AzureEventHubsSourceTask() { + jarManifest = new JarManifest(getClass().getProtectionDomain().getCodeSource().getLocation()); + } + + public AzureEventHubsSourceTask(JarManifest jarManifest) { + this.jarManifest = jarManifest; + } + + @Override + public String version() { + return jarManifest.getVersion(); + } + + @Override + public void start(Map props) { + OffsetStorageReader offsetStorageReader = ofNullable(this.context).flatMap( + context -> ofNullable(context.offsetStorageReader())).orElseThrow(); + AzureEventHubsSourceConfig azureEventHubsSourceConfig = new AzureEventHubsSourceConfig(props); + TopicPartitionOffsetProvider topicPartitionOffsetProvider = new TopicPartitionOffsetProvider(offsetStorageReader); + + ArrayBlockingQueue> recordsQueue = new ArrayBlockingQueue<>( + RECORDS_QUEUE_DEFAULT_SIZE); + Map inputToOutputTopics = KcqlConfigTopicMapper.mapInputToOutputsFromConfig( + azureEventHubsSourceConfig.getString(AzureEventHubsConfigConstants.KCQL_CONFIG)); + blockingQueueProducerProvider = new BlockingQueueProducerProvider(topicPartitionOffsetProvider); + KafkaByteBlockingQueuedProducer producer = blockingQueueProducerProvider.createProducer( + azureEventHubsSourceConfig, recordsQueue, inputToOutputTopics); + EventHubsKafkaConsumerController kafkaConsumerController = new EventHubsKafkaConsumerController( + producer, recordsQueue, inputToOutputTopics); + initialize(kafkaConsumerController, azureEventHubsSourceConfig); + } + + /** + * Initializes the Task. This method shouldn't be called if start() was already called with + * {@link EventHubsKafkaConsumerController} instance. + * + * @param eventHubsKafkaConsumerController {@link EventHubsKafkaConsumerController} for this task + * @param azureEventHubsSourceConfig config for task + */ + public void initialize(EventHubsKafkaConsumerController eventHubsKafkaConsumerController, + AzureEventHubsSourceConfig azureEventHubsSourceConfig) { + this.eventHubsKafkaConsumerController = eventHubsKafkaConsumerController; + closeTimeout = + Duration.of(azureEventHubsSourceConfig.getInt(AzureEventHubsConfigConstants.CONSUMER_CLOSE_TIMEOUT), + ChronoUnit.SECONDS); + log.info("{} initialised.", getClass().getSimpleName()); + } + + + @Override + public List poll() throws InterruptedException { + List poll = + eventHubsKafkaConsumerController.poll(ONE_SECOND_DURATION); + return poll.isEmpty() ? null : poll; + } + + @Override + public void stop() { + ofNullable(eventHubsKafkaConsumerController) + .ifPresent(consumerController -> consumerController.close(closeTimeout)); + } +} diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/BlockingQueueProducer.java b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/BlockingQueueProducer.java new file mode 100644 index 000000000..3cf3e54b9 --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/BlockingQueueProducer.java @@ -0,0 +1,37 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.source; + +import java.time.Duration; + +/** + * Interface of stoppable BockingQueue Producer. + */ +public interface BlockingQueueProducer { + + /** + * Method to start production to specified BlockingQueue. + */ + void start(); + + /** + * Method to stop production to specified BlockingQueue. + * + * @param timeoutDuration maximum time to stop the Producer + */ + void stop(Duration timeoutDuration); + +} diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/BlockingQueueProducerProvider.java b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/BlockingQueueProducerProvider.java new file mode 100644 index 000000000..c9b3babec --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/BlockingQueueProducerProvider.java @@ -0,0 +1,104 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.source; + +import io.lenses.streamreactor.connect.azure.eventhubs.config.AzureEventHubsConfigConstants; +import io.lenses.streamreactor.connect.azure.eventhubs.config.AzureEventHubsSourceConfig; +import io.lenses.streamreactor.connect.azure.eventhubs.config.SourceDataType.KeyValueTypes; +import java.util.Map; +import java.util.Set; +import java.util.UUID; +import java.util.concurrent.BlockingQueue; +import lombok.extern.slf4j.Slf4j; +import org.apache.kafka.clients.consumer.ConsumerConfig; +import org.apache.kafka.clients.consumer.ConsumerRecords; +import org.apache.kafka.clients.consumer.KafkaConsumer; +import org.apache.kafka.common.config.ConfigException; + +/** + * Provider for BlockingQueuedKafkaConsumers. + */ +@Slf4j +public class BlockingQueueProducerProvider implements ProducerProvider { + + private static final boolean STRIP_PREFIX = true; + private static final String EARLIEST_OFFSET = "earliest"; + private static final String LATEST_OFFSET = "latest"; + private static final String CONSUMER_OFFSET_EXCEPTION_MESSAGE = + "allowed values are: earliest/latest"; + private final TopicPartitionOffsetProvider topicPartitionOffsetProvider; + + + public BlockingQueueProducerProvider(TopicPartitionOffsetProvider topicPartitionOffsetProvider) { + this.topicPartitionOffsetProvider = topicPartitionOffsetProvider; + } + + /** + * Instantiates BlockingQueuedKafkaConsumer from given properties. + * + * @param azureEventHubsSourceConfig Config of Task + * @param recordBlockingQueue BlockingQueue for ConsumerRecords + * @param inputToOutputTopics map of input to output topics + * @return BlockingQueuedKafkaConsumer instance. + */ + public KafkaByteBlockingQueuedProducer createProducer( + AzureEventHubsSourceConfig azureEventHubsSourceConfig, + BlockingQueue> recordBlockingQueue, + Map inputToOutputTopics) { + String connectorName = azureEventHubsSourceConfig.getString(AzureEventHubsConfigConstants.CONNECTOR_NAME); + final String clientId = connectorName + "#" + UUID.randomUUID(); + log.info("Attempting to create Client with Id:{}", clientId); + KeyValueTypes keyValueTypes = KeyValueTypes.DEFAULT_TYPES; + + Map consumerProperties = prepareConsumerProperties(azureEventHubsSourceConfig, + clientId, connectorName, keyValueTypes); + + KafkaConsumer kafkaConsumer = new KafkaConsumer<>(consumerProperties); + + boolean shouldSeekToLatest = shouldConsumerSeekToLatest(azureEventHubsSourceConfig); + Set inputTopics = inputToOutputTopics.keySet(); + + return new KafkaByteBlockingQueuedProducer(topicPartitionOffsetProvider, recordBlockingQueue, + kafkaConsumer, keyValueTypes, clientId, inputTopics, shouldSeekToLatest); + } + + private static Map prepareConsumerProperties( + AzureEventHubsSourceConfig azureEventHubsSourceConfig, String clientId, String connectorName, + KeyValueTypes keyValueTypes) { + Map consumerProperties = azureEventHubsSourceConfig.originalsWithPrefix( + AzureEventHubsConfigConstants.CONNECTOR_WITH_CONSUMER_PREFIX, STRIP_PREFIX); + + consumerProperties.put(ConsumerConfig.CLIENT_ID_CONFIG, clientId); + consumerProperties.put(ConsumerConfig.GROUP_ID_CONFIG, connectorName); + consumerProperties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, + keyValueTypes.getKeyType().getDeserializerClass()); + consumerProperties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, + keyValueTypes.getValueType().getDeserializerClass()); + consumerProperties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false"); + return consumerProperties; + } + + private boolean shouldConsumerSeekToLatest(AzureEventHubsSourceConfig azureEventHubsSourceConfig) { + String seekValue = azureEventHubsSourceConfig.getString(AzureEventHubsConfigConstants.CONSUMER_OFFSET); + if (EARLIEST_OFFSET.equalsIgnoreCase(seekValue)) { + return false; + } else if (LATEST_OFFSET.equalsIgnoreCase(seekValue)) { + return true; + } + throw new ConfigException(AzureEventHubsConfigConstants.CONSUMER_OFFSET, seekValue, + CONSUMER_OFFSET_EXCEPTION_MESSAGE); + } +} diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/EventHubsKafkaConsumerController.java b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/EventHubsKafkaConsumerController.java new file mode 100644 index 000000000..6ff27db9f --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/EventHubsKafkaConsumerController.java @@ -0,0 +1,109 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.source; + +import static io.lenses.streamreactor.connect.azure.eventhubs.mapping.SourceRecordMapper.mapSourceRecordIncludingHeaders; + +import io.lenses.streamreactor.connect.azure.eventhubs.source.TopicPartitionOffsetProvider.AzureOffsetMarker; +import io.lenses.streamreactor.connect.azure.eventhubs.source.TopicPartitionOffsetProvider.AzureTopicPartitionKey; +import java.time.Duration; +import java.time.temporal.ChronoUnit; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.TimeUnit; +import lombok.extern.slf4j.Slf4j; +import org.apache.kafka.clients.consumer.ConsumerRecord; +import org.apache.kafka.clients.consumer.ConsumerRecords; +import org.apache.kafka.connect.source.SourceRecord; + +/** + * Class is a bridge between EventHub KafkaConsumers and AzureEventHubsSourceTask. It verifies the + * configuration of kafka consumers and instantiates them, then allows AzureEventHubsSourceTask to + * pull for SourceRecords. + */ +@Slf4j +public class EventHubsKafkaConsumerController { + + private final BlockingQueue> recordsQueue; + private KafkaByteBlockingQueuedProducer queuedKafkaProducer; + private final Map inputToOutputTopics; + + /** + * Constructs EventHubsKafkaConsumerController. + * + * @param queuedKafkaProducer producer to the recordsQueue + * @param recordsQueue queue that contains EventHub records + * @param inputToOutputTopics input to output topics + */ + public EventHubsKafkaConsumerController(KafkaByteBlockingQueuedProducer queuedKafkaProducer, + BlockingQueue> recordsQueue, + Map inputToOutputTopics) { + this.recordsQueue = recordsQueue; + this.queuedKafkaProducer = queuedKafkaProducer; + this.inputToOutputTopics = inputToOutputTopics; + } + + /** + * This method leverages BlockingQueue mechanism that BlockingQueuedKafkaConsumer puts EventHub + * records into. It tries to poll the queue then returns list of SourceRecords + * + * @param duration how often to poll. + * @return list of SourceRecords (can be empty if it couldn't poll from queue) + * @throws InterruptedException if interrupted while polling + */ + public List poll(Duration duration) throws InterruptedException { + List sourceRecords = null; + + queuedKafkaProducer.start(); + + ConsumerRecords consumerRecords = null; + try { + consumerRecords = recordsQueue.poll( + duration.get(ChronoUnit.SECONDS), TimeUnit.SECONDS); + } catch (InterruptedException e) { + log.info("{} has been interrupted on poll", this.getClass().getSimpleName()); + throw e; + } + + if (consumerRecords != null && !consumerRecords.isEmpty()) { + sourceRecords = new ArrayList<>(consumerRecords.count()); + for (ConsumerRecord consumerRecord : consumerRecords) { + + String inputTopic = consumerRecord.topic(); + AzureTopicPartitionKey azureTopicPartitionKey = new AzureTopicPartitionKey( + inputTopic, consumerRecord.partition()); + AzureOffsetMarker offsetMarker = new AzureOffsetMarker(consumerRecord.offset()); + + SourceRecord sourceRecord = mapSourceRecordIncludingHeaders(consumerRecord, + azureTopicPartitionKey, + offsetMarker, inputToOutputTopics.get(inputTopic), + queuedKafkaProducer.getKeyValueTypes().getKeyType().getSchema(), + queuedKafkaProducer.getKeyValueTypes().getValueType().getSchema()); + + sourceRecords.add(sourceRecord); + + } + } + return sourceRecords != null ? sourceRecords : Collections.emptyList(); + } + + public void close(Duration timeoutDuration) { + queuedKafkaProducer.stop(timeoutDuration); + } +} diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/KafkaByteBlockingQueuedProducer.java b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/KafkaByteBlockingQueuedProducer.java new file mode 100644 index 000000000..712b73e8c --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/KafkaByteBlockingQueuedProducer.java @@ -0,0 +1,123 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.source; + +import io.lenses.streamreactor.connect.azure.eventhubs.config.SourceDataType.KeyValueTypes; +import java.time.Duration; +import java.time.temporal.ChronoUnit; +import java.util.Set; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import lombok.Getter; +import lombok.extern.slf4j.Slf4j; +import org.apache.kafka.clients.consumer.Consumer; +import org.apache.kafka.clients.consumer.ConsumerRecords; + +/** + * Abstraction over Kafka {@link Consumer} class that wraps the Consumer into a thread and allows + * it to output its records into a {@link BlockingQueue} shared with {@link EventHubsKafkaConsumerController}. + */ +@Slf4j +public class KafkaByteBlockingQueuedProducer implements BlockingQueueProducer { + private static final Duration DEFAULT_POLL_DURATION = Duration.of(1, ChronoUnit.SECONDS); + private final TopicPartitionOffsetProvider topicPartitionOffsetProvider; + private final BlockingQueue> recordsQueue; + private final Consumer consumer; + private final String clientId; + private final Set inputTopics; + private EventhubsPollingRunnable pollingRunnable; + private final boolean shouldSeekToLatest; + @Getter + private final KeyValueTypes keyValueTypes; + private final AtomicBoolean initialized = new AtomicBoolean(false); + private final AtomicBoolean running = new AtomicBoolean(false); + + /** + * Class is a proxy that allows access to some methods of Kafka Consumer. It's main purpose is to + * create a thread around the consumer and put consumer record into BlockingQueue. After that it + * starts consumption. + * + * @param topicPartitionOffsetProvider TopicPartitionOffsetProvider for subscription handler + * @param recordsQueue BlockingQueue to put records into + * @param consumer Kafka Consumer + * @param keyValueTypes {@link KeyValueTypes} instance indicating key and value + * types + * @param clientId consumer client id + * @param inputTopics kafka inputTopics to consume from + * @param shouldSeekToLatest informs where should consumer seek when there are no + * offsets committed + */ + public KafkaByteBlockingQueuedProducer(TopicPartitionOffsetProvider topicPartitionOffsetProvider, + BlockingQueue> recordsQueue, Consumer consumer, + KeyValueTypes keyValueTypes, String clientId, Set inputTopics, boolean shouldSeekToLatest) { + this.topicPartitionOffsetProvider = topicPartitionOffsetProvider; + this.recordsQueue = recordsQueue; + this.consumer = consumer; + this.clientId = clientId; + this.inputTopics = inputTopics; + this.shouldSeekToLatest = shouldSeekToLatest; + this.keyValueTypes = keyValueTypes; + + start(); + } + + /** + * Starts the production to the records queue. + */ + public void start() { + if (!initialized.getAndSet(true)) { + pollingRunnable = new EventhubsPollingRunnable(); + + new Thread(pollingRunnable).start(); + initialized.set(true); + } + } + + public void stop(Duration timeoutDuration) { + pollingRunnable.close(timeoutDuration); + running.set(false); + } + + private class EventhubsPollingRunnable implements Runnable { + + @Override + public void run() { + running.set(true); + log.info("Subscribing to topics: {}", String.join(",", inputTopics)); + consumer.subscribe(inputTopics, + new AzureConsumerRebalancerListener(topicPartitionOffsetProvider, consumer, shouldSeekToLatest)); + while (running.get()) { + ConsumerRecords consumerRecords = consumer.poll(DEFAULT_POLL_DURATION); + if (consumerRecords != null && !consumerRecords.isEmpty()) { + try { + boolean offer = false; + while (!offer) { + offer = recordsQueue.offer(consumerRecords, 5, TimeUnit.SECONDS); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + log.info("Kafka Consumer with clientId={} has been interrupted on offering", clientId); + } + } + } + } + + void close(Duration timeoutDuration) { + consumer.close(timeoutDuration); + } + } +} diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/ProducerProvider.java b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/ProducerProvider.java new file mode 100644 index 000000000..6b69640c1 --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/ProducerProvider.java @@ -0,0 +1,31 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.source; + +import io.lenses.streamreactor.connect.azure.eventhubs.config.AzureEventHubsSourceConfig; +import java.util.Map; +import java.util.concurrent.BlockingQueue; +import org.apache.kafka.clients.consumer.ConsumerRecords; + +/** + * Interface of a class to produce BlockingQueueProducers. + */ +public interface ProducerProvider { + + BlockingQueueProducer createProducer(AzureEventHubsSourceConfig azureEventHubsSourceConfig, + BlockingQueue> recordBlockingQueue, + Map inputToOutputTopics); +} diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/TopicPartitionOffsetProvider.java b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/TopicPartitionOffsetProvider.java new file mode 100644 index 000000000..6b798d130 --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/source/TopicPartitionOffsetProvider.java @@ -0,0 +1,91 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.source; + +import java.util.HashMap; +import java.util.Optional; +import lombok.extern.slf4j.Slf4j; +import org.apache.kafka.connect.storage.OffsetStorageReader; + +/** + * This class represents an abstraction over OffsetStorageReader that can be freely called by Azure + * EventHub Kafka Consumers when it was initialized once. It helps consumers to find out which + * offset were already committed inside Kafka Connect. + */ +@Slf4j +public final class TopicPartitionOffsetProvider { + + private static final String OFFSET_KEY = "OFFSET"; + + private final OffsetStorageReader offsetStorageReader; + + + public TopicPartitionOffsetProvider(OffsetStorageReader offsetStorageReader) { + this.offsetStorageReader = offsetStorageReader; + } + + /** + * Checks for committed offsets for topic+partition combo. + * + * @param azureTopicPartitionKey key of topic+partition combo. + * + * @return empty optional if topic+partition combo has not committed any offsets or + * AzureOffsetMarker if combo already did commit some. + */ + public Optional getOffset(AzureTopicPartitionKey azureTopicPartitionKey) { + return Optional.ofNullable(offsetStorageReader.offset(azureTopicPartitionKey)) + .map(offsetMap -> (Long) offsetMap.get(OFFSET_KEY)) + .map(AzureOffsetMarker::new); + } + + /** + * This class represents immutable map that represents topic and partition combo used by + * TopicPartitionOffsetProvider. + */ + public static class AzureTopicPartitionKey extends HashMap { + + private static final String TOPIC_KEY = "TOPIC"; + private static final String PARTITION_KEY = "PARTITION"; + + public AzureTopicPartitionKey(String topic, Integer partition) { + this.put(TOPIC_KEY, topic); + this.put(PARTITION_KEY, partition.toString()); + } + + public String getTopic() { + return get(TOPIC_KEY); + } + + public Integer getPartition() { + return Integer.valueOf(get(PARTITION_KEY)); + } + } + + /** + * This class represents immutable map that represents topic and partition combo offset used by + * Kafka Connect SourceRecords. + */ + public static class AzureOffsetMarker extends HashMap { + + public AzureOffsetMarker(Long offset) { + put(OFFSET_KEY, offset); + } + + public Long getOffsetValue() { + return (Long) get(OFFSET_KEY); + } + } +} diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/util/KcqlConfigTopicMapper.java b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/util/KcqlConfigTopicMapper.java new file mode 100644 index 000000000..fc3f5b627 --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/main/java/io/lenses/streamreactor/connect/azure/eventhubs/util/KcqlConfigTopicMapper.java @@ -0,0 +1,77 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.util; + +import io.lenses.kcql.Kcql; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.regex.Matcher; +import java.util.regex.Pattern; +import org.apache.kafka.common.config.ConfigException; + +/** + * Class that represents methods around KCQL topic handling. + */ +public class KcqlConfigTopicMapper { + + private static final String TOPIC_NAME_REGEX = "^[\\w][\\w\\-\\_\\.]*$"; + private static final Pattern TOPIC_NAME_PATTERN = Pattern.compile(TOPIC_NAME_REGEX); + public static final String TOPIC_NAME_ERROR_MESSAGE = + "%s topic %s, name is not correctly specified (It can contain only letters, numbers and hyphens," + + " underscores and dots and has to start with number or letter"; + + /** + * This method parses KCQL statements and fetches input and output topics checking against + * regex for invalid topic names in input and output. + * @param kcqlString string to parse + * @return map of input to output topic names + */ + public static Map mapInputToOutputsFromConfig(String kcqlString) { + List kcqls = Kcql.parseMultiple(kcqlString); + Map inputToOutputTopics = new HashMap<>(kcqls.size()); + List outputTopics = new ArrayList<>(kcqls.size()); + + for (Kcql kcql : kcqls) { + String inputTopic = kcql.getSource(); + String outputTopic = kcql.getTarget(); + + if (!topicNameMatchesAgainstRegex(inputTopic)) { + throw new ConfigException(String.format(TOPIC_NAME_ERROR_MESSAGE, "Input", inputTopic)); + } + if (!topicNameMatchesAgainstRegex(outputTopic)) { + throw new ConfigException(String.format(TOPIC_NAME_ERROR_MESSAGE, "Output", outputTopic)); + } + if (inputToOutputTopics.containsKey(inputTopic)) { + throw new ConfigException(String.format("Input %s cannot be mapped twice.", inputTopic)); + } + if (outputTopics.contains(outputTopic)) { + throw new ConfigException(String.format("Output %s cannot be mapped twice.", outputTopic)); + } + + inputToOutputTopics.put(inputTopic, outputTopic); + outputTopics.add(outputTopic); + } + + return inputToOutputTopics; + } + + private static boolean topicNameMatchesAgainstRegex(String topicName) { + final Matcher matcher = TOPIC_NAME_PATTERN.matcher(topicName); + return matcher.matches(); + } +} diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/main/resources/META-INF/services/org.apache.kafka.connect.source.SourceConnector b/java-connectors/kafka-connect-azure-eventhubs/src/main/resources/META-INF/services/org.apache.kafka.connect.source.SourceConnector new file mode 100644 index 000000000..5c06b3db4 --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/main/resources/META-INF/services/org.apache.kafka.connect.source.SourceConnector @@ -0,0 +1 @@ +io.lenses.streamreactor.connect.azure.eventhubs.source.AzureEventHubsSourceConnector \ No newline at end of file diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/main/resources/azure-eventhubs-ascii.txt b/java-connectors/kafka-connect-azure-eventhubs/src/main/resources/azure-eventhubs-ascii.txt new file mode 100644 index 000000000..0311112e7 --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/main/resources/azure-eventhubs-ascii.txt @@ -0,0 +1,14 @@ + + ████████▀▀▀▀▀███████████████████████████████████████████████████████████████████ + █████▀ ▀████████████████████████████████████████████████████████████████ + ███▀ ▄█████▄ ▀██████████████████████████████████████████████████████████████ + ███ ▄███████▄ ██████ █████▌ █▌ ████ ███ ▄▄ ██ ███ ▄▄ ███ + ███ █████████ ██████ █████▌ ██████▌ ▀██ ██ ██████ ██████ ███████ + ███ ▀███████▀ ██████ █████▌ ██▌ █▄ █ ███▄▄ ██ ███▄▄ ███ + ████▄ ▄███████ █████▌ ██████▌ ███ ███████ █ ███████████ ██ + █████████ ████████████ ▌ █▌ ████▄ ██▄ ▄██ █▄ ▄███ + █████████ ████████████████████████████████████████████████████████████████████ + █████████ ▄████████████████████████████████████████████████████████████████████ + ████████████████████████████████████████████████████████████████████████████████ + █████████████████████████AZURE EVENT HUBS ██████████████████████████████████████ + ████████████████████████████████████████████████████████████████████████████████ diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/config/SourceDataTypeTest.java b/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/config/SourceDataTypeTest.java new file mode 100644 index 000000000..6f2f0a829 --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/config/SourceDataTypeTest.java @@ -0,0 +1,60 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.config; + +import static org.apache.kafka.connect.data.Schema.OPTIONAL_BYTES_SCHEMA; +import static org.junit.jupiter.api.Assertions.assertEquals; + +import org.apache.kafka.common.serialization.ByteArrayDeserializer; +import org.apache.kafka.common.serialization.Deserializer; +import org.apache.kafka.connect.data.Schema; +import org.junit.jupiter.api.Test; + +class SourceDataTypeTest { + + @Test + void fromNameShouldReturnEnum() { + //given + + //when + SourceDataType s = SourceDataType.fromName(SourceDataType.BYTES.name()); + + //then + assertEquals(SourceDataType.BYTES, s); + } + + @Test + void getDeserializerClassShouldReturnSpecifiedDeserializer() { + //given + + //when + Class deserializerClass = SourceDataType.BYTES.getDeserializerClass(); + + //then + assertEquals(ByteArrayDeserializer.class, deserializerClass); + } + + @Test + void getSchema() { + //given + + //when + Schema schema = SourceDataType.BYTES.getSchema(); + + //then + assertEquals(OPTIONAL_BYTES_SCHEMA, schema); + } +} \ No newline at end of file diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/mapping/SourceRecordMapperTest.java b/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/mapping/SourceRecordMapperTest.java new file mode 100644 index 000000000..c5e1ecb35 --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/mapping/SourceRecordMapperTest.java @@ -0,0 +1,125 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.mapping; + +import static java.util.Collections.singletonList; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.from; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import io.lenses.streamreactor.connect.azure.eventhubs.source.TopicPartitionOffsetProvider.AzureOffsetMarker; +import io.lenses.streamreactor.connect.azure.eventhubs.source.TopicPartitionOffsetProvider.AzureTopicPartitionKey; +import java.util.Iterator; +import java.util.Optional; +import org.apache.kafka.clients.consumer.ConsumerRecord; +import org.apache.kafka.common.header.Header; +import org.apache.kafka.common.header.Headers; +import org.apache.kafka.connect.data.Schema; +import org.apache.kafka.connect.source.SourceRecord; +import org.junit.jupiter.api.Test; + +class SourceRecordMapperTest { + private static final String TOPIC = "topic"; + private static final Integer PARTITION = 10; + private static final Long OFFSET = 111L; + private static final Long TIMESTAMP = 2024L; + private static final String HEADER_KEY = "headerKey"; + private static final String OUTPUT_TOPIC = "OUTPUT"; + + @Test + void shouldMapSourceRecordIncludingHeaders() { + //given + AzureTopicPartitionKey topicPartitionKey = new AzureTopicPartitionKey(TOPIC, PARTITION); + AzureOffsetMarker azureOffsetMarker = new AzureOffsetMarker(OFFSET); + + byte[] exampleHeaderValue = new byte[] {1, 10}; + int headerLength = exampleHeaderValue.length; + Header mockedHeader = mock(Header.class); + when(mockedHeader.key()).thenReturn(HEADER_KEY); + when(mockedHeader.value()).thenReturn(exampleHeaderValue); + + Iterator
iterator = singletonList(mockedHeader).iterator(); + Headers mockedHeaders = mock(Headers.class); + when(mockedHeaders.iterator()).thenReturn(iterator); + + ConsumerRecord consumerRecord = mockConsumerRecord(Optional.of(mockedHeaders)); + + //when + Schema stringSchema = Schema.STRING_SCHEMA; + Schema optionalStringSchema = Schema.OPTIONAL_STRING_SCHEMA; + SourceRecord sourceRecord = SourceRecordMapper.mapSourceRecordIncludingHeaders( + consumerRecord, topicPartitionKey, azureOffsetMarker, + OUTPUT_TOPIC, optionalStringSchema, stringSchema); + + //then + assertRecordAttributesAreMappedFromSourceConsumerRecord(sourceRecord, consumerRecord, + OUTPUT_TOPIC, optionalStringSchema, stringSchema, topicPartitionKey, azureOffsetMarker); + verify(consumerRecord).headers(); + assertThat(sourceRecord.headers()).hasSize(1); + assertThat(((byte[])sourceRecord.headers().lastWithName(HEADER_KEY).value())).hasSize(headerLength); + } + + @Test + void mapSourceRecordWithoutHeaders() { + //given + AzureTopicPartitionKey topicPartitionKey = new AzureTopicPartitionKey(TOPIC, PARTITION); + AzureOffsetMarker azureOffsetMarker = new AzureOffsetMarker(OFFSET); + + ConsumerRecord consumerRecord = mockConsumerRecord(Optional.empty()); + + //when + Schema stringSchema = Schema.STRING_SCHEMA; + Schema optionalStringSchema = Schema.OPTIONAL_STRING_SCHEMA; + SourceRecord sourceRecord = SourceRecordMapper.mapSourceRecordWithoutHeaders( + consumerRecord, topicPartitionKey, azureOffsetMarker, OUTPUT_TOPIC, + optionalStringSchema, stringSchema); + + //then + assertRecordAttributesAreMappedFromSourceConsumerRecord(sourceRecord, consumerRecord, + OUTPUT_TOPIC, optionalStringSchema, stringSchema, topicPartitionKey, azureOffsetMarker); + assertThat(sourceRecord.headers()).isEmpty(); + } + + private static ConsumerRecord mockConsumerRecord(Optional mockedHeaders) { + ConsumerRecord consumerRecord = mock(ConsumerRecord.class); + when(consumerRecord.topic()).thenReturn(TOPIC); + when(consumerRecord.partition()).thenReturn(PARTITION); + when(consumerRecord.timestamp()).thenReturn(TIMESTAMP); + + mockedHeaders.ifPresent(headers -> when(consumerRecord.headers()).thenReturn(headers)); + return consumerRecord; + } + + private void assertRecordAttributesAreMappedFromSourceConsumerRecord(SourceRecord mappedRecord, + ConsumerRecord originalRecord, String outputTopic, Schema keySchema, Schema valueSchema, + AzureTopicPartitionKey sourcePartitionKey, AzureOffsetMarker offsetMarker) { + verify(originalRecord).timestamp(); + verify(originalRecord).key(); + verify(originalRecord).value(); + + assertThat(mappedRecord) + .returns(originalRecord.timestamp(), from(SourceRecord::timestamp)) + .returns(sourcePartitionKey, from(SourceRecord::sourcePartition)) + .returns(null, from(SourceRecord::kafkaPartition)) + .returns(offsetMarker, from(SourceRecord::sourceOffset)) + .returns(originalRecord.value(), from(SourceRecord::value)) + .returns(outputTopic, from(SourceRecord::topic)) + .returns(keySchema, from(SourceRecord::keySchema)) + .returns(valueSchema, from(SourceRecord::valueSchema)); + } +} \ No newline at end of file diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureConsumerRebalancerListenerTest.java b/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureConsumerRebalancerListenerTest.java new file mode 100644 index 000000000..03a0e41e5 --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureConsumerRebalancerListenerTest.java @@ -0,0 +1,109 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.source; + +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyList; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import io.lenses.streamreactor.connect.azure.eventhubs.source.TopicPartitionOffsetProvider.AzureOffsetMarker; +import io.lenses.streamreactor.connect.azure.eventhubs.source.TopicPartitionOffsetProvider.AzureTopicPartitionKey; +import java.util.Collections; +import java.util.Optional; +import org.apache.kafka.clients.consumer.Consumer; +import org.apache.kafka.common.TopicPartition; +import org.junit.jupiter.api.Test; + +class AzureConsumerRebalancerListenerTest { + + private final boolean SEEK_TO_EARLIEST = false; + private final boolean SEEK_TO_LATEST = true; + + @Test + void onPartitionsAssignedShouldSeekToBeginningIfOffsetProviderProvidesEmptyOffsetAndSeekingToEarliest() { + //given + Consumer stringKafkaConsumer = mock(Consumer.class); + TopicPartitionOffsetProvider offsetProvider = mock( + TopicPartitionOffsetProvider.class); + AzureConsumerRebalancerListener testObj = + new AzureConsumerRebalancerListener(offsetProvider, stringKafkaConsumer, SEEK_TO_EARLIEST); + String topic = "topic1"; + Integer partition = 1; + TopicPartition topicPartition1 = mock(TopicPartition.class); + when(topicPartition1.topic()).thenReturn(topic); + when(topicPartition1.partition()).thenReturn(partition); + + //when + testObj.onPartitionsAssigned(Collections.singletonList(topicPartition1)); + + //then + verify(topicPartition1, times(1)).topic(); + verify(topicPartition1, times(1)).partition(); + verify(stringKafkaConsumer).seekToBeginning(anyList()); + } + + @Test + void onPartitionsAssignedShouldSeekToEndIfOffsetProviderProvidesEmptyOffsetAndSeekingToLatest() { + //given + Consumer stringKafkaConsumer = mock(Consumer.class); + TopicPartitionOffsetProvider offsetProvider = mock( + TopicPartitionOffsetProvider.class); + AzureConsumerRebalancerListener testObj = + new AzureConsumerRebalancerListener(offsetProvider, stringKafkaConsumer, SEEK_TO_LATEST); + String topic = "topic1"; + Integer partition = 1; + TopicPartition topicPartition1 = mock(TopicPartition.class); + when(topicPartition1.topic()).thenReturn(topic); + when(topicPartition1.partition()).thenReturn(partition); + + //when + testObj.onPartitionsAssigned(Collections.singletonList(topicPartition1)); + + //then + verify(topicPartition1, times(1)).topic(); + verify(topicPartition1, times(1)).partition(); + verify(stringKafkaConsumer).seekToEnd(anyList()); + } + + @Test + void onPartitionsAssignedShouldSeekToSpecificOffsetIfOffsetProviderProvidesIt() { + //given + Long specificOffset = 100L; + Consumer stringKafkaConsumer = mock(Consumer.class); + TopicPartitionOffsetProvider offsetProvider = mock( + TopicPartitionOffsetProvider.class); + when(offsetProvider.getOffset(any(AzureTopicPartitionKey.class))) + .thenReturn(Optional.of(new AzureOffsetMarker(specificOffset))); + AzureConsumerRebalancerListener testObj = + new AzureConsumerRebalancerListener(offsetProvider, stringKafkaConsumer, SEEK_TO_EARLIEST); + String topic = "topic1"; + Integer partition = 1; + TopicPartition topicPartition1 = mock(TopicPartition.class); + when(topicPartition1.topic()).thenReturn(topic); + when(topicPartition1.partition()).thenReturn(partition); + + //when + testObj.onPartitionsAssigned(Collections.singletonList(topicPartition1)); + + //then + verify(topicPartition1, times(1)).topic(); + verify(topicPartition1, times(1)).partition(); + verify(stringKafkaConsumer).seek(topicPartition1, specificOffset); + } +} \ No newline at end of file diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureEventHubsSourceConnectorTest.java b/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureEventHubsSourceConnectorTest.java new file mode 100644 index 000000000..79c6957b4 --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureEventHubsSourceConnectorTest.java @@ -0,0 +1,87 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.source; + +import static org.apache.kafka.connect.source.ExactlyOnceSupport.SUPPORTED; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import io.lenses.streamreactor.common.util.JarManifest; +import io.lenses.streamreactor.connect.azure.eventhubs.config.AzureEventHubsConfigConstants; +import java.lang.reflect.Field; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import org.apache.kafka.connect.source.ExactlyOnceSupport; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.mockito.MockedConstruction; +import org.mockito.Mockito; + +class AzureEventHubsSourceConnectorTest { + + private static final String CONNECTOR_NAME = "AzureEventHubsSource"; + private static final String KCQL = "INSERT INTO OUTPUT1 SELECT * FROM INPUT1;"; + private AzureEventHubsSourceConnector testObj; + + @BeforeEach + void setUp() { + try (MockedConstruction ignored = Mockito.mockConstruction(JarManifest.class)) { + testObj = new AzureEventHubsSourceConnector(); + } + } + + @Test + void taskConfigsShouldMultiplyConfigs() throws NoSuchFieldException, IllegalAccessException { + //given + Map simpleProperties = createSimplePropertiesWithKcql(); + int maxTasks = 3; + + //when + Field configPropertiesField = testObj.getClass() + .getDeclaredField("configProperties"); + configPropertiesField.setAccessible(true); + configPropertiesField.set(testObj, simpleProperties); + List> taskConfigs = testObj.taskConfigs(maxTasks); + + //then + for (Map taskConfig : taskConfigs){ + assertTrue(taskConfig.equals(simpleProperties)); + } + + } + + @Test + void exactlyOnceSupportShouldReturnSupported() { + //given + + //when + ExactlyOnceSupport exactlyOnceSupport = testObj.exactlyOnceSupport(new HashMap<>()); + + //then + assertEquals(SUPPORTED, exactlyOnceSupport); + } + + private Map createSimplePropertiesWithKcql() { + Map properties = Map.of( + AzureEventHubsConfigConstants.CONNECTOR_NAME, CONNECTOR_NAME, + "connector.class", AzureEventHubsSourceConnector.class.getCanonicalName(), + AzureEventHubsConfigConstants.KCQL_CONFIG, KCQL + ); + + return properties; + } +} \ No newline at end of file diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureEventHubsSourceTaskTest.java b/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureEventHubsSourceTaskTest.java new file mode 100644 index 000000000..da709bfa8 --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/AzureEventHubsSourceTaskTest.java @@ -0,0 +1,144 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.source; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertIterableEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.atMostOnce; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import ch.qos.logback.classic.spi.ILoggingEvent; +import ch.qos.logback.core.read.ListAppender; +import io.lenses.streamreactor.common.util.JarManifest; +import io.lenses.streamreactor.connect.azure.eventhubs.config.AzureEventHubsConfigConstants; +import io.lenses.streamreactor.connect.azure.eventhubs.config.AzureEventHubsSourceConfig; +import java.time.Duration; +import java.util.Collections; +import java.util.List; +import org.apache.kafka.connect.source.SourceRecord; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.slf4j.LoggerFactory; + +class AzureEventHubsSourceTaskTest { + + private AzureEventHubsSourceTask testObj; + private ListAppender logWatcher; + private JarManifest mockedJarManifest = mock(JarManifest.class); + + @BeforeEach + void setup() { + mockedJarManifest = mock(JarManifest.class); + testObj = new AzureEventHubsSourceTask(mockedJarManifest); + logWatcher = new ListAppender<>(); + logWatcher.start(); + ((ch.qos.logback.classic.Logger) LoggerFactory.getLogger(AzureEventHubsSourceTask.class)).addAppender(logWatcher); + } + + @AfterEach + void teardown() { + ((ch.qos.logback.classic.Logger) LoggerFactory.getLogger(AzureEventHubsSourceTask.class)).detachAndStopAllAppenders(); + } + + @Test + void stopShouldCallStopOnController() { + //given + Duration thirtySeconds = Duration.ofSeconds(30); + EventHubsKafkaConsumerController mockedController = mock(EventHubsKafkaConsumerController.class); + AzureEventHubsSourceConfig azureEventHubsSourceConfig = mock(AzureEventHubsSourceConfig.class); + when(azureEventHubsSourceConfig.getInt(AzureEventHubsConfigConstants.CONSUMER_CLOSE_TIMEOUT)) + .thenReturn(thirtySeconds.toSecondsPart()); + testObj.initialize(mockedController, azureEventHubsSourceConfig); + + //when + testObj.stop(); + + //then + verify(mockedController, times(1)).close(thirtySeconds); + } + + @Test + void initializeShouldLog() { + //given + EventHubsKafkaConsumerController mockedController = mock(EventHubsKafkaConsumerController.class); + AzureEventHubsSourceConfig azureEventHubsSourceConfig = mock(AzureEventHubsSourceConfig.class); + testObj.initialize(mockedController, azureEventHubsSourceConfig); + + //when + testObj.stop(); + + //then + assertEquals(1, logWatcher.list.size()); + assertEquals("AzureEventHubsSourceTask initialised.", logWatcher.list.get(0).getFormattedMessage()); + } + + @Test + void pollShouldCallPollOnControllerAndReturnNullIfListIsEmpty() throws InterruptedException { + //given + AzureEventHubsSourceConfig azureEventHubsSourceConfig = mock(AzureEventHubsSourceConfig.class); + EventHubsKafkaConsumerController mockedController = mock(EventHubsKafkaConsumerController.class); + testObj.initialize(mockedController, azureEventHubsSourceConfig); + when(mockedController.poll(any(Duration.class))).thenReturn(Collections.emptyList()); + + //when + List poll = testObj.poll(); + + //then + assertNull(poll); + } + + @Test + void pollShouldCallPollOnControllerAndReturnListThatHasElements() throws InterruptedException { + //given + AzureEventHubsSourceConfig azureEventHubsSourceConfig = mock(AzureEventHubsSourceConfig.class); + EventHubsKafkaConsumerController mockedController = mock(EventHubsKafkaConsumerController.class); + testObj.initialize(mockedController, azureEventHubsSourceConfig); + SourceRecord mockedRecord = mock(SourceRecord.class); + List sourceRecords = Collections.singletonList(mockedRecord); + when(mockedController.poll(any(Duration.class))).thenReturn(sourceRecords); + + //when + List poll = testObj.poll(); + + //then + assertNotNull(poll); + assertIterableEquals(sourceRecords, poll); + } + + @Test + void getVersionShouldDelegateToJarManifestGetVersion() { + //given + AzureEventHubsSourceConfig azureEventHubsSourceConfig = mock(AzureEventHubsSourceConfig.class); + EventHubsKafkaConsumerController mockedController = mock(EventHubsKafkaConsumerController.class); + testObj.initialize(mockedController, azureEventHubsSourceConfig); + final String SOME_VERSION = "SOME_VERSION"; + when(mockedJarManifest.getVersion()).thenReturn(SOME_VERSION); + + //when + String version = testObj.version(); + + //then + assertEquals(SOME_VERSION, version); + verify(mockedJarManifest, atMostOnce()).getVersion(); + } +} \ No newline at end of file diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/BlockingQueueProducerProviderTest.java b/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/BlockingQueueProducerProviderTest.java new file mode 100644 index 000000000..3fe1007b7 --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/BlockingQueueProducerProviderTest.java @@ -0,0 +1,108 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.source; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import ch.qos.logback.classic.spi.ILoggingEvent; +import ch.qos.logback.core.read.ListAppender; +import io.lenses.streamreactor.connect.azure.eventhubs.config.AzureEventHubsConfigConstants; +import io.lenses.streamreactor.connect.azure.eventhubs.config.AzureEventHubsSourceConfig; +import java.util.HashMap; +import java.util.concurrent.ArrayBlockingQueue; +import org.apache.kafka.clients.consumer.KafkaConsumer; +import org.apache.kafka.common.config.ConfigException; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.mockito.MockedConstruction; +import org.mockito.Mockito; +import org.slf4j.LoggerFactory; + +class BlockingQueueProducerProviderTest { + + private ListAppender logWatcher; + + @BeforeEach + void setup() { + logWatcher = new ListAppender<>(); + logWatcher.start(); + ((ch.qos.logback.classic.Logger) LoggerFactory.getLogger(BlockingQueueProducerProvider.class)).addAppender(logWatcher); + } + + @AfterEach + void teardown() { + ((ch.qos.logback.classic.Logger) LoggerFactory.getLogger(BlockingQueueProducerProvider.class)).detachAndStopAllAppenders(); + } + + @Test + void whenConstructorInvokedWithoutOffsetParameterThenConfigExceptionIsThrown(){ + //given + AzureEventHubsSourceConfig azureConfigMock = mock(AzureEventHubsSourceConfig.class); + TopicPartitionOffsetProvider mockedOffsetProvider = mock(TopicPartitionOffsetProvider.class); + + + //when + BlockingQueueProducerProvider testObj = new BlockingQueueProducerProvider( + mockedOffsetProvider); + ConfigException configException; + try(MockedConstruction ignored = Mockito.mockConstruction(KafkaConsumer.class)){ + configException = assertThrows(ConfigException.class, () -> { + testObj.createProducer(azureConfigMock, new ArrayBlockingQueue<>(1), + new HashMap<>()); + }); + } + + + //then + assertEquals("Invalid value null for configuration connect.eventhubs.source.default.offset: " + + "allowed values are: earliest/latest", configException.getMessage()); + } + + @Test + void whenConstructorInvokedWithParametersThenMockKafkaConsumerShouldBeCreatedAndLogged(){ + //given + String earliestOffset = "earliest"; + TopicPartitionOffsetProvider mockedOffsetProvider = mock(TopicPartitionOffsetProvider.class); + + AzureEventHubsSourceConfig azureConfigMock = mock(AzureEventHubsSourceConfig.class); + when(azureConfigMock.getString(AzureEventHubsConfigConstants.CONSUMER_OFFSET)).thenReturn( + earliestOffset); + when(azureConfigMock.getString(AzureEventHubsConfigConstants.KCQL_CONFIG)) + .thenReturn("insert into output select * from input"); + + //when + BlockingQueueProducerProvider testObj = new BlockingQueueProducerProvider( + mockedOffsetProvider); + KafkaByteBlockingQueuedProducer consumer; + try(MockedConstruction ignored = Mockito.mockConstruction(KafkaConsumer.class)){ + consumer = testObj.createProducer(azureConfigMock, new ArrayBlockingQueue<>(1), + new HashMap<>()); + } + + //then + verify(azureConfigMock).getString(AzureEventHubsConfigConstants.CONNECTOR_NAME); + assertNotNull(consumer); + assertEquals(1, logWatcher.list.size()); + assertTrue(logWatcher.list.get(0).getFormattedMessage().startsWith("Attempting to create Client with Id")); + } +} \ No newline at end of file diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/EventHubsKafkaConsumerControllerTest.java b/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/EventHubsKafkaConsumerControllerTest.java new file mode 100644 index 000000000..da82a2f29 --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/EventHubsKafkaConsumerControllerTest.java @@ -0,0 +1,183 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.source; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import io.lenses.streamreactor.connect.azure.eventhubs.config.SourceDataType; +import io.lenses.streamreactor.connect.azure.eventhubs.config.SourceDataType.KeyValueTypes; +import java.time.Duration; +import java.time.temporal.ChronoUnit; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.concurrent.ArrayBlockingQueue; +import java.util.stream.Collectors; +import org.apache.kafka.clients.consumer.ConsumerRecord; +import org.apache.kafka.clients.consumer.ConsumerRecords; +import org.apache.kafka.common.header.Headers; +import org.apache.kafka.connect.data.Schema; +import org.apache.kafka.connect.source.SourceRecord; +import org.junit.jupiter.api.Test; + +class EventHubsKafkaConsumerControllerTest { + + private static final String INPUT_TOPIC = "INPUT"; + private static final String OUTPUT_TOPIC = "OUTPUT"; + private static final String INPUT_TOPIC_2 = "INPUT2"; + private static final String OUTPUT_TOPIC_2 = "OUTPUT2"; + private static final int DEFAULT_CAPACITY = 10; + private static final Duration DURATION_2_SECONDS = Duration.of(2, ChronoUnit.SECONDS); + + private EventHubsKafkaConsumerController testObj; + + @Test + void pollShouldPollQueueAndReturnSourceRecords() throws InterruptedException { + //given + Map inputOutputMap = Map.of(INPUT_TOPIC, OUTPUT_TOPIC); + + SourceDataType mockedKeyDataType = mockSourceDataType(); + SourceDataType mockedValueDataType = mockSourceDataType(); + + KeyValueTypes mockedKeyValueTypes = mockKeyValueTypes(mockedKeyDataType, mockedValueDataType); + + KafkaByteBlockingQueuedProducer mockedBlockingProducer = mockByteBlockingProducer(mockedKeyValueTypes); + + ArrayBlockingQueue> recordsQueue = mockRecordsQueue(INPUT_TOPIC); + + //when + testObj = new EventHubsKafkaConsumerController(mockedBlockingProducer, recordsQueue, + inputOutputMap); + List sourceRecords = testObj.poll(DURATION_2_SECONDS); + + //then + verify(mockedBlockingProducer).start(); + verify(mockedKeyDataType, times(1)).getSchema(); + verify(mockedValueDataType, times(1)).getSchema(); + verify(mockedBlockingProducer, times(2)).getKeyValueTypes(); + assertEquals(1, sourceRecords.size()); + } + + @Test + void pollWithMultipleInputsAndOutputsShouldPollQueueAndReturnSourceRecordsToCorrectOutput() + throws InterruptedException { + //given + Map inputOutputMap = Map.of(INPUT_TOPIC, OUTPUT_TOPIC, INPUT_TOPIC_2, + OUTPUT_TOPIC_2); + + SourceDataType mockedKeyDataType = mockSourceDataType(); + SourceDataType mockedValueDataType = mockSourceDataType(); + + KeyValueTypes mockedKeyValueTypes = mockKeyValueTypes(mockedKeyDataType, mockedValueDataType); + + KafkaByteBlockingQueuedProducer mockedBlockingProducer = mockByteBlockingProducer(mockedKeyValueTypes); + + ArrayBlockingQueue> recordsQueue = mockRecordsQueue(INPUT_TOPIC, INPUT_TOPIC_2); + + //when + testObj = new EventHubsKafkaConsumerController(mockedBlockingProducer, recordsQueue, + inputOutputMap); + List sourceRecords = testObj.poll(DURATION_2_SECONDS); + + //then + verify(mockedBlockingProducer).start(); + verify(mockedKeyDataType, times(2)).getSchema(); //1x both records + verify(mockedValueDataType, times(2)).getSchema(); //1x both records + verify(mockedBlockingProducer, times(4)).getKeyValueTypes(); + assertEquals(2, sourceRecords.size()); + assertEquals(OUTPUT_TOPIC, sourceRecords.get(0).topic()); + assertEquals(OUTPUT_TOPIC_2, sourceRecords.get(1).topic()); + } + + @Test + void closeShouldCloseTheProducer() { + //given + KafkaByteBlockingQueuedProducer mockedBlockingProducer = mock( + KafkaByteBlockingQueuedProducer.class); + + Map inputOutputMap = Map.of(INPUT_TOPIC, OUTPUT_TOPIC); + + ArrayBlockingQueue> recordsQueue = mockRecordsQueue(); + + testObj = new EventHubsKafkaConsumerController(mockedBlockingProducer, recordsQueue, + inputOutputMap); + + //when + testObj.close(DURATION_2_SECONDS); + + //then + verify(mockedBlockingProducer).stop(DURATION_2_SECONDS); + } + + private static SourceDataType mockSourceDataType() { + SourceDataType mockedDataType = mock(SourceDataType.class); + when(mockedDataType.getSchema()).thenReturn(Schema.OPTIONAL_STRING_SCHEMA); + return mockedDataType; + } + + private static KeyValueTypes mockKeyValueTypes(SourceDataType keyDataType, SourceDataType valueDataType) { + KeyValueTypes mockedKeyValueTypes = mock(KeyValueTypes.class); + when(mockedKeyValueTypes.getKeyType()).thenReturn(keyDataType); + when(mockedKeyValueTypes.getValueType()).thenReturn(valueDataType); + return mockedKeyValueTypes; + } + + private static ConsumerRecord mockConsumerRecord( + String inputTopic, Optional headersMock) { + ConsumerRecord consumerRecord = mock(ConsumerRecord.class); + headersMock.ifPresent(headers -> when(consumerRecord.headers()).thenReturn(headers)); + when(consumerRecord.topic()).thenReturn(inputTopic); + return consumerRecord; + } + + private static Headers mockEmptyHeaders() { + Headers headersMock = mock(Headers.class); + when(headersMock.iterator()).thenReturn(Collections.emptyIterator()); + return headersMock; + } + + + private static KafkaByteBlockingQueuedProducer mockByteBlockingProducer(KeyValueTypes mockedKeyValueTypes) { + KafkaByteBlockingQueuedProducer mockedBlockingProducer = mock( + KafkaByteBlockingQueuedProducer.class); + when(mockedBlockingProducer.getKeyValueTypes()).thenReturn(mockedKeyValueTypes); + return mockedBlockingProducer; + } + + private static ArrayBlockingQueue> mockRecordsQueue + (String... inputTopics) { + + Headers headersMock = mockEmptyHeaders(); + + List> consumerRecordList = Arrays.stream(inputTopics) + .map(it -> mockConsumerRecord(it, Optional.of(headersMock))) + .collect(Collectors.toList()); + + ConsumerRecords mockedRecords = mock(ConsumerRecords.class); + when(mockedRecords.count()).thenReturn(consumerRecordList.size()); + when(mockedRecords.iterator()).thenReturn(consumerRecordList.iterator()); + + return new ArrayBlockingQueue<>(DEFAULT_CAPACITY, false, List.of(mockedRecords)); + } + + +} \ No newline at end of file diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/KafkaByteBlockingQueuedProducerTest.java b/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/KafkaByteBlockingQueuedProducerTest.java new file mode 100644 index 000000000..844803dee --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/KafkaByteBlockingQueuedProducerTest.java @@ -0,0 +1,51 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.source; + +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; + +import io.lenses.streamreactor.connect.azure.eventhubs.config.SourceDataType.KeyValueTypes; +import java.time.Duration; +import java.time.temporal.ChronoUnit; +import java.util.concurrent.BlockingQueue; +import org.apache.kafka.clients.consumer.Consumer; +import org.junit.jupiter.api.Test; +import org.mockito.internal.util.collections.Sets; + +class KafkaByteBlockingQueuedProducerTest { + + private static final String CLIENT_ID = "clientId"; + private static Consumer consumer = mock(Consumer.class); + + KafkaByteBlockingQueuedProducer testObj = new KafkaByteBlockingQueuedProducer( + mock(TopicPartitionOffsetProvider.class), mock(BlockingQueue.class), + consumer, KeyValueTypes.DEFAULT_TYPES, + CLIENT_ID, Sets.newSet("topic"), false); + + @Test + void closeShouldBeDelegatedToKafkaConsumer() { + //given + Duration tenSeconds = Duration.of(10, ChronoUnit.SECONDS); + + //when + testObj.stop(tenSeconds); + + //then + verify(consumer).close(eq(tenSeconds)); + } +} \ No newline at end of file diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/TopicPartitionOffsetProviderTest.java b/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/TopicPartitionOffsetProviderTest.java new file mode 100644 index 000000000..33490cbc3 --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/source/TopicPartitionOffsetProviderTest.java @@ -0,0 +1,109 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.source; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import io.lenses.streamreactor.connect.azure.eventhubs.source.TopicPartitionOffsetProvider.AzureOffsetMarker; +import io.lenses.streamreactor.connect.azure.eventhubs.source.TopicPartitionOffsetProvider.AzureTopicPartitionKey; +import java.util.HashMap; +import java.util.Map; +import java.util.Optional; +import org.apache.kafka.connect.storage.OffsetStorageReader; +import org.junit.jupiter.api.Test; + +class TopicPartitionOffsetProviderTest { + + @Test + void getOffsetShouldCallOffsetStorageReader() { + //given + OffsetStorageReader offsetStorageReader = mock(OffsetStorageReader.class); + TopicPartitionOffsetProvider topicPartitionOffsetProvider = new TopicPartitionOffsetProvider( + offsetStorageReader); + String topic = "some_topic"; + Integer partition = 1; + + //when + AzureTopicPartitionKey azureTopicPartitionKey = new AzureTopicPartitionKey(topic, partition); + topicPartitionOffsetProvider.getOffset(azureTopicPartitionKey); + + //then + verify(offsetStorageReader).offset(azureTopicPartitionKey); + } + + @Test + void getOffsetShouldReturnEmptyOptionalIfCommitsNotFound() { + //given + OffsetStorageReader offsetStorageReader = mock(OffsetStorageReader.class); + when(offsetStorageReader.offset(any(Map.class))).thenReturn(new HashMap()); + String topic = "some_topic"; + Integer partition = 1; + TopicPartitionOffsetProvider topicPartitionOffsetProvider = new TopicPartitionOffsetProvider( + offsetStorageReader); + + //when + AzureTopicPartitionKey azureTopicPartitionKey = new AzureTopicPartitionKey(topic, partition); + Optional offset = topicPartitionOffsetProvider.getOffset(azureTopicPartitionKey); + + //then + verify(offsetStorageReader).offset(azureTopicPartitionKey); + assertTrue(offset.isEmpty()); + } + + @Test + void getOffsetShouldReturnValidAzureOffsetMarkerIfCommitsFound() { + //given + long offsetOne = 1L; + String OFFSET_KEY = "OFFSET"; + OffsetStorageReader offsetStorageReader = mock(OffsetStorageReader.class); + HashMap offsets = new HashMap<>(); + offsets.put(OFFSET_KEY, offsetOne); + when(offsetStorageReader.offset(any(Map.class))).thenReturn(offsets); + String topic = "some_topic"; + Integer partition = 1; + TopicPartitionOffsetProvider topicPartitionOffsetProvider = new TopicPartitionOffsetProvider( + offsetStorageReader); + + //when + AzureTopicPartitionKey azureTopicPartitionKey = new AzureTopicPartitionKey(topic, partition); + Optional offset = topicPartitionOffsetProvider.getOffset(azureTopicPartitionKey); + + //then + verify(offsetStorageReader).offset(azureTopicPartitionKey); + assertTrue(offset.isPresent()); + assertEquals(offsetOne, offset.get().getOffsetValue()); + } + + @Test + void azureTopicPartitionKeyShouldReturnTopicAndPartitionValues() { + //given + int partition = 10; + String topic = "topic"; + + //when + final AzureTopicPartitionKey azureTopicPartitionKey = new AzureTopicPartitionKey(topic, partition); + + //then + assertEquals(partition, azureTopicPartitionKey.getPartition()); + assertEquals(topic, azureTopicPartitionKey.getTopic()); + } + +} \ No newline at end of file diff --git a/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/util/KcqlConfigTopicMapperTest.java b/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/util/KcqlConfigTopicMapperTest.java new file mode 100644 index 000000000..047f940f2 --- /dev/null +++ b/java-connectors/kafka-connect-azure-eventhubs/src/test/java/io/lenses/streamreactor/connect/azure/eventhubs/util/KcqlConfigTopicMapperTest.java @@ -0,0 +1,108 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.connect.azure.eventhubs.util; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import org.apache.kafka.common.config.ConfigException; +import org.junit.jupiter.api.Test; + +class KcqlConfigTopicMapperTest { + + @Test + void mapInputToOutputsFromConfigForMultipleKcqlStatementsShouldRetunMapOfInputToOutput() { + //given + int numberOfMappings = 3; + List inputs = new ArrayList<>(numberOfMappings); + List outputs = new ArrayList<>(numberOfMappings); + String kcqlTemplate = "insert into %s select * from %s;"; + StringBuilder fullKcql = new StringBuilder(); + + for (int i = 0; i < numberOfMappings; i++) { + String newInput = "INPUT" + i; + String newOutput = "OUTPUT" + i; + + inputs.add(i, newInput); + outputs.add(i, newOutput); + fullKcql.append(String.format(kcqlTemplate, newOutput, newInput)); + } + //when + Map inputToOutputsFromConfig = KcqlConfigTopicMapper.mapInputToOutputsFromConfig( + fullKcql.toString()); + + //then + for (String input : inputToOutputsFromConfig.keySet()){ + int indexOfInput = inputs.indexOf(input); + assertNotEquals(-1, indexOfInput); + assertEquals(inputs.get(indexOfInput), input); + assertEquals(outputs.get(indexOfInput), inputToOutputsFromConfig.get(input)); + } + } + + @Test + void mapInputToOutputsFromConfigShouldntAllowForIllegalNames() { + //given + String illegalInputKcql = "INSERT INTO OUTPUT SELECT * FROM 'INPUT*_'"; + String illegalOutputKcql = "INSERT INTO 'OUTPUT*_' SELECT * FROM INPUT"; + String inputErrorMessage = "Input topic INPUT*_, name is not correctly specified " + + "(It can contain only letters, numbers and hyphens, underscores and " + + "dots and has to start with number or letter"; + String outputErrorMessage = "Output topic OUTPUT*_, name is not correctly specified " + + "(It can contain only letters, numbers and hyphens, underscores and " + + "dots and has to start with number or letter"; + + //when + mapInputToOutputAddertingExceptionWithSpecificMessage(illegalInputKcql, inputErrorMessage); + + mapInputToOutputAddertingExceptionWithSpecificMessage(illegalOutputKcql, outputErrorMessage); + } + + @Test + void mapInputToOutputsFromConfigShouldntAllowForOneToManyMappings() { + //given + String oneInputKcql = "INSERT INTO OUTPUT1 SELECT * FROM INPUT1;"; + String sameInputKcql = "INSERT INTO OUTPUT2 SELECT * FROM INPUT1;"; + String outputErrorMessage = "Input INPUT1 cannot be mapped twice."; + + //when + mapInputToOutputAddertingExceptionWithSpecificMessage(oneInputKcql + sameInputKcql, + outputErrorMessage); + } + + @Test + void mapInputToOutputsFromConfigShouldntAllowForMiltipleInputsToSameOutput() { + //given + String oneInputKcql = "INSERT INTO OUTPUT1 SELECT * FROM INPUT1;"; + String anotherInputToSameOutputKcql = "INSERT INTO OUTPUT1 SELECT * FROM INPUT2;"; + String outputErrorMessage = "Output OUTPUT1 cannot be mapped twice."; + + //when + mapInputToOutputAddertingExceptionWithSpecificMessage( + oneInputKcql + anotherInputToSameOutputKcql, + outputErrorMessage); + } + + private static void mapInputToOutputAddertingExceptionWithSpecificMessage(String illegalKcql, + String expectedMessage) { + assertThrows(ConfigException.class, + () -> KcqlConfigTopicMapper.mapInputToOutputsFromConfig(illegalKcql), expectedMessage); + } +} \ No newline at end of file diff --git a/java-connectors/kafka-connect-common/build.gradle b/java-connectors/kafka-connect-common/build.gradle new file mode 100644 index 000000000..d4f372950 --- /dev/null +++ b/java-connectors/kafka-connect-common/build.gradle @@ -0,0 +1,17 @@ +project(":kafka-connect-common") { + test { + maxParallelForks = 1 + } + + dependencies { + //apache kafka + api group: 'org.apache.kafka', name: 'connect-json', version: kafkaVersion + api group: 'org.apache.kafka', name: 'kafka-clients', version: kafkaVersion + + //confluent - may be needed soon +// implementation group: 'io.confluent', name: 'kafka-json-schema-serializer', version: apacheToConfluentVersionAxis.get(kafkaVersion) +// implementation group: 'io.confluent', name: 'kafka-connect-avro-converter', version: apacheToConfluentVersionAxis.get(kafkaVersion) +// implementation group: 'io.confluent', name: 'kafka-connect-avro-data', version: apacheToConfluentVersionAxis.get(kafkaVersion) +// implementation group: 'io.confluent', name: 'kafka-connect-protobuf-converter', version: apacheToConfluentVersionAxis.get(kafkaVersion) + } +} \ No newline at end of file diff --git a/java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/config/base/BaseConfig.java b/java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/config/base/BaseConfig.java new file mode 100644 index 000000000..6259e23d5 --- /dev/null +++ b/java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/config/base/BaseConfig.java @@ -0,0 +1,33 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.common.config.base; + +import java.util.Map; +import org.apache.kafka.common.config.AbstractConfig; +import org.apache.kafka.common.config.ConfigDef; + +/** + * Class represents base implementation of {@link AbstractConfig}. + */ +public abstract class BaseConfig extends AbstractConfig { + + protected final String connectorPrefix; + + public BaseConfig(String connectorPrefix, ConfigDef definition, Map properties) { + super(definition, properties); + this.connectorPrefix = connectorPrefix; + } +} diff --git a/java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/config/base/intf/ConnectorPrefixed.java b/java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/config/base/intf/ConnectorPrefixed.java new file mode 100644 index 000000000..69d977014 --- /dev/null +++ b/java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/config/base/intf/ConnectorPrefixed.java @@ -0,0 +1,23 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.common.config.base.intf; + +/** + * Represents classes that has a Connector prefix (mostly Configurations). + */ +public interface ConnectorPrefixed { + String connectorPrefix(); +} diff --git a/java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/exception/ConnectorStartupException.java b/java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/exception/ConnectorStartupException.java new file mode 100644 index 000000000..01488d01e --- /dev/null +++ b/java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/exception/ConnectorStartupException.java @@ -0,0 +1,26 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.common.exception; + +/** + * Exception to indicate there's something wrong during Connector's startup. + */ +public class ConnectorStartupException extends RuntimeException { + + public ConnectorStartupException(Throwable cause) { + super(cause); + } +} diff --git a/java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/exception/InputStreamExtractionException.java b/java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/exception/InputStreamExtractionException.java new file mode 100644 index 000000000..a3cf1ee4c --- /dev/null +++ b/java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/exception/InputStreamExtractionException.java @@ -0,0 +1,26 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.common.exception; + +/** + * Exception to indicate there's something wrong when reading from InputStream. + */ +public class InputStreamExtractionException extends RuntimeException { + + public InputStreamExtractionException(Throwable cause) { + super(cause); + } +} diff --git a/java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/util/AsciiArtPrinter.java b/java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/util/AsciiArtPrinter.java new file mode 100644 index 000000000..854da5f20 --- /dev/null +++ b/java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/util/AsciiArtPrinter.java @@ -0,0 +1,49 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.common.util; + +import static java.util.Optional.ofNullable; + +import io.lenses.streamreactor.common.exception.InputStreamExtractionException; +import java.io.InputStream; +import java.util.Optional; +import lombok.extern.slf4j.Slf4j; + +/** + * Class used to print Lenses ASCII art. + */ +@Slf4j +public class AsciiArtPrinter { + + /** + * Method fetches ASCII art and logs it. If it cannot display ASCII art then it logs a + * warning with cause. + * + * @param jarManifest JarManifest of Connector + * @param asciiArtResource URI to ASCII art + */ + public static void printAsciiHeader(JarManifest jarManifest, String asciiArtResource) { + try { + Optional asciiArtStream = ofNullable( + AsciiArtPrinter.class.getResourceAsStream(asciiArtResource)); + asciiArtStream.ifPresent(inputStream -> log.info(InputStreamHandler.extractString(inputStream))); + } catch (InputStreamExtractionException exception) { + log.warn("Unable display ASCIIArt from input stream.", exception); + } + log.info(jarManifest.buildManifestString()); + } + +} diff --git a/java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/util/InputStreamHandler.java b/java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/util/InputStreamHandler.java new file mode 100644 index 000000000..453fed129 --- /dev/null +++ b/java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/util/InputStreamHandler.java @@ -0,0 +1,54 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.common.util; + +import io.lenses.streamreactor.common.exception.InputStreamExtractionException; +import java.io.IOException; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.io.Reader; +import java.nio.charset.StandardCharsets; +import lombok.extern.slf4j.Slf4j; + +/** + * Helper class to allow easy manipulation of InputStreams. + */ +@Slf4j +public class InputStreamHandler { + + /** + * Extracts String from InputStream byte buffer. Method is using char type buffer of + * length of 1024 which should make it performant but not take too much space by JVM. + * + * @param inputStream {@link InputStream to read} + * @return String representation of inputStream + * @throws InputStreamExtractionException with cause + */ + public static String extractString(InputStream inputStream) { + int bufferSize = 1024; + char[] buffer = new char[bufferSize]; + StringBuilder out = new StringBuilder(); + try (Reader in = new InputStreamReader(inputStream, StandardCharsets.UTF_8)) { + for (int numRead; (numRead = in.read(buffer, 0, buffer.length)) > 0; ) { + out.append(buffer, 0, numRead); + } + } catch (IOException ioException) { + throw new InputStreamExtractionException(ioException); + } + return out.toString(); + } + +} diff --git a/java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/util/JarManifest.java b/java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/util/JarManifest.java new file mode 100644 index 000000000..0306b5389 --- /dev/null +++ b/java-connectors/kafka-connect-common/src/main/java/io/lenses/streamreactor/common/util/JarManifest.java @@ -0,0 +1,142 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.common.util; + +import static io.lenses.streamreactor.common.util.JarManifest.ManifestAttributes.GIT_HASH; +import static io.lenses.streamreactor.common.util.JarManifest.ManifestAttributes.GIT_REPO; +import static io.lenses.streamreactor.common.util.JarManifest.ManifestAttributes.GIT_TAG; +import static io.lenses.streamreactor.common.util.JarManifest.ManifestAttributes.KAFKA_VER; +import static io.lenses.streamreactor.common.util.JarManifest.ManifestAttributes.REACTOR_DOCS; +import static io.lenses.streamreactor.common.util.JarManifest.ManifestAttributes.REACTOR_VER; +import static java.util.Optional.of; +import static java.util.Optional.ofNullable; + +import io.lenses.streamreactor.common.exception.ConnectorStartupException; +import java.io.File; +import java.io.IOException; +import java.net.URISyntaxException; +import java.net.URL; +import java.util.HashMap; +import java.util.Map; +import java.util.Optional; +import java.util.jar.Attributes; +import java.util.jar.JarFile; +import java.util.jar.Manifest; + +/** + * Class that reads JAR Manifest files so we can easily get some of the properties from it. + */ +public class JarManifest { + + private static final String UNKNOWN = "unknown"; + private static final String NEW_LINE = System.getProperty("line.separator"); + private static final String SEMICOLON = ":"; + private final Map jarAttributes = new HashMap<>(); + + /** + * Creates JarManifest. + * @param location Jar file location + */ + public JarManifest(URL location) { + Manifest manifest; + + try (JarFile jarFile = new JarFile(new File(location.toURI()))) { + manifest = jarFile.getManifest(); + } catch (URISyntaxException | IOException e) { + throw new ConnectorStartupException(e); + } + extractMainAttributes(manifest.getMainAttributes()); + } + + /** + * Creates JarManifest. + * @param jarFile + */ + public JarManifest(JarFile jarFile) { + Manifest manifest; + try { + Optional jarFileOptional = of(jarFile); + manifest = jarFileOptional.get().getManifest(); + } catch (NullPointerException | IOException e) { + throw new ConnectorStartupException(e); + } + extractMainAttributes(manifest.getMainAttributes()); + } + + private void extractMainAttributes(Attributes mainAttributes) { + jarAttributes.put(REACTOR_VER.getAttributeName(), + ofNullable(mainAttributes.getValue(REACTOR_VER.getAttributeName())).orElse(UNKNOWN)); + jarAttributes.put(KAFKA_VER.getAttributeName(), + ofNullable(mainAttributes.getValue(KAFKA_VER.getAttributeName())).orElse(UNKNOWN)); + jarAttributes.put(GIT_REPO.getAttributeName(), + ofNullable(mainAttributes.getValue(GIT_REPO.getAttributeName())).orElse(UNKNOWN)); + jarAttributes.put(GIT_HASH.getAttributeName(), + ofNullable(mainAttributes.getValue(GIT_HASH.getAttributeName())).orElse(UNKNOWN)); + jarAttributes.put(GIT_TAG.getAttributeName(), + ofNullable(mainAttributes.getValue(GIT_TAG.getAttributeName())).orElse(UNKNOWN)); + jarAttributes.put(REACTOR_DOCS.getAttributeName(), + ofNullable(mainAttributes.getValue(REACTOR_DOCS.getAttributeName())).orElse(UNKNOWN)); + } + + /** + * Get StreamReactor version. + */ + public String getVersion() { + return jarAttributes.getOrDefault(REACTOR_VER.getAttributeName(), ""); + } + + /** + * Get all manifest file attributes in a {@link String} form. + */ + public String buildManifestString() { + StringBuilder manifestBuilder = new StringBuilder(); + manifestBuilder.append(REACTOR_VER.attributeName).append(SEMICOLON) + .append(jarAttributes.get(REACTOR_VER.getAttributeName())).append(NEW_LINE); + manifestBuilder.append(KAFKA_VER.attributeName).append(SEMICOLON) + .append(jarAttributes.get(KAFKA_VER.getAttributeName())).append(NEW_LINE); + manifestBuilder.append(GIT_REPO.attributeName).append(SEMICOLON) + .append(jarAttributes.get(GIT_REPO.getAttributeName())).append(NEW_LINE); + manifestBuilder.append(GIT_HASH.attributeName).append(SEMICOLON) + .append(jarAttributes.get(GIT_HASH.getAttributeName())).append(NEW_LINE); + manifestBuilder.append(GIT_TAG.attributeName).append(SEMICOLON) + .append(jarAttributes.get(GIT_TAG.getAttributeName())).append(NEW_LINE); + manifestBuilder.append(REACTOR_DOCS.attributeName).append(SEMICOLON) + .append(jarAttributes.get(REACTOR_DOCS.getAttributeName())).append(NEW_LINE); + return manifestBuilder.toString(); + } + + /** + * Enum that represents StreamReactor's important parameters from Manifest file. + */ + public enum ManifestAttributes { + REACTOR_VER("StreamReactor-Version"), + KAFKA_VER("Kafka-Version"), + GIT_REPO("Git-Repo"), + GIT_HASH("Git-Commit-Hash"), + GIT_TAG("Git-Tag"), + REACTOR_DOCS("StreamReactor-Docs"); + + private final String attributeName; + + ManifestAttributes(String attributeName) { + this.attributeName = attributeName; + } + + public String getAttributeName() { + return attributeName; + } + } +} diff --git a/java-connectors/kafka-connect-common/src/main/resources/logback.xml b/java-connectors/kafka-connect-common/src/main/resources/logback.xml new file mode 100644 index 000000000..920828a04 --- /dev/null +++ b/java-connectors/kafka-connect-common/src/main/resources/logback.xml @@ -0,0 +1,27 @@ + + + + + + + + + + + + + + + + + + + + + %d{ISO8601} %-5p [%t] [%c] [%M:%L] %m%n + + + + + + \ No newline at end of file diff --git a/java-connectors/kafka-connect-common/src/test/java/io/lenses/streamreactor/common/util/AsciiArtPrinterTest.java b/java-connectors/kafka-connect-common/src/test/java/io/lenses/streamreactor/common/util/AsciiArtPrinterTest.java new file mode 100644 index 000000000..188d6ab13 --- /dev/null +++ b/java-connectors/kafka-connect-common/src/test/java/io/lenses/streamreactor/common/util/AsciiArtPrinterTest.java @@ -0,0 +1,79 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.common.util; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import ch.qos.logback.classic.spi.ILoggingEvent; +import ch.qos.logback.core.read.ListAppender; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.slf4j.LoggerFactory; + +class AsciiArtPrinterTest { + + private static final String SOME_MANIFEST_DATA = "SOME_MANIFEST_DATA"; + private ListAppender logWatcher; + + @BeforeEach + void setUp() { + logWatcher = new ListAppender<>(); + logWatcher.start(); + ((ch.qos.logback.classic.Logger) LoggerFactory.getLogger(AsciiArtPrinter.class)).addAppender(logWatcher); + } + + @AfterEach + void teardown() { + ((ch.qos.logback.classic.Logger) LoggerFactory.getLogger(AsciiArtPrinter.class)).detachAndStopAllAppenders(); + } + + @Test + void printAsciiHeaderShouldPrintAsciiAndJarManifestIfBothValid() { + //given + JarManifest jarManifest = mock(JarManifest.class); + when(jarManifest.buildManifestString()).thenReturn(SOME_MANIFEST_DATA); + String testingAsciiArt = "/testingAsciiArt.txt"; + + //when + AsciiArtPrinter.printAsciiHeader(jarManifest, testingAsciiArt); + + //when + verify(jarManifest).buildManifestString(); + assertEquals(2, logWatcher.list.size()); + assertTrue(logWatcher.list.get(0).getFormattedMessage().startsWith("###########")); + assertTrue(logWatcher.list.get(1).getFormattedMessage().equals(SOME_MANIFEST_DATA)); + } + + @Test + void printAsciiHeaderShouldPrintOnlyJarManifestIfCouldntMakeInpurStreamToAscii() { + //given + JarManifest jarManifest = mock(JarManifest.class); + when(jarManifest.buildManifestString()).thenReturn(SOME_MANIFEST_DATA); + String testingAsciiArt = "/testingAsciiArta.txt"; + + //when + AsciiArtPrinter.printAsciiHeader(jarManifest, testingAsciiArt); + + //when + verify(jarManifest).buildManifestString(); + assertEquals(1, logWatcher.list.size()); + assertTrue(logWatcher.list.get(0).getFormattedMessage().equals(SOME_MANIFEST_DATA)); + } +} \ No newline at end of file diff --git a/java-connectors/kafka-connect-common/src/test/java/io/lenses/streamreactor/common/util/JarManifestTest.java b/java-connectors/kafka-connect-common/src/test/java/io/lenses/streamreactor/common/util/JarManifestTest.java new file mode 100644 index 000000000..a1d7188bf --- /dev/null +++ b/java-connectors/kafka-connect-common/src/test/java/io/lenses/streamreactor/common/util/JarManifestTest.java @@ -0,0 +1,82 @@ +/* + * Copyright 2017-2024 Lenses.io Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.lenses.streamreactor.common.util; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import io.lenses.streamreactor.common.util.JarManifest.ManifestAttributes; +import java.io.IOException; +import java.util.jar.Attributes; +import java.util.jar.JarFile; +import java.util.jar.Manifest; +import org.junit.jupiter.api.Test; + +class JarManifestTest { + + private static final String UNKNOWN = "unknown"; + + JarManifest testObj; + + @Test + void getVersionShouldReturnStreamReactorVersionIfIncludedInManifest() throws IOException { + // given + final String STREAM_REACTOR_VERSION = "1.2.3"; + JarFile jarFile = mock(JarFile.class); + Manifest manifest = mock(Manifest.class); + Attributes attributes = mock(Attributes.class); + + when(jarFile.getManifest()).thenReturn(manifest); + when(manifest.getMainAttributes()).thenReturn(attributes); + when(attributes.getValue(ManifestAttributes.REACTOR_VER.getAttributeName())).thenReturn(STREAM_REACTOR_VERSION); + + testObj = new JarManifest(jarFile); + + // when + String streamReactorVersion = testObj.getVersion(); + + //then + verify(jarFile).getManifest(); + verify(manifest).getMainAttributes(); + verify(attributes).getValue(ManifestAttributes.REACTOR_VER.getAttributeName()); + assertEquals(streamReactorVersion, STREAM_REACTOR_VERSION); + } + + @Test + void getVersionShouldReturnUnknownVersionIfNotIncludedInManifest() throws IOException { + // given + JarFile jarFile = mock(JarFile.class); + Manifest manifest = mock(Manifest.class); + Attributes attributes = mock(Attributes.class); + + when(jarFile.getManifest()).thenReturn(manifest); + when(manifest.getMainAttributes()).thenReturn(attributes); + when(attributes.getValue(ManifestAttributes.REACTOR_VER.getAttributeName())).thenReturn(null); + + testObj = new JarManifest(jarFile); + + // when + String streamReactorVersion = testObj.getVersion(); + + //then + verify(jarFile).getManifest(); + verify(manifest).getMainAttributes(); + verify(attributes).getValue(ManifestAttributes.REACTOR_VER.getAttributeName()); + assertEquals(UNKNOWN, streamReactorVersion); + } +} \ No newline at end of file diff --git a/java-connectors/kafka-connect-common/src/test/resources/testingAsciiArt.txt b/java-connectors/kafka-connect-common/src/test/resources/testingAsciiArt.txt new file mode 100644 index 000000000..9c9544df5 --- /dev/null +++ b/java-connectors/kafka-connect-common/src/test/resources/testingAsciiArt.txt @@ -0,0 +1,5 @@ +########### +##TESTING## +###ASCII### +####ART#### +########### \ No newline at end of file diff --git a/java-connectors/kafka-connect-query-language/.gitignore b/java-connectors/kafka-connect-query-language/.gitignore deleted file mode 100644 index 66712446c..000000000 --- a/java-connectors/kafka-connect-query-language/.gitignore +++ /dev/null @@ -1,28 +0,0 @@ -/docs/build -/*.iml -/*/sample.db -/.idea -/.gradle -/build -/configs -/*/build -*.class -*.log -temp - -# sbt specific -.cache -.history -.lib/ -dist/* -target/ -lib_managed/ -src_managed/ -project/boot/ -project/plugins/project/ - -# Scala-IDE specific -.scala_dependencies -.worksheet -src/main/java/com/datamountaineer/connector/config/antlr4/ -/src/main/java/com/datamountaineer/kcql/antlr4/ diff --git a/java-connectors/kafka-connect-query-language/build.gradle b/java-connectors/kafka-connect-query-language/build.gradle new file mode 100644 index 000000000..780a8e5ae --- /dev/null +++ b/java-connectors/kafka-connect-query-language/build.gradle @@ -0,0 +1,57 @@ +project(":kafka-connect-query-language") { + apply plugin: 'antlr' + + test { + maxParallelForks = 1 + } + + ext { + antlrVersion = "4.13.1" + } + + repositories { + mavenLocal() + mavenCentral() + } + + dependencies { + antlr group: 'org.antlr', name: 'antlr4', version: antlrVersion + implementation group: 'org.antlr', name: 'antlr4-runtime', version: antlrVersion + } + + generateGrammarSource { + arguments += ["-package", "io.lenses.kcql.antlr4"] + outputDirectory = new File("${project.projectDir}/build/generated/antlr/main/java/io/lenses/kcql/antlr4") + } + + sourceSets { + main.java.srcDirs += 'build/generated/antlr/main/java' + main.antlr.srcDirs = ['src/main/antlr4'] + } + + task sourcesJar(type: Jar) { + archiveClassifier = 'sources' + from sourceSets.main.allSource + } + + task javadocJar(type: Jar) { + archiveClassifier = 'javadoc' + from javadoc + } + + tasks.withType(Tar) { + compression Compression.GZIP + extension = 'tgz' + } + + + artifacts { + archives javadocJar, sourcesJar + } + + //order should be: generateGrammarSource -> updateLicenseMain -> compileJava + checkLicenseMain.dependsOn("generateGrammarSource") + + task compile(dependsOn: 'compileJava') + task fatJarNoTest(dependsOn: 'shadowJar') +} \ No newline at end of file diff --git a/java-connectors/kafka-connect-query-language/checkstyle/checkstyle.xml b/java-connectors/kafka-connect-query-language/checkstyle/checkstyle.xml deleted file mode 100644 index a46727da8..000000000 --- a/java-connectors/kafka-connect-query-language/checkstyle/checkstyle.xml +++ /dev/null @@ -1,86 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/java-connectors/settings.gradle b/java-connectors/settings.gradle new file mode 100644 index 000000000..842b4f643 --- /dev/null +++ b/java-connectors/settings.gradle @@ -0,0 +1,5 @@ +rootProject.name = 'java-reactor' +include 'kafka-connect-common', +'kafka-connect-azure-eventhubs', +'kafka-connect-query-language' +