Breaking changes for type removal #132

VachaShah · 2022-03-30T18:32:32Z

Description

This PR contains the first set of breaking changes for 2.0 related to OpenSearch 2.0 - opensearch-project/OpenSearch#2480

The following changes are included in this PR:

A CI workflow test-integration-unreleased.yml which runs integ tests for the client against unreleased OpenSearch. This will help in future development to prepare clients with the latest unreleased OpenSearch versions. Currently this workflow will test against 2.0 OpenSearch.
The CI workflow test-integration.yml is updated to work for released versions which the client is compatible with.
Upgraded the OpenSearch version in the client to 2.0.0-alpha1-SNAPSHOT.
Type mapping removal across the APIs and related test changes.
Remove indices.exists_type and indices.flush_synced and related files.

Issues Resolved

Part of #139

Check List

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

reta · 2022-03-30T18:52:24Z

.github/workflows/build.yml

@@ -17,6 +17,19 @@ jobs:
        with:
          java-version: '14'
          distribution: 'adopt'
+
+      - name: Checkout OpenSearch


Hm ... why we are building opensearch-project/opensearch instead of using published snapshots from https://aws.oss.sonatype.org/content/repositories/snapshots/?

Ah my bad! I can use that for build and unit test. I will update the workflows for the same.

reta · 2022-03-30T18:54:26Z

java-client/src/main/java/org/opensearch/client/opensearch/core/BulkRequest.java

-				if (propsSet == (_index | _type)) {
-					StringBuilder buf = new StringBuilder();
-					buf.append("/");
-					SimpleEndpoint.pathEncode(request.index, buf);


Seems like _index should not be dropped

The _index part is before this if condition. https://github.com/opensearch-project/opensearch-java/pull/132/files/1e01fbeae03463bf2d47b8d395bce3701901448d#diff-2bd95eccdb4274c5118be928fb9f466034f19fc2cdcfc3e5c82609585af94e0aL576

Ah ... not visible in changeset, thank you!

reta · 2022-03-30T21:57:46Z

.github/workflows/test-integration-unreleased.yml

+        # Reference: https://github.com/opensearch-project/OpenSearch/blob/2.0/distribution/docker/build.gradle#L190
+      - name: Run Docker Image
+        run: |
+          docker run -p 9200:9200 -p 9600:9600 -d -e "discovery.type=single-node" -e "bootstrap.memory_lock=true" opensearch:test


I am 100% sure there is a staging repository for Docker images, @dblock @peterzhuamazon could you please refer to the right one?

Yes there is one, but the staging image is with all the plugins which isn't ready yet. This is to test against min OpenSearch image. This also helps in client development when a new version of OpenSearch is being developed.

I also tried the route of writing Dockerfile and related things to create a docker image for min OpenSearch and use the script in opensearch-build to use the build and run the image. But then I found this gradle task that runs in assemble for OpenSearch so used this one. LMK if you think the Dockerfile route is better. I can use that instead.

I think publishing min Docker images (along with artifacts) would be the best option, may be we could quickly address that? Another possibility is to use OpenSearch test scaffolding (available as set of Gradle plugins) to run clusters from the test suites. What do you think?

I didn't use the Gradle plugins since I wanted to create a CI way for all the clients. It can work for java client but we would still end up in the same place for other clients. The main purpose of this is for clients to be ready to be released when a new version of OpenSearch is being released.

Agree, thank you, publishing min Docker images (along with artifacts) would be the answer, right?

Yes that would be the best option. Till that happens, is it fine if we use this change so we can start the development for 2.0? WDYT?

Yeah, sure, I think it is viable option

I believe we will be adding API for plugins too and we need unreleased complete distribution for testing. Shall we create an issue to track this if it is not created already?

Yes, please do, I think it will useful for many

VachaShah · 2022-03-30T22:28:20Z

I will merge the changes from #134 once that is approved and merged and update the workflows with the appropriate JDK versions.

Signed-off-by: Vacha Shah <[email protected]>

VachaShah · 2022-04-01T07:39:16Z

@reta @CEHENKLE @VijayanB A review again please? Just merged the changes from the main branch for the workflows and removed indices.exists_type as part of the breaking changes.

owaiskazi19 · 2022-04-01T19:11:24Z

@VachaShah I see java client is using checkstyle for formatting. Moving forward do we have any plan to have Spotless as the formatting tool?

VachaShah · 2022-04-01T19:15:52Z

@VachaShah I see java client is using checkstyle for formatting. Moving forward do we have any plan to have Spotless as the formatting tool?

No plans as such but you can create a PR :)

VijayanB

LGTM. Thanks for making the changes.

VachaShah · 2022-04-01T21:33:36Z

Not sure why the whitesource check is failing, the dependency was updated in #129. Will keep an eye on it.

dermot-hardy · 2022-06-24T12:36:51Z

java-client/build.gradle.kts

    val jacksonVersion = "2.12.6"
    val jacksonDatabindVersion = "2.12.6.1"

    // Apache 2.0
    implementation("org.opensearch.client", "opensearch-rest-client", opensearchVersion)
    testImplementation("org.opensearch.test", "framework", opensearchVersion)
+    implementation("org.apache.logging.log4j", "log4j-core", "2.17.2")


Is this introducing a hard-coded dependency on the Log4j framework? Surely the client should only be dependent on the facade?

I think you may be right @dermot-hardy! Care to open an issue/contribute a fix?

VachaShah requested review from a team and madhusudhankonda as code owners March 30, 2022 18:32

reta reviewed Mar 30, 2022

View reviewed changes

reta previously approved these changes Mar 30, 2022

View reviewed changes

CEHENKLE previously approved these changes Mar 30, 2022

View reviewed changes

VachaShah added 4 commits March 31, 2022 16:25

Adding support to test against unreleased OpenSearch

adac661

Signed-off-by: Vacha Shah <[email protected]>

Upgrading OpenSearch version to 2.0.0-SNAPSHOT and Java version

e629651

Signed-off-by: Vacha Shah <[email protected]>

Remove type mapping

5898cc0

Signed-off-by: Vacha Shah <[email protected]>

Using snapshots from sonatype for 2.0 OpenSearch snapshot

a8d1300

Signed-off-by: Vacha Shah <[email protected]>

VachaShah dismissed stale reviews from CEHENKLE and reta via c8b8720 April 1, 2022 01:09

VachaShah force-pushed the breaking-changes-remove-type branch from 35b7c06 to c8b8720 Compare April 1, 2022 01:09

Updating workflows and removing exists_type

d82f4b6

Signed-off-by: Vacha Shah <[email protected]>

VachaShah force-pushed the breaking-changes-remove-type branch from c8b8720 to d82f4b6 Compare April 1, 2022 03:34

Fixing integration tests

0b0a118

Signed-off-by: Vacha Shah <[email protected]>

VachaShah requested review from VijayanB and CEHENKLE April 1, 2022 07:35

CEHENKLE approved these changes Apr 1, 2022

View reviewed changes

reta approved these changes Apr 1, 2022

View reviewed changes

VachaShah added breaking 2.0 labels Apr 1, 2022

This was referenced Apr 1, 2022

Breaking changes for 2.0 opensearch-project/opensearch-dsl-py#43

Closed

Breaking changes for 2.0 #139

Closed

Breaking changes for 2.0 opensearch-project/opensearch-js#217

Closed

VachaShah requested a review from dblock April 1, 2022 19:55

VijayanB approved these changes Apr 1, 2022

View reviewed changes

VachaShah merged commit 02a313f into opensearch-project:main Apr 1, 2022

VachaShah mentioned this pull request Apr 1, 2022

[FEATURE] Test against latest SNAPSHOTs of OpenSearch v. Next #123

Closed

VachaShah deleted the breaking-changes-remove-type branch April 5, 2022 21:57

naveentatikonda mentioned this pull request Apr 21, 2022

Breaking changes for 2.0 opensearch-project/logstash-input-opensearch#36

Open

dermot-hardy reviewed Jun 24, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Breaking changes for type removal #132

Breaking changes for type removal #132

VachaShah commented Mar 30, 2022 •

edited

Loading

reta Mar 30, 2022

VachaShah Mar 30, 2022

reta Mar 30, 2022

VachaShah Mar 30, 2022

reta Mar 30, 2022

reta Mar 30, 2022

VachaShah Mar 30, 2022 •

edited

Loading

VachaShah Mar 30, 2022 •

edited

Loading

reta Mar 30, 2022

VachaShah Mar 30, 2022 •

edited

Loading

reta Mar 30, 2022

VachaShah Mar 30, 2022

reta Mar 30, 2022

VijayanB Mar 31, 2022 •

edited

Loading

reta Mar 31, 2022

VachaShah commented Mar 30, 2022

VachaShah commented Apr 1, 2022

owaiskazi19 commented Apr 1, 2022

VachaShah commented Apr 1, 2022

VijayanB left a comment

VachaShah commented Apr 1, 2022

dermot-hardy Jun 24, 2022

dblock Jun 27, 2022

dermot-hardy Jul 3, 2022

Breaking changes for type removal #132

Breaking changes for type removal #132

Conversation

VachaShah commented Mar 30, 2022 • edited Loading

Description

Issues Resolved

Check List

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VachaShah Mar 30, 2022 • edited Loading

Choose a reason for hiding this comment

VachaShah Mar 30, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VachaShah Mar 30, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VijayanB Mar 31, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VachaShah commented Mar 30, 2022

VachaShah commented Apr 1, 2022

owaiskazi19 commented Apr 1, 2022

VachaShah commented Apr 1, 2022

VijayanB left a comment

Choose a reason for hiding this comment

VachaShah commented Apr 1, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VachaShah commented Mar 30, 2022 •

edited

Loading

VachaShah Mar 30, 2022 •

edited

Loading

VachaShah Mar 30, 2022 •

edited

Loading

VachaShah Mar 30, 2022 •

edited

Loading

VijayanB Mar 31, 2022 •

edited

Loading