[Backport 2.x] Add support for API versioning and fail open in weighted shard routing #5779

…lusterManager'. (opensearch-project#4032) (opensearch-project#4048)

…roject#3796) (opensearch-project#3990) * Parallelize stale blobs deletion during snapshot delete Signed-off-by: Piyush Daftary <[email protected]> * Adding test which throws exception Signed-off-by: Piyush Daftary <[email protected]> * Adusting identation for spotlessJavaCheck Signed-off-by: Piyush Daftary <[email protected]> * Adding more description to MAX_SHARD_BLOB_DELETE_BATCH_SIZE Signed-off-by: Piyush Daftary <[email protected]> * Renaming max_shard_blob_delete_batch_size to max_snapshot_shard_blob_delete_batch_size Signed-off-by: Piyush Daftary <[email protected]> (cherry picked from commit 1c787e8) Co-authored-by: piyush <[email protected]>

…hdfs (opensearch-project#3764) (opensearch-project#3783) * Bump commons-configuration2 in /plugins/repository-hdfs Bumps commons-configuration2 from 2.7 to 2.8.0. --- updated-dependencies: - dependency-name: org.apache.commons:commons-configuration2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit 5a1dbbe) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ManagerService' (opensearch-project#4022) (opensearch-project#4050) To support inclusive language, the `master` terminology is going to be replaced by `cluster manager` in the code base. - Deprecate class `MasterService` and create alternative class `ClusterManagerService`. - Add a unit test to validate the method `ClusterService.getMasterService()` can still return an object in type `MasterService`. - Rename all the existing references of `MasterService` to `ClusterManagerService`, and rename the local variable names. - Deprecate public methods `ClusterServiceUtils.createMasterService(...)` and create alternative methods `createClusterManagerService(...)` Note: The class `ClusterManagerService` is a subclass of `MasterService`, the inheritance relationship is opposite from most of the other classes with `Master` in the name (that covered by issue opensearch-project#1684). The reason is: There is a public method that has return value in type `MasterService`, https://github.com/opensearch-project/OpenSearch/blob/388c80ad94529b1d9aad0a735c4740dce2932a32/server/src/main/java/org/opensearch/cluster/service/ClusterService.java#L221 And in the code for Performance Analyzer plugin, there is a local variable in type `MasterService`: https://github.com/opensearch-project/performance-analyzer/blob/5ee4809ac1cda6517ed871aeb12c6635203e7f1d/src/main/java/org/opensearch/performanceanalyzer/collectors/MasterServiceEventMetrics.java#L219 If making the old class `MasterService` a subclass of the new class `ClusterManagerService`, the above usage will be broken. Reversing the inheritance relationship, I'm able to keep the backwards compatibility of the method `getMasterService()` while deprecating the class `MasterService` and encourage using a new class `ClusterManagerService`. Signed-off-by: Tianli Feng <[email protected]> (cherry picked from commit 740f75d)

…n directory 'test/framework' (opensearch-project#4051) (opensearch-project#4057) To support inclusive language, the master terminology is going to be replaced by cluster manager in the code base. After the class MasterService has been deprecated and class ClusterManagerService has been created in opensearch-project#4022 / commit 740f75d, the classes in `test/framework` directory with name 'MasterService' can be deprecated and renamed. - Rename the following classes in `test/framework` directory: FakeThreadPoolMasterService -> FakeThreadPoolClusterManagerService BlockMasterServiceOnMaster -> BlockClusterManagerServiceOnClusterManager BusyMasterServiceDisruption -> BusyClusterManagerServiceDisruption In the following commit, I will add back the above 3 classes with the old name and deprecate them, to keep the backwards compatibility. Signed-off-by: Tianli Feng <[email protected]> (cherry picked from commit bea5d1a) Co-authored-by: Tianli Feng <[email protected]>

…ect#4037) Bucket aggregations compute bucket doc_count values by incrementing the doc_count by 1 for every document collected in the bucket. When using summary fields (such as aggregate_metric_double) one field may represent more than one document. To provide this functionality this commit implements a new field mapper (named doc_count field mapper). This field is a positive integer representing the number of documents aggregated in a single summary field. Bucket aggregations check if a field of type doc_count exists in a document and take this value into consideration when computing doc counts. Note: This originated from upstream PR 64503. Signed-off-by: Petar Dzepina <[email protected]> (cherry picked from commit fb7d81a) Co-authored-by: Petar Dzepina <[email protected]>

…r to trigger when Segment replication is turned on and Primary shard refreshes (opensearch-project#4044) * Adding CheckpointRefreshListener to trigger when Segment replication is turned on and Primary shard refreshes (opensearch-project#3108) Signed-off-by: Rishikesh1159 <[email protected]>

…er and BusyMasterServiceDisruption (opensearch-project#4058) (opensearch-project#4068) To support inclusive language, the master terminology is going to be replaced by cluster manager in the code base. In a previous PR opensearch-project#4051 / commit opensearch-project@bea5d1a, 3 classes in `test/framework` directory that contains `master` in the name were renamed: ``` FakeThreadPoolMasterService -> FakeThreadPoolClusterManagerService BlockMasterServiceOnMaster -> BlockClusterManagerServiceOnClusterManager BusyMasterServiceDisruption -> BusyClusterManagerServiceDisruption ``` This is a following commit to add back the classes with the old name to keep the backwards compatibility. The classes with the old name will be subclass of the classes with new name, so that maintaining one implementation can support the usage for two classes. Signed-off-by: Tianli Feng <[email protected]> (cherry picked from commit cebeb8c) Co-authored-by: Tianli Feng <[email protected]>

…ject#3462) (opensearch-project#4086) * [Backport 2.x] Making all the copies a multiple of attribute count Signed-off-by: Gaurav Bafna <[email protected]>

opensearch-project#4087) * [Backport 2.x] Support task resource tracking in OpenSearch * Reopens changes from opensearch-project#2639 (reverted in opensearch-project#3046) to add a framework for task resource tracking. Currently, SearchTask and SearchShardTask support resource tracking but it can be extended to any other task. * Fixed a race-condition when Task is unregistered before its threads are stopped * Improved error handling and simplified task resource tracking completion listener * Avoid registering listeners on already completed tasks Signed-off-by: Ketan Verma <[email protected]>

…pensearch-project#3822) [Backport 2.x] Integrate Engine with decoupled Translog interfaces Co-authored-by: Bukhtawar Khan <[email protected]> Co-authored-by: Satyajit Ganguly <[email protected]>

…4088) Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit bab2e59) Signed-off-by: Andriy Redko <[email protected]>

…pensearch-project#4092) Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit d1ec4ea) Co-authored-by: Andriy Redko <[email protected]>

…ice to orchestrate replication events. (opensearch-project#4074) * [Segment Replication] Add SegmentReplicationTargetService to orchestrate replication events. (opensearch-project#3439) * Add SegmentReplicationTargetService to orchestrate replication events. This change introduces boilerplate classes for Segment Replication and a target service to orchestrate replication events. It also includes two refactors of peer recovery components for reuse. 1. Rename RecoveryFileChunkRequest to FileChunkRequest and extract code to handle throttling into ReplicationTarget. 2. Extracts a component to execute retryable requests over the transport layer. Signed-off-by: Marc Handalian <[email protected]> * Code cleanup. Signed-off-by: Marc Handalian <[email protected]> * Make SegmentReplicationTargetService component final so that it can not be extended by plugins. Signed-off-by: Marc Handalian <[email protected]> * Removing Wildcard imports Signed-off-by: Rishikesh1159 <[email protected]> Co-authored-by: Marc Handalian <[email protected]>

Signed-off-by: Kartik Ganesh <[email protected]>

…ject#4054) (opensearch-project#4078) Signed-off-by: Daniel Widdis <[email protected]> (cherry picked from commit 707ca13) Co-authored-by: Daniel Widdis <[email protected]>

…pensearch-project#4095) * Changes to encapsulate Translog into TranslogManager. Signed-off-by: Satyajit Ganguly <[email protected]> Co-authored-by: Bukhtawar Khan <[email protected]>

…arch-project#4103) (opensearch-project#4114)

…ter' in name (opensearch-project#4121) (opensearch-project#4122) Signed-off-by: Xue Zhou <[email protected]> (cherry picked from commit ba9cdcd) Co-authored-by: Xue Zhou <[email protected]>

…4077) (opensearch-project#4108) Signed-off-by: Peter Zhu <[email protected]> (cherry picked from commit 27c5493) Co-authored-by: Peter Zhu <[email protected]>

…ct#4140) * Integrate task consumers to capture task resource information during unregister. Add consumer that logs topN expensive search tasks Co-authored-by: Sruti Parthiban <[email protected]>

…en total ops higher compared to expected (opensearch-project#4143) (opensearch-project#4144) Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit 203f44e) Co-authored-by: Andriy Redko <[email protected]>

…pensearch-project#4154) Signed-off-by: Kartik Ganesh <[email protected]>

* Upgrade dependencies (opensearch-project#4047) * Upgrading dependencies for hadoop and aws-java-sdk Signed-off-by: Vacha Shah <[email protected]> * Fixing precommit Signed-off-by: Vacha Shah <[email protected]> * Upgrading transitive dependencies Signed-off-by: Vacha Shah <[email protected]> * Excluding transitive dependencies Signed-off-by: Vacha Shah <[email protected]> (cherry picked from commit 88f5537) * Removing seg-rep IT class from the scope of this backport The original change included a javadoc update to the IT class, but the test class was never backported from opensearch-project#3743 to 2.x. Since the scope of this change is to backport dependency upgrades only, this class has been removed. Signed-off-by: Kartik Ganesh <[email protected]> Co-authored-by: Vacha Shah <[email protected]>

…chestrating replication events. (opensearch-project#4128) * [Segment Replication] Added source-side classes for orchestrating replication events (opensearch-project#3470) This change expands on the existing SegmentReplicationSource interface and its corresponding Factory class by introducing an implementation where the replication source is a primary shard (PrimaryShardReplicationSource). These code paths execute on the target. The primary shard implementation creates the requests to be send to the source/primary shard. Correspondingly, this change also defines two request classes for the GET_CHECKPOINT_INFO and GET_SEGMENT_FILES requests as well as an abstract superclass. A CopyState class has been introduced that captures point-in-time, file-level details from an IndexShard. This implementation mirrors Lucene's NRT CopyState implementation. Finally, a service class has been introduce for segment replication that runs on the source side (SegmentReplicationSourceService) which handles these two types of incoming requests. This includes private handler classes that house the logic to respond to these requests, with some functionality stubbed for now. The service class also uses a simple map to cache CopyState objects that would be needed by replication targets. Unit tests have been added/updated for all new functionality. Signed-off-by: Kartik Ganesh <[email protected]> * Remove abstract getProcessedLocalCheckpoint() from Engine.java as it is a breaking change and fix other uses of getProcessedLocalCheckpoint() Signed-off-by: Rishikesh1159 <[email protected]> * Apply spotlessCheck Signed-off-by: Rishikesh1159 <[email protected]> * Fixing error in ReadOnlyEngineTests by casting. Signed-off-by: Rishikesh1159 <[email protected]> * Addressing comments on PR. Signed-off-by: Rishikesh1159 <[email protected]> Co-authored-by: Kartik Ganesh <[email protected]>

…ch-project#4150) (opensearch-project#4167) Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit 0f2ed70) Co-authored-by: Andriy Redko <[email protected]>

…ct#3525 opensearch-project#3533 opensearch-project#3540 opensearch-project#3943 opensearch-project#3963 From main branch (opensearch-project#4181) * Resolving import conflict in Node.java and mergining PR opensearch-project#3525. Signed-off-by: Rishikesh1159 <[email protected]> * Resolving conflicts and merging PR opensearch-project#3533. Signed-off-by: Rishikesh1159 <[email protected]> * Resolving conflicts and Merging PR opensearch-project#3540. Signed-off-by: Rishikesh1159 <[email protected]> * Applying spotlesscheck and fixing wildcard imports. Signed-off-by: Rishikesh1159 <[email protected]> * [Segment Replication] Fixing flaky test failure happening for testShardAlreadyReplicating() (opensearch-project#3943) * Fixing flaky test failure happening for testShardAlreadyReplicating() Signed-off-by: Rishikesh1159 <[email protected]> * Fix possible flaky test for testBeforeIndexShardClosed_CancelsOngoingReplications() (opensearch-project#3963) * Fixing flaky test failure happening for testShardAlreadyReplicating() Signed-off-by: Rishikesh1159 <[email protected]> * Removing assert segrep() in getProcessedLocalCheckpoint() of Index.shard class. Signed-off-by: Rishikesh1159 <[email protected]> * Adding back assert statement and make index setting to segment replication in SegmentReplicationSourceHandlerTests and SegmentReplicationTargetServiceTests. Signed-off-by: Rishikesh1159 <[email protected]> * Revert "Adding back assert statement and make index setting to segment replication in SegmentReplicationSourceHandlerTests and SegmentReplicationTargetServiceTests." Signed-off-by: Rishikesh1159 <[email protected]> This reverts commit 8c5753b. Signed-off-by: Rishikesh1159 <[email protected]> Co-authored-by: Marc Handalian <[email protected]> Co-authored-by: Poojita Raj <[email protected]>

…4213) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit 0dbbd72) Co-authored-by: Andriy Redko <[email protected]>

…server to the geo module. (opensearch-project#4179) Refactors the GeoBoundsAggregation for geo_point types from the core server to the geo module. Signed-off-by: Navneet Verma <[email protected]>

Co-authored-by: opensearch-ci-bot <[email protected]>

…aining segment replication changes (opensearch-project#4243) * [Segment Replication] Checkpoint Replay on Replica Shard (opensearch-project#3658) * Adding Latest Recevied checkpoint, replay checkpoint logic along with tests Signed-off-by: Rishikesh1159 <[email protected]> * [Segment Replication] Wire up segment replication with peer recovery and add ITs. (opensearch-project#3743) * Add null check when computing max segment version. With segment replication enabled it is possible Lucene does not set the SegmentInfos min segment version, leaving the default value as null. Signed-off-by: Marc Handalian <[email protected]> * Update peer recovery to set the translogUUID of replicas to the UUID generated on the primary. This change updates the UUID when the translog is created to the value stored in the passed segment userdata. This is to ensure during failover scenarios that the replica can be promoted and not have a uuid mismatch with the value stored in user data. Signed-off-by: Marc Handalian <[email protected]> * Wire up Segment Replication under the feature flag. This PR wires up segment replication and adds some initial integration tests. Signed-off-by: Marc Handalian <[email protected]> * Add test to ensure replicas use primary translog uuid with segrep. Signed-off-by: Marc Handalian <[email protected]> * Update SegmentReplicationIT to assert previous commit points are valid and SegmentInfos can be built. Fix nitpicks in PR feedback. Signed-off-by: Marc Handalian <[email protected]> * Fix test with Assert.fail to include a message. Signed-off-by: Marc Handalian <[email protected]> * Disable shard idle with segment replication. (opensearch-project#4118) This change disables shard idle when segment replication is enabled. Primary shards will only push out new segments on refresh, we do not want to block this based on search behavior. Replicas will only refresh on an externally provided SegmentInfos, so we do not want search requests to hang waiting for a refresh. Signed-off-by: Marc Handalian <[email protected]> * Fix isAheadOf logic for ReplicationCheckpoint comparison (opensearch-project#4112) Signed-off-by: Suraj Singh <[email protected]> * Fix waitUntil refresh policy for segrep enabled indices. (opensearch-project#4137) Signed-off-by: Marc Handalian <[email protected]> * Add IndexShard#getLatestReplicationCheckpoint behind segrep enable feature flag (opensearch-project#4163) * Add IndexShard#getLatestReplicationCheckpoint behind segrep enable feature flag Signed-off-by: Suraj Singh <[email protected]> * Address review comment. Move tests to SegmentReplicationIndexShardTests Signed-off-by: Suraj Singh <[email protected]> * Add segrep enbaled index settings in TargetServiceTests, SourceHandlerTests Signed-off-by: Suraj Singh <[email protected]> * [Segment Replication] Fix OngoingSegmentReplications to key by allocation ID instead of DiscoveryNode. (opensearch-project#4182) * Fix OngoingSegmentReplications to key by allocation ID instead of DiscoveryNode. This change fixes segrep to work with multiple shards per node by keying ongoing replications on allocation ID. This also updates cancel methods to ensure state is properly cleared on shard cancel. Signed-off-by: Marc Handalian <[email protected]> * Clean up cancel methods. Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Marc Handalian <[email protected]> * [Bug] [Segment Replication] Update store metadata recovery diff logic to ignore missing files causing exception (opensearch-project#4185) * Update Store for segment replication dif Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Poojita Raj <[email protected]> * Update recoveryDiff logic to ingore missing files causing exception on replica during copy Signed-off-by: Suraj Singh <[email protected]> * Address review comments Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Suraj Singh <[email protected]> Co-authored-by: Poojita Raj <[email protected]> * [Segment Replication] Adding PrimaryMode check before publishing checkpoint and processing a received checkpoint. (opensearch-project#4157) * Adding PrimaryMode check before publishing checkpoint. Signed-off-by: Rishikesh1159 <[email protected]> * Applying spotless check Signed-off-by: Rishikesh1159 <[email protected]> * Moving segrep specific tests to SegmentReplicationIndexShardTests. Signed-off-by: Rishikesh1159 <[email protected]> * Adding logic and tests for rejecting checkpoints if shard is in PrimaryMode. Signed-off-by: Rishikesh1159 <[email protected]> * Applying ./gradlew :server:spotlessApply. Signed-off-by: Rishikesh1159 <[email protected]> * Applying ./gradlew :server:spotlessApply Signed-off-by: Rishikesh1159 <[email protected]> * Changing log level to warn in shouldProcessCheckpoint() of IndexShard.java class. Signed-off-by: Rishikesh1159 <[email protected]> * Removing unnecessary lazy logging in shouldProcessCheckpoint(). Signed-off-by: Rishikesh1159 <[email protected]> Signed-off-by: Rishikesh1159 <[email protected]> * [Segment Replication] Wait for documents to replicate to replica shards (opensearch-project#4236) * [Segment Replication] Add thread sleep to account for replica lag in delete operations test Signed-off-by: Suraj Singh <[email protected]> * Address review comments, assertBusy on doc count rather than sleep Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Suraj Singh <[email protected]> * Segment Replication - Remove unnecessary call to markAllocationIdAsInSync. (opensearch-project#4224) This PR Removes an unnecessary call to markAllocationIdAsInSync on the primary shard when replication events complete. Recovery will manage this initial call. Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Marc Handalian <[email protected]> * Segment Replication - Add additional unit tests for update & delete ops. (opensearch-project#4237) * Segment Replication - Add additional unit tests for update & delete operations. Signed-off-by: Marc Handalian <[email protected]> * Fix spotless. Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Rishikesh1159 <[email protected]> Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Poojita Raj <[email protected]> Co-authored-by: Marc Handalian <[email protected]> Co-authored-by: Suraj Singh <[email protected]> Co-authored-by: Poojita Raj <[email protected]>

… with higher replication checkpoint (opensearch-project#4041) (opensearch-project#4252) * [Segment Replication] Update PrimaryShardAllocator to prefer replicas having higher replication checkpoint Signed-off-by: Suraj Singh <[email protected]> * Use empty replication checkpoint to avoid NPE Signed-off-by: Suraj Singh <[email protected]> * Update NodeGatewayStartedShards to optionally wire in/out ReplicationCheckpoint field Signed-off-by: Suraj Singh <[email protected]> * Use default replication checkpoint causing EOF errors on empty checkpoint * Add indexSettings to GatewayAllocator to allow ReplicationCheckpoint comparator only for segrep enabled indices * Add unit tests for primary term first replica promotion & comparator fix * Fix NPE on empty IndexMetadata * Remove settings from AllocationService and directly inject in GatewayAllocator * Add more unit tests and minor code clean up Signed-off-by: Suraj Singh <[email protected]> * Address review comments & integration test Signed-off-by: Suraj Singh <[email protected]> * Fix comparator on null ReplicationCheckpoint Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Suraj Singh <[email protected]>

…bat manager" (opensearch-project#4289) (opensearch-project#4293) * [BUG] Update opensearch-service-x64.exe parameters to //ES for Execute Service. Update opensearch-service-mgr.exe parameters to //ES for Edit Service. Add code comments for the Apache Commons Daemon. Signed-off-by: Alex Burck <[email protected]> * update changelog with pull request link Signed-off-by: Alex Burck <[email protected]> (cherry picked from commit a3fb105) Signed-off-by: Alex Burck <[email protected]> Signed-off-by: Alex Burck <[email protected]>

…ensearch-project#4284) * Add workflow for changelog verification Signed-off-by: Kunal Kotwani <[email protected]> * Update format for changelog, add developer documentation Signed-off-by: Kunal Kotwani <[email protected]> * Update link reference to be relative to project Signed-off-by: Kunal Kotwani <[email protected]> * Fix links for CHANGELOG versions Signed-off-by: Kunal Kotwani <[email protected]> * Update contribution guide Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> (cherry picked from commit 5327767) Co-authored-by: Kunal Kotwani <[email protected]>

…pensearch-project#4298) * Update changelog contribution guide Signed-off-by: Kunal Kotwani <[email protected]> * Fix reference to pull request Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> (cherry picked from commit 5dd7947)

…oject#4303) Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> (cherry picked from commit 1bfabed) Co-authored-by: Kunal Kotwani <[email protected]>

…#4311) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit c62cecb) Co-authored-by: Andriy Redko <[email protected]>

…search-project#4309) * Do not fail replica shard due to primary closure (opensearch-project#4133) This commit prevents a replica shard from being failed in the case that a replication action to a replica is canceled due to the primary shard being closed. Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Andrew Ross <[email protected]> (cherry picked from commit 1dbb63a) * Update version label in OpenSearchException Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: Andrew Ross <[email protected]>

…ns on GeoPoint from server folder to geo module.(opensearch-project#4071) (opensearch-project#4072) (opensearch-project#4180) (opensearch-project#4281) The changes also includes: * Updated Search plugin to provide the interface so that plugins can also register the composite aggregations * Added YAML test for the geo_grid, geo_tile and composite aggregation Signed-off-by: Navneet Verma <[email protected]>

…stBuilder from package 'org.opensearch.action.support.master.info' for subclasses (opensearch-project#4307) (opensearch-project#4324) * Restore using the class ClusterInfoRequest and ClusterInfoRequestBuilder from package 'org.opensearch.action.support.master.info' for subclasses (opensearch-project#4307) * Restore using the class ClusterInfoRequest and ClusterInfoRequestBuilder from package 'org.opensearch.action.support.master.info' for subclasses Signed-off-by: Tianli Feng <[email protected]> * Add changelog Signed-off-by: Tianli Feng <[email protected]> Signed-off-by: Tianli Feng <[email protected]> Co-authored-by: Andrew Ross <[email protected]> * Add changelog in 2.x section Signed-off-by: Tianli Feng <[email protected]> Signed-off-by: Tianli Feng <[email protected]> Co-authored-by: Andrew Ross <[email protected]>

…nsearch-project#4135) (opensearch-project#4325) * Support shard promotion with Segment Replication. This change adds basic failover support with segment replication. Once selected, a replica will commit and reopen a writeable engine. Signed-off-by: Marc Handalian <[email protected]> * Add check to ensure a closed shard does not publish checkpoints. Signed-off-by: Marc Handalian <[email protected]> * Clean up in SegmentReplicationIT. Signed-off-by: Marc Handalian <[email protected]> * PR feedback. Signed-off-by: Marc Handalian <[email protected]> * Fix merge conflict. Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Marc Handalian <[email protected]>

…ch-project#4347) The package `org.yaml:snakeyaml` before version 1.31 are vulnerable to Denial of Service (DoS) due missing to nested depth limitation for collections. Details at https://nvd.nist.gov/vuln/detail/CVE-2022-25857 Signed-off-by: Rabi Panda <[email protected]> (cherry picked from commit 4bccdbe) Co-authored-by: Rabi Panda <[email protected]>

… (opensearch-project#4345) Signed-off-by: Kartik Ganesh <[email protected]> Signed-off-by: Kartik Ganesh <[email protected]> (cherry picked from commit beb09af) Co-authored-by: Kartik Ganesh <[email protected]>

…nsearch-project#4305) (opensearch-project#4335) * [BUG] Create logs directory before running OpenSearch on Windows Signed-off-by: Alex Burck <[email protected]> * update changlog pr link Signed-off-by: Alex Burck <[email protected]> (cherry picked from commit 7ea6e88) Signed-off-by: Alex Burck <[email protected]> Signed-off-by: Alex Burck <[email protected]>

…eplicationState (opensearch-project#4367) * Added timing data and more granular stages to SegmentReplicationState (opensearch-project#4222) * Added timing data and more granular stages to SegmentReplicationState This change introduces instrumentation logging that measures the latency of the various stages of segment replication as seen by each replica. Logs have also been added to the source node for checkpoint publishing and checkpoint metadata responses. All logging is currently at the TRACE level. Signed-off-by: Kartik Ganesh <[email protected]> * Fixing SegmentReplicationTarget tests Signed-off-by: Kartik Ganesh <[email protected]> * Incorporated PR feedback Signed-off-by: Kartik Ganesh <[email protected]> * Fixing SegmentReplicationTargetService tests Signed-off-by: Kartik Ganesh <[email protected]> Signed-off-by: Kartik Ganesh <[email protected]> (cherry picked from commit a2ba3a8) * Update changelog for backport pr. Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Marc Handalian <[email protected]> Co-authored-by: Kartik Ganesh <[email protected]> Co-authored-by: Marc Handalian <[email protected]>

…oject#4374) Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit 236f2f6) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]>

* Added bwc version 2.2.2 * Add changelog Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Kunal Kotwani <[email protected]>

…pensearch-project#4380) * [Remote Store] Upload segments to remote store post refresh (opensearch-project#3460) * Add RemoteDirectory interface to copy segment files to/from remote store Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> * Add index level setting for remote store Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> * Add RemoteDirectoryFactory and use RemoteDirectory instance in RefreshListener Co-authored-by: Sachin Kale <[email protected]> Signed-off-by: Sachin Kale <[email protected]> * Upload segment to remote store post refresh Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> Signed-off-by: Sachin Kale <[email protected]> * [Remote Store] Inject remote store in IndexShard instead of RemoteStoreRefreshListener (opensearch-project#3703) * Inject remote store in IndexShard instead of RemoteStoreRefreshListener Signed-off-by: Sachin Kale <[email protected]> * Pass supplier of RepositoriesService to RemoteDirectoryFactory Signed-off-by: Sachin Kale <[email protected]> * Create isRemoteStoreEnabled function for IndexShard Signed-off-by: Sachin Kale <[email protected]> * Explicitly close remoteStore on indexShard close Signed-off-by: Sachin Kale <[email protected]> * Change RemoteDirectory.close to a no-op Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> * [Remote Store] Add remote store restore API implementation (opensearch-project#3642) * Add remote restore API implementation Signed-off-by: Sachin Kale <[email protected]> * [Remote Store] Add support to add nested settings for remote store (opensearch-project#4060) * Add support to add nested settings for remote store Signed-off-by: Sachin Kale <[email protected]> * [Remote Store] Add rest endpoint for remote store restore (opensearch-project#3576) * Add rest endpoint for remote store restore Signed-off-by: Sachin Kale <[email protected]> * [Remote Store] Add validator that forces segment replication type before enabling remote store (opensearch-project#4175) * Add validator that forces segment replication type before enabling remote store Signed-off-by: Sachin Kale <[email protected]> * [Remote Store] Change remote_store setting validation message to make it more clear (opensearch-project#4199) * Change remote_store setting validation message to make it more clear Signed-off-by: Sachin Kale <[email protected]> * [Remote Store] Add RemoteSegmentStoreDirectory to interact with remote segment store (opensearch-project#4020) * Add RemoteSegmentStoreDirectory to interact with remote segment store Signed-off-by: Sachin Kale <[email protected]> * Use RemoteSegmentStoreDirectory instead of RemoteDirectory (opensearch-project#4240) * Use RemoteSegmentStoreDirectory instead of RemoteDirectory Signed-off-by: Sachin Kale <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]>

…n. (opensearch-project#4225) (opensearch-project#4387) * Segment Replication. Fix Cancellation of replication events. This PR updates segment replication paths to correctly cancel replication events on the primary and replica. In the source service, any ongoing event for a primary that is sending to a replica that shuts down or is promoted as a new primary are cancelled. In the target service, any ongoing event for a replica that is promoted as a new primary or is fetching from a primary that shuts down. It wires up SegmentReplicationSourceService as an IndexEventListener so that it can respond to events and cancel any ongoing transfer state. This change also includes some test cleanup for segment replication to rely on actual components over mocks. Signed-off-by: Marc Handalian <[email protected]> Fix to not start/stop SegmentReplicationSourceService as a lifecycle component with feature flag off. Signed-off-by: Marc Handalian <[email protected]> Update logic to properly mark SegmentReplicationTarget as cancelled when cancel initiated by primary. Signed-off-by: Marc Handalian <[email protected]> Minor updates from self review. Signed-off-by: Marc Handalian <[email protected]> * Add missing changelog entry. Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Marc Handalian <[email protected]> (cherry picked from commit 19d1a2b)

…opensearch-project#4395) * Fix token usage for changelog helper Signed-off-by: Kunal Kotwani <[email protected]> * Add conditional check for dependabot steps Signed-off-by: Kunal Kotwani <[email protected]> * Add dependency section Signed-off-by: Kunal Kotwani <[email protected]> * Bug fixes for dependabot changelog verifier Signed-off-by: Kunal Kotwani <[email protected]> * Update the changelog Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> (cherry picked from commit 4a6e937)

… replica promotion (opensearch-project#4365) (opensearch-project#4397) * [Segment Replication] Bump segment infos counter before commit during replica promotion Signed-off-by: Suraj Singh <[email protected]> * Add changelog entry Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Suraj Singh <[email protected]>

…d primary on onNewCheckpoint on replica (opensearch-project#4363) (opensearch-project#4396) * [Segment Replication] Add check to cancel ongoing replication with old primary on onNewCheckpoint on replica Signed-off-by: Suraj Singh <[email protected]> * Add changelog entry Signed-off-by: Suraj Singh <[email protected]> * Address review comments Signed-off-by: Suraj Singh <[email protected]> * Address review comments 2 Signed-off-by: Suraj Singh <[email protected]> * Test failures Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Suraj Singh <[email protected]>

…search-project#4253) (opensearch-project#4418) * Add index specific setting for remote repository * Fix for failover incremental uploads Signed-off-by: Sachin Kale <[email protected]>

…uting metadata snapshot on primary shards. (opensearch-project#4366) (opensearch-project#4422) * Segment Replication - Fix NoSuchFileException errors caused when computing metadata snapshot on primary shards. This change fixes the errors that occur when computing metadata snapshots on primary shards from the latest in-memory SegmentInfos. The error occurs when a segments_N file that is referenced by the in-memory infos is deleted as part of a concurrent commit. The segments themselves are incref'd by IndexWriter.incRefDeleter but the commit file (Segments_N) is not. This change resolves this by ignoring the segments_N file when computing metadata for CopyState and only sending incref'd segment files to replicas. Signed-off-by: Marc Handalian <[email protected]> * Fix spotless. Signed-off-by: Marc Handalian <[email protected]> * Update StoreTests.testCleanupAndPreserveLatestCommitPoint to assert additional segments are deleted. Signed-off-by: Marc Handalian <[email protected]> * Rename snapshot to metadataMap in CheckpointInfoResponse. Signed-off-by: Marc Handalian <[email protected]> * Refactor segmentReplicationDiff method to compute off two maps instead of MetadataSnapshots. Signed-off-by: Marc Handalian <[email protected]> * Fix spotless. Signed-off-by: Marc Handalian <[email protected]> * Revert catchall in SegmentReplicationSourceService. Signed-off-by: Marc Handalian <[email protected]> * Revert log lvl change. Signed-off-by: Marc Handalian <[email protected]> * Fix SegmentReplicationTargetTests Signed-off-by: Marc Handalian <[email protected]> * Cleanup unused logger. Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Marc Handalian <[email protected]> Co-authored-by: Suraj Singh <[email protected]> Signed-off-by: Marc Handalian <[email protected]> Co-authored-by: Marc Handalian <[email protected]>

…port client (opensearch-project#4386) (opensearch-project#4424) * [Segment Replication] Extend FileChunkWriter to allow cancel on retryable transport client Signed-off-by: Suraj Singh <[email protected]> * Add changelog entry Signed-off-by: Suraj Singh <[email protected]> * Address review comments Signed-off-by: Suraj Singh <[email protected]> * Integration test Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Suraj Singh <[email protected]>

…ancelOngoingReplication unit test (opensearch-project#4414) (opensearch-project#4425) * [Semgnet Replication] Update flaky testOnNewCheckpointFromNewPrimaryCancelOngoingReplication unit test Signed-off-by: Suraj Singh <[email protected]> * Add changelog entry Signed-off-by: Suraj Singh <[email protected]> * Update changelog entry Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Suraj Singh <[email protected]>

* Revert to Netty 4.1.79.Final (opensearch-project#4428) Signed-off-by: Craig Perkins <[email protected]> Signed-off-by: Craig Perkins <[email protected]> (cherry picked from commit fb64a85) * Remove cherry-pick conflict in CHANGELOG.md Signed-off-by: Craig Perkins <[email protected]> * Remove netty-codec-http2-4.1.79.Final.jar.sha1 Signed-off-by: Craig Perkins <[email protected]> Signed-off-by: Craig Perkins <[email protected]>

…e needed to process getSegmentFiles. (opensearch-project#4434) * Fix timeout issue by calculating time needed to process getSegmentFiles. Signed-off-by: Rishikesh1159 <[email protected]> * Fix PR link in change log. Signed-off-by: Rishikesh1159 <[email protected]> * Addressing comments from PR. Signed-off-by: Rishikesh1159 <[email protected]> Signed-off-by: Rishikesh1159 <[email protected]>

…rch-project#4328) (opensearch-project#4442) * [Bug]: gradle check failing with java heap OutOfMemoryError Signed-off-by: Andriy Redko <[email protected]> * Fork JavaCompile task Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit ccf575a) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]>

* Add bwcVersion 1.3.6 to 2.x causing build failures Signed-off-by: Suraj Singh <[email protected]> * Add changelog entry Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Suraj Singh <[email protected]>

…egmentInfos instead of relying on segments_N from primary shards. (opensearch-project#4450) * Segment Replication - Fix NoSuchFileException errors caused when computing metadata snapshot on primary shards. (opensearch-project#4366) * Segment Replication - Fix NoSuchFileException errors caused when computing metadata snapshot on primary shards. This change fixes the errors that occur when computing metadata snapshots on primary shards from the latest in-memory SegmentInfos. The error occurs when a segments_N file that is referenced by the in-memory infos is deleted as part of a concurrent commit. The segments themselves are incref'd by IndexWriter.incRefDeleter but the commit file (Segments_N) is not. This change resolves this by ignoring the segments_N file when computing metadata for CopyState and only sending incref'd segment files to replicas. Signed-off-by: Marc Handalian <[email protected]> * Fix spotless. Signed-off-by: Marc Handalian <[email protected]> * Update StoreTests.testCleanupAndPreserveLatestCommitPoint to assert additional segments are deleted. Signed-off-by: Marc Handalian <[email protected]> * Rename snapshot to metadataMap in CheckpointInfoResponse. Signed-off-by: Marc Handalian <[email protected]> * Refactor segmentReplicationDiff method to compute off two maps instead of MetadataSnapshots. Signed-off-by: Marc Handalian <[email protected]> * Fix spotless. Signed-off-by: Marc Handalian <[email protected]> * Revert catchall in SegmentReplicationSourceService. Signed-off-by: Marc Handalian <[email protected]> * Revert log lvl change. Signed-off-by: Marc Handalian <[email protected]> * Fix SegmentReplicationTargetTests Signed-off-by: Marc Handalian <[email protected]> * Cleanup unused logger. Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Marc Handalian <[email protected]> Co-authored-by: Suraj Singh <[email protected]> * [Segment Replication] - Update replicas to commit SegmentInfos instead of relying on segments_N from primary shards. (opensearch-project#4402) * Segment Replication - Update replicas to commit SegmentInfos instead of relying on segments_N from primary shards. This change updates replicas to commit SegmentInfos before the shard is closed, on receiving a new commit point from a primary, and when a new primary is detected. This change also makes the public commitSegmentInfos on NRTEngine obsolete, refactoring IndexShard to simply call reset on the engine. Signed-off-by: Marc Handalian <[email protected]> * Remove noise & extra log statement. Signed-off-by: Marc Handalian <[email protected]> * PR feedback. Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Marc Handalian <[email protected]> * [Segment Replicaiton] Fix merge conflict. Update EngineTestCase.getTranslog() Signed-off-by: Suraj Singh <[email protected]> * Spotless check apply fixes Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Suraj Singh <[email protected]> Co-authored-by: Marc Handalian <[email protected]>

Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Suraj Singh <[email protected]>

* 2.3.0 release notes Signed-off-by: Suraj Singh <[email protected]> * Add changelog entry Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Suraj Singh <[email protected]>

… when install path includes a space. (opensearch-project#4362) (opensearch-project#4472) * Bugfix: Prevent escaping of in `else-if` statement by setting variable without double quotes. Signed-off-by: Mike Wilkinson <[email protected]> * Add changelog entry for fix Signed-off-by: Mike Wilkinson <[email protected]> * Escape double quotes for environment variables set by `opensearch-env.bat`. Explicitly apply quotes where those environment variables are invoked. Signed-off-by: Mike Wilkinson <[email protected]> Signed-off-by: Mike Wilkinson <[email protected]> Co-authored-by: Mike Wilkinson <[email protected]> Signed-off-by: Mike Wilkinson <[email protected]> Co-authored-by: mwilkinson-imo <[email protected]> Co-authored-by: Mike Wilkinson <[email protected]>

…ontext (opensearch-project#4484) (opensearch-project#4500) Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> Co-authored-by: Ketan Verma <[email protected]>

…mission' 'accessDeclaredMembers' when trying to get snapshot with S3 IRSA (opensearch-project#4469) (opensearch-project#4520) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Co-authored-by: Suraj Singh <[email protected]> (cherry picked from commit 8366ea3) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Co-authored-by: Suraj Singh <[email protected]>

* Added bwc version 2.3.1 * Add changelog of adding bwc version 2.3.1 Signed-off-by: Tianli Feng <[email protected]>

opensearch-project#4391) (opensearch-project#4421) * Adding support for labels on version bump PRs, skip label support for changelog verifier Signed-off-by: Kunal Kotwani <[email protected]> * Add changelog Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> (cherry picked from commit 0c10674)

…project#4563) * Updated jackson and snakeyml dependencies Signed-off-by: Owais Kazi <[email protected]> * Updated changelog Signed-off-by: Owais Kazi <[email protected]> * Update changelog with the 2.x PR number Signed-off-by: Owais Kazi <[email protected]> * Updated chaneglog Signed-off-by: Owais Kazi <[email protected]> Signed-off-by: Owais Kazi <[email protected]>

…e into log pattern of log4j property file (opensearch-project#4569) Signed-off-by: Tianli Feng <[email protected]>

…h-project#4494) (opensearch-project#4566) Fixes a bug to not fail the entire document when "ignore_malformed" is set to true. Allowing the valid fields to be indexed and ignore only the malformed fields. (cherry picked from commit 45c9597) Signed-off-by: Hauck <[email protected]>

…rch-project#4462) * add s390x support in systemcallfilter opensearch-project#4000 Signed-off-by: Andrija Pantovic <[email protected]> * add gradle cfg for s390x Signed-off-by: Andrija Pantovic <[email protected]> * change assertion for architecture Signed-off-by: Andrija Pantovic <[email protected]> * change assertion for architecture Signed-off-by: Andrija Pantovic <[email protected]> * update changelog Signed-off-by: Andrija Pantovic <[email protected]> Signed-off-by: Andrija Pantovic <[email protected]> Signed-off-by: Andrija Pantovic <[email protected]> (cherry picked from commit 517c19c) Co-authored-by: Andrija Pantovic <[email protected]>

opensearch-project#4642) Related commit: 0e9f74e Signed-off-by: Heemin Kim <[email protected]> (cherry picked from commit 2c0277b)

…etion (opensearch-project#4570) (opensearch-project#4670) * Fixed the SnapshotsInProgress error during index deletion (opensearch-project#4570) Signed-off-by: Xue Zhou <[email protected]> * Add changelog Signed-off-by: Xue Zhou <[email protected]> Signed-off-by: Xue Zhou <[email protected]>

…roject#4667) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]>

…ct#4647) * Updating hadoop-hdfs to 3.3.4 Signed-off-by: Sarat Vemulapalli <[email protected]> * Fixing typo Signed-off-by: Sarat Vemulapalli <[email protected]> Signed-off-by: Sarat Vemulapalli <[email protected]>

…4627) (opensearch-project#4678) * Fixed day of year defaulting for round up parser (opensearch-project#4627) Signed-off-by: Vishal Sarda <[email protected]> (cherry picked from commit a17692d)

…arch-project#4616) * Adding create pit service layer changes Signed-off-by: Bharathwaj G <[email protected]> * Adding delete pit service layer changes Signed-off-by: Bharathwaj G <[email protected]> * Add service layer changes for list all PITs Signed-off-by: Bharathwaj G <[email protected]> * Add Point In Time Node Stats API ServiceLayer Changes Signed-off-by: Bharathwaj G <[email protected]> * Add changes to Point in time segments API service layer Signed-off-by: Bharathwaj G <[email protected]> * Add changes for Create PIT and Delete PIT rest layer and rest high level client Signed-off-by: Bharathwaj G <[email protected]> * Added restlayer changes for pit stats Signed-off-by: Bharathwaj G <[email protected]> * 2.4 specific fixes Signed-off-by: Bharathwaj G <[email protected]> * Modified cat shards test for pit stats Signed-off-by: Bharathwaj G <[email protected]> * Added rest layer changes for List all PITs and PIT segments Signed-off-by: Bharathwaj G <[email protected]>

…nsearch-project#4651) * 496: Added filters, indexRouting and searchRouting when doing rollover on alias Signed-off-by: Stevan Buzejic <[email protected]> * 496: Removed unused map property in metada alias rollover tests Signed-off-by: Stevan Buzejic <[email protected]> Signed-off-by: Stevan Buzejic <[email protected]> Co-authored-by: Stevan Buzejic <[email protected]> (cherry picked from commit 54f8fdd) Co-authored-by: Stevan Buzejic <[email protected]>

Signed-off-by: Tianli Feng <[email protected]> Signed-off-by: Tianli Feng <[email protected]>

…arch-project#4716) Signed-off-by: Rishikesh1159 <[email protected]> Signed-off-by: Rishikesh1159 <[email protected]>

…etAdditionalRoles() (opensearch-project#4653) Signed-off-by: Tianli Feng <[email protected]>

) * Bump from 1.4.1 to 1.5.1 Signed-off-by: Rishikesh1159 <[email protected]> * Update change log with correct PR link Signed-off-by: Rishikesh1159 <[email protected]> Signed-off-by: Rishikesh1159 <[email protected]>

…ase (opensearch-project#4729) This field is only used for old snapshot generations and needs to be removed in the next major version. This commit sets up the removal for backwards compatibility support. Signed-off-by: Nicholas Walter Knize <[email protected]>

…ability (opensearch-project#4689) (opensearch-project#4739) Signed-off-by: Tianli Feng <[email protected]> Signed-off-by: Tianli Feng <[email protected]>

…rch-project#4156) and Further simplification of the ZIP publication implementation (opensearch-project#4360) (opensearch-project#4733) * This is a backport of opensearch-project#4156 and opensearch-project#4360 to 2.x * opensearch-project#4156 was cherry-picked from 7fe5830 - Resolving conflicts in CHANGELOG.md * opensearch-project#4360 was cherry-picked from ab6849f - Resolving conflicts in CHANGELOG.md There is one noticeable difference, the `project.group` property can be empty in which case it will fall back to default value: `org.opensearch.plugin`. See <opensearch-project#4156 (comment)> for more details. Signed-off-by: Lukáš Vlček <[email protected]> Signed-off-by: Lukáš Vlček <[email protected]>

…xx tasks (opensearch-project#4742) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]>

…project#4219) Sets analyzer to regex query string search Signed-off-by: yogev mets <[email protected]> (cherry picked from commit ea4cfcc) Co-authored-by: Yogev Mets <[email protected]> Co-authored-by: Nicholas Walter Knize <[email protected]>

…arch-project#4780) * Introduce experimental searchable snapshot API (opensearch-project#4680) This commit adds a new parameter to the snapshot restore API to restore to a new type of "remote snapshot" index where, unlike traditional snapshot restores, the index data is not all downloaded to disk and instead is read on-demand at search time. The feature is functional with this commit, and includes a simple end-to-end integration test, but is far from complete. See tracking issue opensearch-project#2919 for the rest of the work planned/underway. All new capabilities are gated behind a new searchable snapshot feature flag. Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Andrew Ross <[email protected]> (cherry picked from commit 431bdeb) * Fix assertion in test of ShardStateMetadata (opensearch-project#4774) This test randomly fails if the enum selected isn't the same as the one choosen by the legacy constructor. Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Andrew Ross <[email protected]> (cherry picked from commit 1a7aaba) * Update version check for 2.4 Checking on or after version 3.0 does not work (or make sense) since this is on the 2.x branch. Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Andrew Ross <[email protected]>

…search-project#4779) (opensearch-project#4781) * Update Jackson Databind to 2.13.4.2 (addressing CVE-2022-42003) (opensearch-project#4779) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit 12f26d3) * Update changelog Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> Co-authored-by: Andriy Redko <[email protected]>

…oject#4754) (opensearch-project#4758) Adds log4j-jul dependency (along with license and notice files) and configures the java.util.logging log4j adapater to route all JUL logs to the proper opensearch locations setup through the LogConfigurator for Lucene 9.4 compatibility. Signed-off-by: Nicholas Walter Knize <[email protected]> (cherry picked from commit beb9915)

…t#4828) (opensearch-project#4833) Disables merge on refresh for DiskThresholdDeciderIT.testRestoreSnapshotAllocationDoesNotExceedWatermark. Signed-off-by: Nicholas Walter Knize <[email protected]> (cherry picked from commit 49363fb)

…pensearch-project#4834) Drops the trailing "e" in taxes, dresses, watches, dishes etc that otherwise cause mismatches with plural and singular forms. Signed-off-by: Nicholas Walter Knize <[email protected]> Co-authored-by: Mark Harwood <[email protected]> Co-authored-by: Nicholas Walter Knize <[email protected]> (cherry picked from commit c92846d)

…h-project#4761) (opensearch-project#4818) * Refactor BalancedAllocator.Balancer to LocalShardsBalancer (opensearch-project#4761) * Refactor BalancedAllocator.Balancer to LocalShardsBalancer Signed-off-by: Kunal Kotwani <[email protected]> * Deprecate Balancer to maintain BWC Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> (cherry picked from commit 1d65485) * Update changelog Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]>

…ository-hdfs. (opensearch-project#4854) * Backport 4790 to 2.x Signed-off-by: Ryan Bogan <[email protected]> * Fixed merge conflicts Signed-off-by: Ryan Bogan <[email protected]> * Removed unused SHA Signed-off-by: Ryan Bogan <[email protected]> Signed-off-by: Ryan Bogan <[email protected]> Co-authored-by: Marc Handalian <[email protected]>

…or TotalHits lower bound (opensearch-project#4867) * Fix AbstractStringFieldDataTestCase tests to account for TotalHits lower bound (opensearch-project#4270) Fixes tests to account for TotalHits uncertainty as of Lucene 9. Signed-off-by: Daniel Widdis <[email protected]> (cherry picked from commit 4643620) * Added CHANGELOG. Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> Co-authored-by: Daniel Widdis <[email protected]> Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]>

This feature was backported in opensearch-project#4219 and will only be present in version 2.4 and newer. The test must be skipped for mixed cluster cases with nodes older than 2.4. Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Andrew Ross <[email protected]>

…ject#4158) (opensearch-project#4857) * Introduce Remote translog feature flag (opensearch-project#4158) * Introduce Remote translog feature flag * Limit combinations of remote segment and translog store to valid ones Signed-off-by: Bukhtawar Khan <[email protected]> (cherry picked from commit a469a3c) * Add CHANGELOG entry Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: Bukhtawar Khan <[email protected]>

…rch-project#4848) This is back-port of opensearch-project#4772 Signed-off-by: Lukáš Vlček <[email protected]> (cherry picked from commit eb33015)

…-project#4597) (opensearch-project#4842) * Support of GeoJson Point for GeoPoint field (opensearch-project#4597) * Support of GeoJson Point for GeoPoint field See opensearch-project/geospatial#152 Signed-off-by: Heemin Kim <[email protected]> (cherry picked from commit a282d39) * Run geojson yaml test for 2.4 and above only Signed-off-by: Heemin Kim <[email protected]> Signed-off-by: Heemin Kim <[email protected]>

…ster 1.x (opensearch-project#4812) (opensearch-project#4875) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit d85f8cd) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]>

…h-project#4884) * add missing no-jdk gradle targets for distributions by convention all supported platforms should also have a gradle target to produce a no-jdk build and that target should be called `no-jdk-[platform]`. this is a first step in the path to publishing the no-jdk distributions as part of the release, as there are use-cases where a specific JDK should be used instead of the packaged one and it's cleaner to use a small distribution which doesn't bring along an unneeded JDK (the JDK greatly increases the resulting distribution since). note that i failed to rename the `linux-s390x` target, it seems that there's too much generic build code out there which tries to stitch together that name and then fails if it doesn't find it. this is part of opensearch-build#99. Signed-off-by: Ralph Ursprung <[email protected]> * add basic README for `:distribution` this is primarily to document the JDK & No JDK setup (to codify it for the future), however the README can be extended in the future to document more information which is currently hidden away only somewhere in the code. Signed-off-by: Ralph Ursprung <[email protected]> Signed-off-by: Ralph Ursprung <[email protected]> (cherry picked from commit dfa1118)

…r remote translog enabled indices (opensearch-project#4318) (opensearch-project#4873) * [Remote Store] Change behaviour in replica recovery for remote translog enabled indices (opensearch-project#4318) Signed-off-by: Ashish Singh <[email protected]> * Fixes ReplicaRecoveryWithRemoteTranslogOnPrimaryTests Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Ashish Singh <[email protected]>

…uster (opensearch-project#4832) * Backport 4787 to 2.x Signed-off-by: Ryan Bogan <[email protected]> * Fixed merge conflicts Signed-off-by: Ryan Bogan <[email protected]> * Fixed Changelog Signed-off-by: Ryan Bogan <[email protected]> * Fixed Changelog Signed-off-by: Ryan Bogan <[email protected]> * Fixed merge conflicts Signed-off-by: Ryan Bogan <[email protected]> * Addressed more merge conflicts Signed-off-by: Ryan Bogan <[email protected]> Signed-off-by: Ryan Bogan <[email protected]> Co-authored-by: Marc Handalian <[email protected]>

…pensearch-project#4895) Replicas are bootstrapped using the recovery path, as opposed to the restore path used for creating the primary shard. This has been broken in the initial implementation of searchable snapshots. The fix here is to put in the appropriate checks to avoid failing during recovery. I've also updated the integration test to ensure the recovery path is always exercised during testing. Signed-off-by: Andrew Ross <[email protected]> (cherry picked from commit 6ac0c7c)

Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]>

Signed-off-by: Craig Perkins <[email protected]> Signed-off-by: Craig Perkins <[email protected]>

…pensearch-project#4891) the build script should be located in this repository instead of centrally in [opensearch-build][] to allow per-branch specific builds (newer versions might have newer requirements, which currently cannot be reflected). the file was copied 1:1 from `scripts/components/OpenSearch/build.sh` as of commit 9e1aa38 in the [opensearch-build][] repository. this is part of opensearch-build#99 and needs to be backported to all active branches because the file here will only be used by the build once the central one in opensearch-build has been removed which in turn can only happen when all active branches have the `build.sh`. [opensearch-build]: https://github.com/opensearch-project/opensearch-build Signed-off-by: Ralph Ursprung <[email protected]> Signed-off-by: Ralph Ursprung <[email protected]> (cherry picked from commit e357246) Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> Signed-off-by: Ralph Ursprung <[email protected]> Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> Co-authored-by: Ralph Ursprung <[email protected]>

…ject#4898) Signed-off-by: Ryan Bogan <[email protected]> Signed-off-by: Ryan Bogan <[email protected]>

…ensearch-project#4900) (opensearch-project#4917) With this commit, appropriate exception is thrown when an array of four values are provided for point type field. Signed-off-by: Heemin Kim <[email protected]> (cherry picked from commit 0f477a2)

…ch-project#4921) Change access modifier from default to protected. This will help to build new geo based aggregation outside OpenSearch, by depending on GeoGrid as base class. Signed-off-by: Vijayan Balasubramanian <[email protected]> Signed-off-by: Vijayan Balasubramanian <[email protected]>

…nsearch-project#4889) (opensearch-project#4910) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]>

…project#4791) (opensearch-project#4794) * Bump Tika from 2.4.0 to 2.5.0 addressing CVE-2022-33879. Signed-off-by: Marc Handalian <[email protected]> * Add missing SHAs. Signed-off-by: Marc Handalian <[email protected]> * Update changelog with PR info. Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Marc Handalian <[email protected]> (cherry picked from commit f1995b9) Signed-off-by: Vacha Shah <[email protected]> Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Vacha Shah <[email protected]>

…opensearch-project#4935) Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]>

…roject#4934) Signed-off-by: Andriy Redko <[email protected]>

* Upgrade woodstox (opensearch-project#4947) * Bump woodstox-core to 6.4.0 Signed-off-by: Vacha Shah <[email protected]> * Update CHANGELOG Signed-off-by: Vacha Shah <[email protected]> Signed-off-by: Vacha Shah <[email protected]> * Update CHANGELOG for 2.x Signed-off-by: Vacha Shah <[email protected]> * Remove unused SHAs Signed-off-by: Vacha Shah <[email protected]> Signed-off-by: Vacha Shah <[email protected]>

…arch-project#4764) (opensearch-project#4958) This commit adds a new cluster block that is added by default to all searchable snapshot indexes to block writes and metadata changes. (cherry picked from commit c4fc1bc) Signed-off-by: Vishal Sarda <[email protected]> Co-authored-by: Vishal Sarda <[email protected]>

…or windows gradle check (opensearch-project#4941) (opensearch-project#4961) * Fix for failing checkExtraction and checkLicense tasks Signed-off-by: Poojita Raj <[email protected]> * changelog added Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Poojita Raj <[email protected]>

…log not purging on index (opensearch-project#4975) * Fix bug of replica shard's translog not purging on index flush and fixing related unit tests Signed-off-by: Rishikesh1159 <[email protected]> * Update change log Signed-off-by: Rishikesh1159 <[email protected]> Signed-off-by: Rishikesh1159 <[email protected]>

…tor (opensearch-project#4695) (opensearch-project#4953) * Adding feature to exclude indexes starting with dot from shard validator Signed-off-by: Nishchay Malhotra <[email protected]> (cherry picked from commit 04eb817)

…ct#4902) (opensearch-project#4993) with commit e357246 the `build.sh` has been moved to this repository and with commit dfa1118 the missing no-jdk builds have been added. with these preliminary works done we can now start building the no-jdk distributions as part of the release build so that they can then be published. this is part of opensearch-project/opensearch-build#99. Signed-off-by: Ralph Ursprung <[email protected]> Signed-off-by: Ralph Ursprung <[email protected]> Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]> (cherry picked from commit 8c9ca4e)

…-project#4982) * Fix dependencies (opensearch-project#4963) * Upgrading kotlin-stdlib and excluding jetty-http Signed-off-by: Vacha Shah <[email protected]> * Upgrading indirect snakeyaml dependency Signed-off-by: Vacha Shah <[email protected]> * Update CHANGELOG Signed-off-by: Vacha Shah <[email protected]> * Using snakeyaml version from version file Signed-off-by: Vacha Shah <[email protected]> * Upgrading jetty-http instead of excluding since it is used Signed-off-by: Vacha Shah <[email protected]> * Extracting kotlin version Signed-off-by: Vacha Shah <[email protected]> Signed-off-by: Vacha Shah <[email protected]> * Update CHANGELOG Signed-off-by: Vacha Shah <[email protected]> Signed-off-by: Vacha Shah <[email protected]>

* Bumping 2.x to 2.5.0 Signed-off-by: Vacha Shah <[email protected]> * Update CHANGELOG Signed-off-by: Vacha Shah <[email protected]> Signed-off-by: Vacha Shah <[email protected]>

Bumps [protobuf-java](https://github.com/protocolbuffers/protobuf) from 3.21.7 to 3.21.8. - [Release notes](https://github.com/protocolbuffers/protobuf/releases) - [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/generate_changelog.py) - [Commits](protocolbuffers/protobuf@v3.21.7...v3.21.8) --- updated-dependencies: - dependency-name: com.google.protobuf:protobuf-java dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Vacha Shah <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

opensearch-project#5009) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit 34b6b09) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…t#4821) (opensearch-project#5010) Signed-off-by: Sergey Nuyanzin <[email protected]> Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]> (cherry picked from commit 763ef13) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Andriy Redko <[email protected]>

…pensearch-project#5024) * Add remote allocator for searchable snapshots (opensearch-project#4870) Signed-off-by: Kunal Kotwani <[email protected]> Part of this commit was developed from code and concepts initially implemented in Amazon OpenSearch Service as part of the UltraWarm feature. Thank you to the following developers and the entire UltraWarm team. Co-authored-by: Min Zhou <[email protected]> Co-authored-by: Ankit Malpani <[email protected]> Co-authored-by: Rohit Nair <[email protected]> Co-authored-by: Sorabh Hamirwasia <[email protected]> Co-authored-by: Ankit Jain <[email protected]> Co-authored-by: Tianru Zhou <[email protected]> Co-authored-by: Neetika Singhal <[email protected]> Co-authored-by: Amit Khandelwal <[email protected]> Co-authored-by: Vigya Sharma <[email protected]> Co-authored-by: Prateek Sharma <[email protected]> Co-authored-by: Venkata Jyothsna Donapati <[email protected]> Co-authored-by: Vlad Rozov <[email protected]> Co-authored-by: Mohit Agrawal <[email protected]> Co-authored-by: Shweta Thareja <[email protected]> Co-authored-by: Palash Hedau <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Piyush Daftary <[email protected]> Co-authored-by: Payal Maheshwari <[email protected]> Co-authored-by: Kunal Khatua <[email protected]> Co-authored-by: Gulshan Kumar <[email protected]> Co-authored-by: Rushi Agrawal <[email protected]> Co-authored-by: Ketan Verma <[email protected]> Co-authored-by: Gaurav Chandani <[email protected]> Co-authored-by: Dharmesh Singh <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> (cherry picked from commit 9119104) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md * Update changelog Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Kunal Kotwani <[email protected]>

* Add DecommissionService and helper to execute awareness attribute decommissioning opensearch-project#4084 * Add APIs (GET/PUT) to decommission awareness attribute opensearch-project#4261 * Controlling discovery for decommissioned nodes opensearch-project#4590 * Fix decommission status update to non leader nodes opensearch-project#4800 * Remove redundant field from GetDecommissionStateResponse opensearch-project#4751 * Service Layer changes for Recommission API opensearch-project#4320 * Recommission api level support opensearch-project#4604 * Fix bug in AwarenessAttributeDecommissionIT opensearch-project#4822 Signed-off-by: Rishab Nahata <[email protected]>

…-project#5023) (opensearch-project#5025) Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Marc Handalian <[email protected]> (cherry picked from commit 0adb74d) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ch-project#4892) (opensearch-project#5042) Part of this commit was developed from code and concepts initially implemented in Amazon OpenSearch Service as part of the UltraWarm feature. Thank you to the following developers and the entire UltraWarm team. Co-authored-by: Min Zhou <[email protected]> Co-authored-by: Ankit Malpani <[email protected]> Co-authored-by: Rohit Nair <[email protected]> Co-authored-by: Sorabh Hamirwasia <[email protected]> Co-authored-by: Ankit Jain <[email protected]> Co-authored-by: Tianru Zhou <[email protected]> Co-authored-by: Neetika Singhal <[email protected]> Co-authored-by: Amit Khandelwal <[email protected]> Co-authored-by: Vigya Sharma <[email protected]> Co-authored-by: Prateek Sharma <[email protected]> Co-authored-by: Venkata Jyothsna Donapati <[email protected]> Co-authored-by: Vlad Rozov <[email protected]> Co-authored-by: Mohit Agrawal <[email protected]> Co-authored-by: Shweta Thareja <[email protected]> Co-authored-by: Palash Hedau <[email protected]> Co-authored-by: Saurabh Singh <[email protected]> Co-authored-by: Piyush Daftary <[email protected]> Co-authored-by: Payal Maheshwari <[email protected]> Co-authored-by: Kunal Khatua <[email protected]> Co-authored-by: Gulshan Kumar <[email protected]> Co-authored-by: Rushi Agrawal <[email protected]> Co-authored-by: Ketan Verma <[email protected]> Co-authored-by: Gaurav Chandani <[email protected]> Co-authored-by: Dharmesh Singh <[email protected]> Signed-off-by: Ahmad AbuKhalil <[email protected]> (cherry picked from commit 1466350) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <[email protected]>

… (opensearch-project#5045) Per the contract of InputStream#read(byte[], int, int): If len is zero, then no bytes are read and 0 is returned SlicedInputStream had a bug where if a zero length was passed then it would drain all the underlying streams and return -1. This was uncovered by using InputStream#readAllBytes in new code under development. The only existing usage of SlicedInputStream should not be vulnerable to this bug. I've also added a check for invalid arguments and created tests to ensure the proper exceptions are thrown per the InputStream contract. In the test I've replaced a "readFully" method with an equivalent "readNBytes" that was introduced in Java 11. Signed-off-by: Andrew Ross <[email protected]> (cherry picked from commit 6571db7) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <[email protected]>

…t#5007) (opensearch-project#5047) Signed-off-by: Vacha Shah <[email protected]>

* Fixing PIT flaky tests Signed-off-by: Bharathwaj G <[email protected]> (cherry picked from commit 0c80a7a) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <[email protected]>

…ource consumption (opensearch-project#5039) * [Backport 2.x] Added in-flight cancellation of SearchShardTask based on resource consumption (opensearch-project#4575) This feature aims to identify and cancel resource intensive SearchShardTasks if they have breached certain thresholds. This will help in terminating problematic queries which can put nodes in duress and degrade the cluster performance. * [Backport 2.x] Added resource usage trackers for in-flight cancellation of SearchShardTask (opensearch-project#4805) 1. CpuUsageTracker: cancels tasks if they consume too much CPU 2. ElapsedTimeTracker: cancels tasks if they consume too much time 3. HeapUsageTracker: cancels tasks if they consume too much heap * [Backport 2.x]Added search backpressure stats API Added search backpressure stats to the existing node/stats API to describe: 1. the number of cancellations (currently for SearchShardTask only) 2. the current state of TaskResourceUsageTracker Signed-off-by: Ketan Verma <[email protected]>

* Weighted round-robin scheduling policy for shard coordination traffic… (opensearch-project#4241) * Add PUT api to update shard routing weights (opensearch-project#4272) * Add GET api to get shard routing weights (opensearch-project#4275) * Fix weighted routing metadata deserialization error during node restart (opensearch-project#4691) * Delete API for weighted round robin search routing (opensearch-project#4400) * Mark apis experimental Signed-off-by: Anshu Agarwal <[email protected]>

…nt enforcement (opensearch-project#5053) * Add AutoExpandReplica in the validation of replica count enforcement Signed-off-by: Arpit Bandejiya <[email protected]>

* Fail weight update when decommission ongoing and fail decommission when attribute not weighed away (opensearch-project#4839) * Add changes for graceful node decommission (opensearch-project#4586) * Add delay timeout for decommission request (opensearch-project#4931) * Integ Tests for Awareness Attribute Decommissioning (opensearch-project#4715) Signed-off-by: Rishab Nahata <[email protected]> Signed-off-by: pranikum <[email protected]>

…on found' when detailed_error disabled (opensearch-project#4708) * Fixed misunderstanding message 'No OpenSearchException found' when detailed_error disabled Signed-off-by: Xue Zhou <[email protected]>

* Add basic thorttler/exponential backoff policy for retry/Defination of throttling exception (opensearch-project#3856) * Corrected Java doc for Throttler * Changed the default behaviour of Throttler to return Optional * Removed generics from Throttler and used String as key * Ignore backport / autocut / dependabot branches for gradle checks on push * Master node changes for master task throttling (opensearch-project#3882) * Data node changes for master task throttling (opensearch-project#4204) * Onboarding of few task types to throttling (opensearch-project#4542) * Fix timeout exception and Add Integ test for Master task throttling (opensearch-project#4588) * Complete TODO for version change and removed unused classes(Throttler and Semaphore) (opensearch-project#4846) * Remove V1 version from throttling testcase Signed-off-by: Dhwanil Patel <[email protected]>

…t#5077) (opensearch-project#5079) Signed-off-by: Marc Handalian <[email protected]> Add Changelog entry. Signed-off-by: Marc Handalian <[email protected]> Revert unintentional removal of memory assertion. Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Marc Handalian <[email protected]> Co-authored-by: Marc Handalian <[email protected]>

) (opensearch-project#5080) Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> (cherry picked from commit 657d559) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…nsearch-project#5086) Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> (cherry picked from commit 082f059) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

The 2.x CHANGELOG is now empty, given that the 2.4 release was created and all 2.4 changes have moved to that version's release notes file. This also brings in the 2.4 release notes. Signed-off-by: Andrew Ross <[email protected]>

…rch-project#5117) Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> (cherry picked from commit 3ffa46a) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…t#5107) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit 75d5ed5) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…arch-project#5138) Signed-off-by: dblock <[email protected]> Signed-off-by: dblock <[email protected]>

…s. (opensearch-project#5142) * Fix error handling while reading analyzer mapping rules Add new parseWordList method that takes a parser as a parameter. It reads custom rules from settings or a file, parses and handles errors. Make error messages less verbose for rules files outside config directory. Signed-off-by: Rabi Panda <[email protected]> * Fix: raise error on malformed CSV. Signed-off-by: dblock <[email protected]> Signed-off-by: Rabi Panda <[email protected]> Signed-off-by: dblock <[email protected]> Co-authored-by: Rabi Panda <[email protected]>

…pensearch-project#4978) Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> (cherry picked from commit 171d141) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…rch-project#5154) This also removes a Jackson dependency update from the changelog because it was backported all the way back to 2.4 so therefore is not unreleased here any more. Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Andrew Ross <[email protected]>

…e cause chain (opensearch-project#5081) (opensearch-project#5084) Signed-off-by: Xue Zhou <[email protected]> (cherry picked from commit d479853)

…hrottlingIT is flaky. (opensearch-project#5153) * Fix: org.opensearch.clustermanager.ClusterManagerTaskThrottlingIT is flaky. (opensearch-project#5152) Signed-off-by: dblock <[email protected]> Signed-off-by: dblock <[email protected]> (cherry picked from commit 88757b4) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Fix race condition in testThrottlingForSingleNode. Signed-off-by: dblock <[email protected]> Signed-off-by: dblock <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: dblock <[email protected]>

…search-project#4743) (opensearch-project#5116) * alreadyClosedExceptionFix Signed-off-by: Poojita Raj <[email protected]> * adding changelog entry Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Poojita Raj <[email protected]> (cherry picked from commit 37d1eba) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…5168) (opensearch-project#5179) This updates the boundary condition in an assertion in two tests in ShardIndexingPressureConcurrentExecutionTests. I could reliably reproduce errors here by running: ``` ./gradlew ':server:test' -Dtests.iters=10000 --tests "org.opensearch.index.ShardIndexingPressureConcurrentExecutionTests.testReplicaThreadedUpdateToShardLimits" ``` On every error the value that failed was exactly 0.95 and failed the less than check. The change here is to accept 0.95, and also refactor the test to give a better error message on failure. Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Andrew Ross <[email protected]> (cherry picked from commit 3423f44) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…h-project#5178) [QA: None] Signed-off-by: Xiao Cui <[email protected]> Signed-off-by: Xiao Cui <[email protected]> (cherry picked from commit dcb80bc) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Xiao Cui <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…n/tools/plugin-cli (opensearch-project#5148) * Bump bcpg-fips from 1.0.5.1 to 1.0.7.1 in /distribution/tools/plugin-cli (opensearch-project#5102) * Bump bcpg-fips from 1.0.5.1 to 1.0.7.1 in /distribution/tools/plugin-cli Bumps bcpg-fips from 1.0.5.1 to 1.0.7.1. --- updated-dependencies: - dependency-name: org.bouncycastle:bcpg-fips dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit 5c1d084) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md * add changelog Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Poojita Raj <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Poojita Raj <[email protected]>

…pensearch-project#5188) (opensearch-project#5191) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit 650039c) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…gest-attachment (opensearch-project#5183) * Bump commons-compress from 1.21 to 1.22 in /plugins/ingest-attachment (opensearch-project#5104) * Bump commons-compress from 1.21 to 1.22 in /plugins/ingest-attachment Bumps commons-compress from 1.21 to 1.22. --- updated-dependencies: - dependency-name: org.apache.commons:commons-compress dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit fadf409) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md * add changelog Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Poojita Raj <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Poojita Raj <[email protected]>

…project#5103) (opensearch-project#5201) * Bump geoip2 from 3.0.1 to 3.0.2 in /modules/ingest-geoip Bumps [geoip2](https://github.com/maxmind/GeoIP2-java) from 3.0.1 to 3.0.2. - [Release notes](https://github.com/maxmind/GeoIP2-java/releases) - [Changelog](https://github.com/maxmind/GeoIP2-java/blob/main/CHANGELOG.md) - [Commits](maxmind/GeoIP2-java@v3.0.1...v3.0.2) --- updated-dependencies: - dependency-name: com.maxmind.geoip2:geoip2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * modify changelog Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Poojita Raj <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Poojita Raj <[email protected]> (cherry picked from commit de2b84d) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…t#4683) (opensearch-project#4688) (opensearch-project#5206) This test fails pretty reliably if I run it on repeat. I believe the problem is that the test assumes the function will take longer than 2ms, which is likely not a valid assumption in all cases. Fortunately, I can pass in a zero duration which is guaranteed to timeout even if the system clock does not advance at all. Also moved the assertions out of the callback into the main test method, otherwise the assertion error messages would get buried and the test report would just show a timeout error. Signed-off-by: Andrew Ross <[email protected]>

…ensearch-project#4544) (opensearch-project#5207) Signed-off-by: Daniel Widdis <[email protected]> Signed-off-by: Daniel Widdis <[email protected]> (cherry picked from commit 7ebb2af) Co-authored-by: Daniel Widdis <[email protected]>

…earch-project#5204) The updated changelog process is to include changes intended to be backported into the `[Unreleased 2.x]` section of the changelog on main. It should be backported to the 2.x branch so the CHANGELOG.md file should not be skipped. Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Andrew Ross <[email protected]> (cherry picked from commit f0925f7) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…t#5158) (opensearch-project#5165) The test used an absolute path which is incorrect as it made a platform dependent assumption. This change fixes that. Signed-off-by: Rabi Panda <[email protected]>

…t#5246) (opensearch-project#5260) The change made in AbstractPointGeometryFieldMapper class with commit 0503897 introduced a performace degradation during point data indexing. Reverting it therefore. The change is about consolidating array format parsing code for point type in a single place to acheive following benefits. 1. Allow plugins to override array parsing logic. Plugins can add its own parsing logic for point field by providing object parser. However, it cannot override array format. Therefore, plugin have to use whatever implemented in AbstractPointGeometryFieldMapper class. 2. Enhanced code quality by removing duplicated code There is no change in functionality because 1. There is no change in functionality in OpenSearch and 2. No plugins have its own parsing logic for point data in array format yet. Signed-off-by: Heemin Kim <[email protected]> Signed-off-by: Heemin Kim <[email protected]> (cherry picked from commit 3583b80) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Heemin Kim <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…5252) (opensearch-project#5261) * Pick non decommissioned node for client requests Signed-off-by: Rishab Nahata <[email protected]> * Fix spotless check Signed-off-by: Rishab Nahata <[email protected]> * Fix Signed-off-by: Rishab Nahata <[email protected]> * Empty-Commit Signed-off-by: Rishab Nahata <[email protected]> * Empty-Commit Signed-off-by: Rishab Nahata <[email protected]> * Empty-Commit Signed-off-by: Rishab Nahata <[email protected]> * Empty-Commit Signed-off-by: Rishab Nahata <[email protected]> * Fix Signed-off-by: Rishab Nahata <[email protected]> * Empty-Commit Signed-off-by: Rishab Nahata <[email protected]> * Empty-Commit Signed-off-by: Rishab Nahata <[email protected]> Signed-off-by: Rishab Nahata <[email protected]> (cherry picked from commit 3c02b43) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Rishab Nahata <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…t. (opensearch-project#5266) * Display test failures in the gradle check status report. (opensearch-project#5200) * Display test failures in the gradle check status report. Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> * Add counts of failures. Signed-off-by: dblock <[email protected]> * Limit to 10 failures. Signed-off-by: dblock <[email protected]> Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> Signed-off-by: dblock <[email protected]> (cherry picked from commit 51991e0) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Quote failures. Signed-off-by: dblock <[email protected]> Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> Signed-off-by: dblock <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: dblock <[email protected]>

…#5225) (opensearch-project#5241) The large data tables necessitating this exclusion were refactored away a long time ago. Signed-off-by: Andrew Ross <[email protected]> (cherry picked from commit 671820d) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…earch-project#5281) As per gradle [docs] add support to remove timestamps and package with same order which is required from [reproducible] builds This is a result of the discussion in opensearch-project/opensearch-plugin-template-java#38 to apply for all plugins going forward. [docs]: https://docs.gradle.org/current/userguide/working_with_files.html#sec:reproducible_archives [reproducible]: https://reproducible-builds.org/ Signed-off-by: Leonidas Spyropoulos <[email protected]> Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]> (cherry picked from commit e44158d)

…ect#5278) (opensearch-project#5286) The `sendBodyAsSourceParam` method determines whether it should send the body as query parameter. It used to check the content length of the entity which in some cases (e.g. regex expression) is much shorter than the encoded-url and may result in a TooLongHttpLineException. This commit makes an additional check on the length of the url-encoded string. To avoid unnecessary computation it is only checked when the RandomizedTest.rarely() returns true. Signed-off-by: Rabi Panda <[email protected]>

…#5285) (opensearch-project#5292) Based on the seed, the recursive createTestInstance method generates too many aggregations resulting in an exception. This change reduces the number of generated aggregations. Signed-off-by: Rabi Panda <[email protected]> (cherry picked from commit 1bb29e7) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Rabi Panda <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Co-authored-by: opensearch-ci-bot <[email protected]>

…pensearch-project#5298) Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Andrew Ross <[email protected]> (cherry picked from commit c3b4266) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Signed-off-by: Ryan Bogan <[email protected]> Signed-off-by: Ryan Bogan <[email protected]>

…-project#5300) Signed-off-by: dblock <[email protected]> Signed-off-by: dblock <[email protected]>

…tests (opensearch-project#5262) (opensearch-project#5296) * Make viaNode more deterministic fixing flaky AwarenessAttributeDecommissionIT tests Signed-off-by: Rishab Nahata <[email protected]> (cherry picked from commit 059b614) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Rishab Nahata <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…edForOneZone failure (opensearch-project#5289) (opensearch-project#5309) Signed-off-by: Anshu Agarwal <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…fs (opensearch-project#5184) * Bump gson from 2.9.0 to 2.10 in /plugins/repository-hdfs (opensearch-project#4948) * Bump gson from 2.9.0 to 2.10 in /plugins/repository-hdfs Bumps [gson](https://github.com/google/gson) from 2.9.0 to 2.10. - [Release notes](https://github.com/google/gson/releases) - [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md) - [Commits](google/gson@gson-parent-2.9.0...gson-parent-2.10) --- updated-dependencies: - dependency-name: com.google.code.gson:gson dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit 38d510c) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md * Add changelog entry Signed-off-by: Kartik Ganesh <[email protected]> Signed-off-by: Kartik Ganesh <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Kartik Ganesh <[email protected]>

… (opensearch-project#5326) * Bump joni from 2.1.43 to 2.1.44 in /libs/grok Bumps [joni](https://github.com/jruby/joni) from 2.1.43 to 2.1.44. - [Release notes](https://github.com/jruby/joni/releases) - [Commits](jruby/joni@joni-2.1.43...joni-2.1.44) --- updated-dependencies: - dependency-name: org.jruby.joni:joni dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit 55f218c) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…re (opensearch-project#4996) (opensearch-project#5185) * Bump protobuf-java from 3.21.8 to 3.21.9 in /test/fixtures/hdfs-fixture Bumps [protobuf-java](https://github.com/protocolbuffers/protobuf) from 3.21.8 to 3.21.9. - [Release notes](https://github.com/protocolbuffers/protobuf/releases) - [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/generate_changelog.py) - [Commits](protocolbuffers/protobuf@v3.21.8...v3.21.9) --- updated-dependencies: - dependency-name: com.google.protobuf:protobuf-java dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit 7cbf86a) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ch-project#5330) * Bump gradle-extra-configurations-plugin from 7.0.0 to 8.0.0 (opensearch-project#4808) * Bump gradle-extra-configurations-plugin from 7.0.0 to 8.0.0 Bumps [gradle-extra-configurations-plugin](https://github.com/nebula-plugins/gradle-extra-configurations-plugin) from 7.0.0 to 8.0.0. - [Release notes](https://github.com/nebula-plugins/gradle-extra-configurations-plugin/releases) - [Changelog](https://github.com/nebula-plugins/gradle-extra-configurations-plugin/blob/main/CHANGELOG.md) - [Commits](nebula-plugins/gradle-extra-configurations-plugin@v7.0.0...v8.0.0) --- updated-dependencies: - dependency-name: com.netflix.nebula:gradle-extra-configurations-plugin dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> * add changelog Signed-off-by: Poojita Raj <[email protected]> * add changelog Signed-off-by: Poojita Raj <[email protected]> * Fix fully-qualified plugin name after version bump Signed-off-by: Kartik Ganesh <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Kartik Ganesh <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Poojita Raj <[email protected]> Co-authored-by: Kartik Ganesh <[email protected]> (cherry picked from commit a0d3073) * Bump jcodings from 1.0.57 to 1.0.58 in /libs/grok (opensearch-project#5233) * Bump jcodings from 1.0.57 to 1.0.58 in /libs/grok Bumps [jcodings](https://github.com/jruby/jcodings) from 1.0.57 to 1.0.58. - [Release notes](https://github.com/jruby/jcodings/releases) - [Commits](jruby/jcodings@jcodings-1.0.57...jcodings-1.0.58) --- updated-dependencies: - dependency-name: org.jruby.jcodings:jcodings dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit cbb0953) * Bump google-http-client-jackson2 from 1.35.0 to 1.42.3 in /plugins/repository-gcs (opensearch-project#5234) * Bump google-http-client-jackson2 in /plugins/repository-gcs Bumps [google-http-client-jackson2](https://github.com/googleapis/google-http-java-client) from 1.35.0 to 1.42.3. - [Release notes](https://github.com/googleapis/google-http-java-client/releases) - [Changelog](https://github.com/googleapis/google-http-java-client/blob/main/CHANGELOG.md) - [Commits](googleapis/google-http-java-client@v1.35.0...v1.42.3) --- updated-dependencies: - dependency-name: com.google.http-client:google-http-client-jackson2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit 933e8c3) * Bump azure-core from 1.33.0 to 1.34.0 and azure-core-http-netty from 1.12.4 to 1.12.7 in /plugins/repository-azure (opensearch-project#5235) * Bump azure-core from 1.33.0 to 1.34.0 This change also reverts the ignoreMissingClasses addition from the prior commit since it is incorrect. Signed-off-by: Kartik Ganesh <[email protected]> * Bump azure-core-http-netty in /plugins/repository-azure Bumps [azure-core-http-netty](https://github.com/Azure/azure-sdk-for-java) from 1.12.4 to 1.12.7. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-core-http-netty_1.12.4...azure-core-http-netty_1.12.7) --- updated-dependencies: - dependency-name: com.azure:azure-core-http-netty dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Kartik Ganesh <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Kartik Ganesh <[email protected]> (cherry picked from commit 0c6411a) * Bump maxmind-db from 2.0.0 to 2.1.0 in /modules/ingest-geoip (opensearch-project#5236) * Bump maxmind-db from 2.0.0 to 2.1.0 in /modules/ingest-geoip Bumps [maxmind-db](https://github.com/maxmind/MaxMind-DB-Reader-java) from 2.0.0 to 2.1.0. - [Release notes](https://github.com/maxmind/MaxMind-DB-Reader-java/releases) - [Changelog](https://github.com/maxmind/MaxMind-DB-Reader-java/blob/main/CHANGELOG.md) - [Commits](maxmind/MaxMind-DB-Reader-java@v2.0.0...v2.1.0) --- updated-dependencies: - dependency-name: com.maxmind.db:maxmind-db dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit 7dc137f) * Bump json-schema-validator from 1.0.69 to 1.0.73 (opensearch-project#5316) * Bump json-schema-validator from 1.0.69 to 1.0.73 Bumps [json-schema-validator](https://github.com/networknt/json-schema-validator) from 1.0.69 to 1.0.73. - [Release notes](https://github.com/networknt/json-schema-validator/releases) - [Changelog](https://github.com/networknt/json-schema-validator/blob/master/CHANGELOG.md) - [Commits](networknt/json-schema-validator@1.0.69...1.0.73) --- updated-dependencies: - dependency-name: com.networknt:json-schema-validator dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> * Fix changelog Signed-off-by: Kartik Ganesh <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Kartik Ganesh <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Kartik Ganesh <[email protected]> (cherry picked from commit 0f666ef) * Bump proto-google-common-protos from 2.8.0 to 2.10.0 in /plugins/repository-gcs (opensearch-project#5318) * Bump proto-google-common-protos in /plugins/repository-gcs Bumps [proto-google-common-protos](https://github.com/googleapis/java-iam) from 2.8.0 to 2.10.0. - [Release notes](https://github.com/googleapis/java-iam/releases) - [Changelog](https://github.com/googleapis/java-iam/blob/main/CHANGELOG.md) - [Commits](https://github.com/googleapis/java-iam/commits) --- updated-dependencies: - dependency-name: com.google.api.grpc:proto-google-common-protos dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Add changelog entry Signed-off-by: Kartik Ganesh <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Kartik Ganesh <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Kartik Ganesh <[email protected]> (cherry picked from commit 0f78b9f) * Bump protobuf-java from 3.21.7 to 3.21.9 in /plugins/repository-hdfs (opensearch-project#5319) * Bump protobuf-java from 3.21.7 to 3.21.9 in /plugins/repository-hdfs Bumps [protobuf-java](https://github.com/protocolbuffers/protobuf) from 3.21.7 to 3.21.9. - [Release notes](https://github.com/protocolbuffers/protobuf/releases) - [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/generate_changelog.py) - [Commits](protocolbuffers/protobuf@v3.21.7...v3.21.9) --- updated-dependencies: - dependency-name: com.google.protobuf:protobuf-java dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Add changelog entry Signed-off-by: Kartik Ganesh <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Kartik Ganesh <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Kartik Ganesh <[email protected]> (cherry picked from commit 438369c) * Updates to get a successful build SHA updates and Gradle build file updates Signed-off-by: Kartik Ganesh <[email protected]> Signed-off-by: Kartik Ganesh <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…T/test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials} flaky (opensearch-project#5325) (opensearch-project#5335) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit a1ceff4) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…pensearch-project#5340) The randomly-generated input for this test may contain non-URL-safe characters (such as `%`) so it needs to be URL encoded as well. Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Andrew Ross <[email protected]> (cherry picked from commit 152f3f4) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…5308) (opensearch-project#5343) Signed-off-by: PritLadani <[email protected]> (cherry picked from commit bdb8efe) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: PritLadani <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]>

Prevent deletion of snapshots that are backing searchable snapshot indexes Signed-off-by: Vishal Sarda <[email protected]> (cherry picked from commit 08ac17f) Signed-off-by: Vishal Sarda <[email protected]>

…t#5368) (opensearch-project#5372) Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: Andrew Ross <[email protected]>

Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Andrew Ross <[email protected]>

Signed-off-by: Andriy Redko <[email protected]>

…pensearch-project#5384) The existing bulk api silently ignores bulk item requests that have an invalid action. This change rejects those requests. Signed-off-by: Rabi Panda <[email protected]> Signed-off-by: Rabi Panda <[email protected]>

…t#5355) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit fd495a6) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…t#5393) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]>

…) (opensearch-project#5402) * Fix flaky test testReplicaReceivesGenIncrease. This fixes a race condition in segment replication specific test code that waited for a replication event to complete using a CountDownLatch. If the latch completes before the actual replication event the replica state is unpredictable. This changeis the latch's wait limit from 1 to 30 seconds. Signed-off-by: Marc Handalian <[email protected]> * Fix replicateSegments when there is more than one replica. Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Marc Handalian <[email protected]> (cherry picked from commit 9e50e3f) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…n Windows. (opensearch-project#5397) (opensearch-project#5403) Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Marc Handalian <[email protected]> (cherry picked from commit d2d87d9) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…opensearch-project#5415) When the new 'cancelled' field was introduced it was a miss not to increment the version number on the mapping definitions for the .tasks index. This commit fixes that oversight, as well as modifies the existing backward compatiblity test to ensure that it will catch future mistakes like this one. Closes opensearch-project#5376 Signed-off-by: Andrew Ross <[email protected]> (cherry picked from commit 4616dfa) Signed-off-by: Andrew Ross <[email protected]>

…ject#5283) (opensearch-project#5420) Signed-off-by: Rishab Nahata <[email protected]>

…search-project#5399) (opensearch-project#5410) (cherry picked from commit 0210b76) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…roject#5439) (opensearch-project#5450) Add conditional check on assertNull to fix flaky tests. Signed-off-by: Rishikesh1159 <[email protected]> (cherry picked from commit 0bd3141) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Rishikesh1159 <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

) Signed-off-by: Dhwanil Patel <[email protected]> Signed-off-by: Dhwanil Patel <[email protected]>

…he Mime4j 0.8.8, Apache Poi 5.2.3, Apache PdfBox 2.0.27 (opensearch-project#5448) (opensearch-project#5468) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit 5500114) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…T/test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials} flaky: randomizing basePath (opensearch-project#5482) (opensearch-project#5483) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit 2416d37) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

) (opensearch-project#5493) Fixes the wildcard query to not normalize the pattern when case_insensitive is set by the user. This is achieved by creating a new normalizedWildcardQuery method so that query_string queries (which do not support case sensitivity) can still normalize the pattern when the default analyzer is used; maintaining existing behavior. Signed-off-by: Nicholas Walter Knize <[email protected]> (cherry picked from commit ce25dec) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Nicholas Walter Knize <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ch-project#4902)" (opensearch-project#5466) This reverts commit 8c9ca4e. It seems that this wasn't entirely the correct way and is currently blocking us from removing the `build.sh` from the `opensearch-build` repository (i.e. this `build.sh` here is not yet being used). See the discussion in opensearch-project/opensearch-build#2835 for further details. Signed-off-by: Ralph Ursprung <[email protected]> Signed-off-by: Ralph Ursprung <[email protected]>

…opensearch-project#5498) * Add max_shard_size parameter for Shrink API Signed-off-by: Gao Binlong <[email protected]> * add change log Signed-off-by: Gao Binlong <[email protected]> * fix yaml test failed Signed-off-by: Gao Binlong <[email protected]> * optimize the code Signed-off-by: Gao Binlong <[email protected]> * fix test failed Signed-off-by: Gao Binlong <[email protected]> * optimize changelog & code Signed-off-by: Gao Binlong <[email protected]> Signed-off-by: Gao Binlong <[email protected]> (cherry picked from commit 953a3d6)

…ject#5460) (opensearch-project#5499) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]>

…ch-project#5507) Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> (cherry picked from commit 0f651b8) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ze on Windows. (opensearch-project#5511) (opensearch-project#5512)

…arch-project#5491) (opensearch-project#5506) Signed-off-by: Poojita Raj <[email protected]> (cherry picked from commit 7108ee5)

Signed-off-by: Ryan Bogan <[email protected]> Signed-off-by: Ryan Bogan <[email protected]>

Signed-off-by: Rishikesh1159 <[email protected]> Signed-off-by: Rishikesh1159 <[email protected]>

…ject#5510) Signed-off-by: Xue Zhou <[email protected]> Signed-off-by: Xue Zhou <[email protected]> (cherry picked from commit 67977a2) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Xue Zhou <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…s and corresponding action (opensearch-project#5541) * [Backport-2.x] Atomically update cluster state with decommission status and corresponding action (opensearch-project#5093) * Atomically update the cluster state with decommission status and its corresponding action in the same execute call Signed-off-by: Rishab Nahata <[email protected]>

…earch-project#4955) (opensearch-project#5539) * [Backport 2.x]Pre conditions check to allow weight updates for non decommissioned attribute Signed-off-by: Rishab Nahata <[email protected]>

Signed-off-by: Ryan Bogan <[email protected]> Signed-off-by: Ryan Bogan <[email protected]>

) (opensearch-project#5544) * Adding support to register settings dynamically Signed-off-by: Ryan Bogan <[email protected]> * Update CHANGELOG Signed-off-by: Ryan Bogan <[email protected]> * Removed unnecessary registerSetting methods Signed-off-by: Ryan Bogan <[email protected]> * Change setting registration order Signed-off-by: Ryan Bogan <[email protected]> * Add unregisterSettings method Signed-off-by: Ryan Bogan <[email protected]> * Remove unnecessary feature flag Signed-off-by: Ryan Bogan <[email protected]> Signed-off-by: Ryan Bogan <[email protected]> (cherry picked from commit 08cd06f) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Ryan Bogan <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ensearch-project#5523) Bumps commons-compress from 1.21 to 1.22. --- updated-dependencies: - dependency-name: org.apache.commons:commons-compress dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit bceb40c) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…pensearch-project#5551) Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Suraj Singh <[email protected]> (cherry picked from commit ec5144b) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ect#5555) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit 1710786) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Added bwc version 2.4.2 Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> * Add 1.3.8 bwc version Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> Signed-off-by: Suraj Singh <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]>

…) (opensearch-project#5584) * Refactor Object to Fuzziness type for all query builders Signed-off-by: noCharger <[email protected]> * Revise on bwc Signed-off-by: noCharger <[email protected]> * Update change log Signed-off-by: noCharger <[email protected]> Signed-off-by: noCharger <[email protected]> Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]> (cherry picked from commit d3f6dfa) Signed-off-by: Louis Chu <[email protected]> Signed-off-by: noCharger <[email protected]> Signed-off-by: Louis Chu <[email protected]> Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]>

* Backporting auto release workflow to 2.x Signed-off-by: Sarat Vemulapalli <[email protected]> * Adding Changelog Signed-off-by: Sarat Vemulapalli <[email protected]> * Adding Changelog Signed-off-by: Sarat Vemulapalli <[email protected]> Signed-off-by: Sarat Vemulapalli <[email protected]>

…ransport Actions for extensions (opensearch-project#5596) Signed-off-by: Ryan Bogan <[email protected]> Signed-off-by: Ryan Bogan <[email protected]>

…ngs (opensearch-project#5598) Signed-off-by: Dhwanil Patel <[email protected]>

…arch-project#5573) * Add CI bundle pattern to distribution download (opensearch-project#5348) * Add CI bundle pattern for ivy repo Signed-off-by: Zelin Hao <[email protected]> * Gradle update Signed-off-by: Zelin Hao <[email protected]> * Extract path Signed-off-by: Zelin Hao <[email protected]> * Change with customDistributionDownloadType Signed-off-by: Zelin Hao <[email protected]> * Add default for exception handle Signed-off-by: Zelin Hao <[email protected]> * Add documentations Signed-off-by: Zelin Hao <[email protected]> Signed-off-by: Zelin Hao <[email protected]> (cherry picked from commit fe8fd67) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Reorder logics to fix tests Signed-off-by: Zelin Hao <[email protected]> Signed-off-by: Zelin Hao <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Zelin Hao <[email protected]>

…pensearch-project#5617) (opensearch-project#5619) Signed-off-by: Dhwanil Patel <[email protected]> (cherry picked from commit 9c03010) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…createComponents support for extensions (opensearch-project#5621) Signed-off-by: Ryan Bogan <[email protected]> Signed-off-by: Ryan Bogan <[email protected]>

…nsions (opensearch-project#5629) Signed-off-by: Ryan Bogan <[email protected]> Signed-off-by: Ryan Bogan <[email protected]>

…opensearch-project#5654) Signed-off-by: Daniel Widdis <[email protected]> (cherry picked from commit cc2b704) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Daniel Widdis <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

) (opensearch-project#5653) Signed-off-by: Daniel Widdis <[email protected]> (cherry picked from commit 30ac284) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Daniel Widdis <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…rch-project#5656) (opensearch-project#5657) * Add waiting time to account for Thread.sleep inaccuracy Signed-off-by: Daniel Widdis <[email protected]> (cherry picked from commit d248643) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Daniel Widdis <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

… setting (opensearch-project#5633) (opensearch-project#5647) * Add version check during task submission for bwc for static threshold setting Signed-off-by: Dhwanil Patel <[email protected]> (cherry picked from commit ea1cc9d)

…search-project#5683) Standardize snapshot indices parsing so that combinations of included and excluded indices are treated the same regardless of the order they are listed in. Signed-off-by: Stephen Crawford <[email protected]> (cherry picked from commit 57d4485) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Stephen Crawford <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Signed-off-by: Rishikesh1159 <[email protected]> Signed-off-by: Rishikesh1159 <[email protected]>

…ch-project#5684) * Bump gradle-info-plugin from 11.3.3 to 12.0.0 Signed-off-by: Rishikesh1159 <[email protected]> * Update correct dependency version in changelog Signed-off-by: Rishikesh1159 <[email protected]> Signed-off-by: Rishikesh1159 <[email protected]>

@gbbafna

…project#5668) (opensearch-project#5690) * Adding @gbbafna to Opensearch maintainers Signed-off-by: Bukhtawar Khan <[email protected]>

…ts API (opensearch-project#5691) * Validate attributes of routing nodes for Routing Weights API (opensearch-project#5607) * Changes following things in Routing Weights API and Decommission Action - a. Consider only routing node's attribute for validation instead of all nodes b. Ensure no more than 50% node has weight zero c. Removal of decommission action's direct dependence on force zone awareness Signed-off-by: Rishab Nahata <[email protected]>

@mch2

…shots (opensearch-project#5429) * Enhance searchable snapshots to enable a read-only view of older snapshots This change removes the guardrails around N-1 backward compatibility and uses Lucene's "expert" APIs to read snapshots (Lucene segments) older than N-1 on a best-effort basis. The functionality is gated by an additional feature flag, separate from the searchable snapshots flag. Note that the Lucene integration is rather inefficient because the necessary "expert" Lucene APIs are still package-private. Signed-off-by: Kartik Ganesh <[email protected]> * Added some unit tests This change also includes a test index ZIP file for the unit tests. The change also introduces a bug fix in the readAnySegmentsInfo method to close the reader before returning the SegmentInfos object - this avoids dangling/open file handles. Signed-off-by: Kartik Ganesh <[email protected]> * Incorporating PR feedback Signed-off-by: Kartik Ganesh <[email protected]> * Incorporate PR comments from andrross Signed-off-by: Kartik Ganesh <[email protected]> * Remove use of IndexSetting for minimum version for snapshots backwards compatibility Signed-off-by: Kartik Ganesh <[email protected]> * Moved ES 6.3.0 test data to a subdirectory This change also includes an update to the file name to clarify that it is an ES index, and changing the associated markdown file name to just README.md. All tests that reference this ZIP file have corresponding changes to the path they reference. Signed-off-by: Kartik Ganesh <[email protected]> * Update unit tests to use try-with-resources Signed-off-by: Kartik Ganesh <[email protected]> * Added FeatureFlagSetter helper class Also refactored unit test classes to use the helper class. Signed-off-by: Kartik Ganesh <[email protected]> * Incorporating PR feedback from @mch2 Signed-off-by: Kartik Ganesh <[email protected]> * Fix IndexSettingsTests Updated the asserts in IndexSettingsTests to account for the new defaulting behavior. Signed-off-by: Kartik Ganesh <[email protected]> Signed-off-by: Kartik Ganesh <[email protected]>

…opensearch-project#5732) *[Backport 2.x] Add default replica count config settings Signed-off-by: Arpit Bandejiya <[email protected]>

Signed-off-by: Sachin Kale <[email protected]>

…rch-project#4480) (opensearch-project#5693) * [Backport 2.x] Introduce remote translog transfer support Signed-off-by: Bukhtawar Khan <[email protected]>

…project#5725) Signed-off-by: Gaurav Bafna <[email protected]>

…pensearch-project#5731) * Remove PRRL creation/deletion in peer recovery of remote store enabled replica (opensearch-project#4954) * Add RecoverySourceHandlerFactory for extensibility Signed-off-by: Ashish Singh <[email protected]> * recoverToTarget made extensible to allow multiple implementations Signed-off-by: Ashish Singh <[email protected]> * Remove PRRL after SendFileStep in Peer Recovery Signed-off-by: Ashish Singh <[email protected]> * Incorporate PR review feedback Signed-off-by: Ashish Singh <[email protected]> * Empty-Commit Signed-off-by: Ashish Singh <[email protected]> * Incorporate PR review feedback Signed-off-by: Ashish Singh <[email protected]> * Empty-Commit Signed-off-by: Ashish Singh <[email protected]> * Empty-Commit Signed-off-by: Ashish Singh <[email protected]> * Remove CHANGELOG entry as this is incremental PR Signed-off-by: Ashish Singh <[email protected]> * Incorporate PR review feedback Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Ashish Singh <[email protected]> * Enhance CheckpointState to support no-op replication (opensearch-project#5282) * CheckpointState enhanced to support no-op replication Signed-off-by: Ashish Singh <[email protected]> Co-authored-by: Bukhtawar Khan<[email protected]> Signed-off-by: Ashish Singh <[email protected]> * Add transport action for primary term validation for remote-backed indices (opensearch-project#5616) Add transport action for primary term validation for remote-backed indices Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Ashish Singh <[email protected]>

…pensearch-project#5231) (opensearch-project#5688) * Adding core changes for API to check per awareness attribute health. Signed-off-by: Nishchay Malhotra <[email protected]>

…hanges (opensearch-project#5757) * Introduce TranslogFactory for Local/Remote Translog support (opensearch-project#4172) * Introduce TranslogFactory for Local/Remote Translog support Signed-off-by: Bukhtawar Khan <[email protected]> * [Remote Translog] Introduce remote translog with upload functionality (opensearch-project#5392) * Introduce remote translog with upload functionality Signed-off-by: Gaurav Bafna <[email protected]> Co-authored-by: Bukhtawar Khan <[email protected]> * Enable creation of indices using Remote Translog (opensearch-project#5638) * Enable creation of indices using Remote Translog behind a setting and feature flag Signed-off-by: Gaurav Bafna <[email protected]> * [Remote Translog] Add support for downloading files from remote translog (opensearch-project#5649) * Add support to download translog from remote store during recovery Signed-off-by: Sachin Kale <[email protected]> * Integrate remote translog download on failover (opensearch-project#5699) * Integrate remote translog download on failover Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Bukhtawar Khan <[email protected]> Signed-off-by: Gaurav Bafna <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Signed-off-by: Ashish Singh <[email protected]>

…h-project#5758) * fix graph filter out of bound error Signed-off-by: Mingshi Liu <[email protected]> * add changelog Signed-off-by: Mingshi Liu <[email protected]> * run gradle spotlessApply Signed-off-by: Mingshi Liu <[email protected]> * reproduce error in unit test Signed-off-by: Mingshi Liu <[email protected]> * format to pass spotlessApply Signed-off-by: Mingshi Liu <[email protected]> * organize package Signed-off-by: Mingshi Liu <[email protected]> Signed-off-by: Mingshi Liu <[email protected]> (cherry picked from commit 6a7a9a1) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Mingshi Liu <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…pensearch-project#5733) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]>

… weights in the response (opensearch-project#5680) * Add support for discovered master and remove local weights in the weighted routing API response Signed-off-by: Anshu Agarwal <[email protected]>

* Support API versioning for weighted shard routing Signed-off-by: Anshu Agarwal <[email protected]>

…pensearch-project#5072) * Fail open requests on search shard failures ( Signed-off-by: Anshu Agarwal <[email protected]>

[Backport 2.x] Add support for API versioning and fail open in weighted shard routing #5779

[Backport 2.x] Add support for API versioning and fail open in weighted shard routing #5779

Commits on Jul 30, 2022

Commits on Aug 1, 2022

Commits on Aug 2, 2022

Commits on Aug 3, 2022

Commits on Aug 4, 2022

Commits on Aug 5, 2022

Commits on Aug 8, 2022

Commits on Aug 9, 2022

Commits on Aug 11, 2022

Commits on Aug 15, 2022

Commits on Aug 16, 2022

Commits on Aug 17, 2022

Commits on Aug 18, 2022

Commits on Aug 24, 2022

Commits on Aug 25, 2022

Commits on Aug 26, 2022

Commits on Aug 27, 2022

Commits on Aug 30, 2022

Commits on Aug 31, 2022

Commits on Sep 1, 2022

Commits on Sep 2, 2022

Commits on Sep 3, 2022

Commits on Sep 5, 2022

Commits on Sep 6, 2022

Commits on Sep 7, 2022

Commits on Sep 8, 2022

Commits on Sep 9, 2022

Commits on Sep 14, 2022

Commits on Sep 15, 2022

Commits on Sep 16, 2022

Commits on Sep 20, 2022

Commits on Sep 22, 2022

Commits on Sep 23, 2022

Commits on Sep 27, 2022

Commits on Sep 30, 2022

Commits on Oct 4, 2022

Commits on Oct 5, 2022

Commits on Oct 6, 2022

Commits on Oct 7, 2022

Commits on Oct 10, 2022

Commits on Oct 11, 2022

Commits on Oct 13, 2022

Commits on Oct 17, 2022

Commits on Oct 18, 2022

Commits on Oct 19, 2022

Commits on Oct 20, 2022

Commits on Oct 21, 2022

Commits on Oct 24, 2022

Commits on Oct 25, 2022

Commits on Oct 26, 2022

Commits on Oct 27, 2022

Commits on Oct 28, 2022

Commits on Oct 29, 2022

Commits on Oct 31, 2022

Commits on Nov 1, 2022

Commits on Nov 2, 2022

Commits on Nov 3, 2022

Commits on Nov 4, 2022

Commits on Nov 7, 2022

Commits on Nov 8, 2022

Commits on Nov 9, 2022

Commits on Nov 10, 2022

Commits on Nov 11, 2022

Commits on Nov 15, 2022

Commits on Nov 16, 2022

Commits on Nov 17, 2022

Commits on Nov 18, 2022

Commits on Nov 19, 2022

Commits on Nov 22, 2022

Commits on Nov 23, 2022

Commits on Nov 25, 2022

Commits on Nov 26, 2022

Commits on Nov 28, 2022

Commits on Nov 30, 2022

Commits on Dec 1, 2022

Commits on Dec 5, 2022

Commits on Dec 6, 2022

Commits on Dec 7, 2022