[Weighted Shard Routing] Fail open requests on search shard failures #5072

anshu1106 · 2022-11-04T09:22:53Z

Description

Fail open shard copies request to go to the nodes in weighed away zone. This helps in reducing the number of 5xx responses as well as better shard availability. In other case, where shard search requests return 4xx due to throttling from shard copies in non-weighed away zone, we don't need to fail open. So basically if the request fails due to internal issues in the cluster we fail open for search requests.

Issues Resolved

#4735

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2022-11-04T10:50:30Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/6353/
CommitID: e64c59e
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2022-11-09T14:19:50Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/6672/
CommitID: d0015fb
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Anshu Agarwal <[email protected]>

github-actions · 2022-11-09T15:35:44Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/6675/
CommitID: 02f1fa8
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Anshu Agarwal <[email protected]>

github-actions · 2022-11-10T04:20:40Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/6733/
CommitID: 9e69711
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Anshu Agarwal <[email protected]>

github-actions · 2022-11-10T05:46:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/6737/
CommitID: c0e5b67
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Anshu Agarwal <[email protected]>

github-actions · 2022-11-10T12:11:24Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/6745/
CommitID: f3006a5
Please examine the workflow log, locate, and copy-paste the failure below, then iterate to green.
Is the failure a flaky test unrelated to your change?

gbbafna

With this change, we will also affect the output of search_shards api which will now return all the shards including weight 0 . This might be acceptable behavior, but worth calling out.

server/src/main/java/org/opensearch/cluster/routing/IndexShardRoutingTable.java

server/src/main/java/org/opensearch/cluster/routing/WeightedRoutingHelper.java

gbbafna · 2022-12-05T13:24:57Z

server/src/main/java/org/opensearch/action/fieldcaps/TransportFieldCapabilitiesIndexAction.java

+            // This checks if the shard is present in data node with weighted routing weight set to 0,
+            // In such cases we fail open, if shard search request for the shard from other shard copies fail with non
+            // retryable exception.


A note : we should test out cross cluster search works with this as well .

Signed-off-by: Anshu Agarwal <[email protected]>

github-actions · 2022-12-06T10:43:23Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/7643/
CommitID: 2834af8
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Anshu Agarwal <[email protected]>

Bukhtawar · 2023-01-09T16:08:34Z

server/src/main/java/org/opensearch/cluster/routing/IndexShardRoutingTable.java

@@ -352,8 +354,8 @@ public ShardIterator activeInitializingShardsWeightedIt(
                logger.debug("no shard copies found for shard id [{}] for node attribute with weight zero", shardId);
            }
        }
-
-        return new PlainShardIterator(shardId, ordered);
+        orderedListWithDistinctShards = new ArrayList<>(new LinkedHashSet<>(ordered));


Suggest

ordered.stream().distinct().collect(Collectors.toList())

Bukhtawar · 2023-01-09T16:12:50Z

server/src/main/java/org/opensearch/cluster/routing/FailAwareWeightedRouting.java

+    public FailAwareWeightedRouting(Exception e, ClusterState clusterState) {
+        this.exception = e;
+        this.clusterState = clusterState;
+    }


Please use only cluster state in the constructor so that it could be used as a singleton instance and move exception to the method findNext and log the same exception

I think we can't make FailAwareWeightedRouting singleton since even cluster state can change between requests

Yeah bad edit, I initially meant ClusterService for singleton but forgot to remove.

Got it. I have made FailAwareWeightedRouting singleton but we can't pass ClusterService in the constructor since ClusterService instance is not available in AbstractSearchAsyncAction.

Bukhtawar · 2023-01-09T16:13:06Z

server/src/main/java/org/opensearch/cluster/routing/FailAwareWeightedRouting.java

+    }
+
+    private void logFailOpen(ShardId shardID) {
+        logger.info(() -> new ParameterizedMessage("{}: Fail open executed", shardID));


nit: will prefer this inline

Bukhtawar · 2023-01-09T16:24:17Z

server/src/main/java/org/opensearch/cluster/routing/FailAwareWeightedRouting.java

+        RestStatus.INTERNAL_SERVER_ERROR,
+        RestStatus.NOT_IMPLEMENTED,
+        RestStatus.BAD_GATEWAY,
+        RestStatus.SERVICE_UNAVAILABLE,
+        RestStatus.GATEWAY_TIMEOUT,
+        RestStatus.HTTP_VERSION_NOT_SUPPORTED,
+        RestStatus.INSUFFICIENT_STORAGE
+    );


Please remove

RestStatus.NOT_IMPLEMENTED

RestStatus.HTTP_VERSION_NOT_SUPPORTED

RestStatus.INSUFFICIENT_STORAGE

github-actions · 2023-01-09T16:24:40Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9082/
CommitID: f376eab
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Anshu Agarwal <[email protected]>

github-actions · 2023-01-09T17:22:19Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9089/
CommitID: ef961bd
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-09T17:40:01Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9090/
CommitID: 97ea739
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-09T17:40:58Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9086/
CommitID: 33a04b5
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Bukhtawar · 2023-01-09T17:56:23Z

Test failures : #5766

Signed-off-by: Anshu Agarwal <[email protected]>

github-actions · 2023-01-10T04:02:30Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9125/
CommitID: 7df7358
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Bukhtawar · 2023-01-10T06:43:36Z

server/src/main/java/org/opensearch/cluster/routing/FailAwareWeightedRouting.java

+public enum FailAwareWeightedRouting {
+    INSTANCE;
+
+    private static final Logger logger = LogManager.getLogger(FailAwareWeightedRouting.class);
+
+    private final static List<RestStatus> internalErrorRestStatusList = List.of(
+        RestStatus.INTERNAL_SERVER_ERROR,
+        RestStatus.BAD_GATEWAY,
+        RestStatus.SERVICE_UNAVAILABLE,
+        RestStatus.GATEWAY_TIMEOUT
+    );
+
+    public static FailAwareWeightedRouting getInstance() {
+        return INSTANCE;
+
+    }


I don't think this is the most apt way to create singletons, you could achieve it by a static property

public static final FailAwareWeightedRouting INSTANCE = new FailAwareWeightedRouting();

Bukhtawar · 2023-01-10T06:44:15Z

server/src/main/java/org/opensearch/cluster/routing/FailAwareWeightedRouting.java

+        while (next != null && isWeighedAway(next.currentNodeId(), clusterState)) {
+            ShardRouting nextShard = next;
+            if (canFailOpen(nextShard.shardId(), exception, clusterState)) {
+                logger.info(() -> new ParameterizedMessage("{}: Fail open executed due to exception {}", nextShard.shardId(), exception));


nitL incorrect usage of logger with exception

…pensearch-project#5072) * Fail open requests on search shard failures ( Signed-off-by: Anshu Agarwal <[email protected]>

#5784) * [Weighted Routing] Add support for discovered master and remove local weights in the response (#5680) * Add support for discovered master and remove local weights in the weighted routing API response Signed-off-by: Anshu Agarwal <[email protected]> * [Weighted Shard Routing] API versioning (#5255) * Support API versioning for weighted shard routing Signed-off-by: Anshu Agarwal <[email protected]> * [Weighted Shard Routing] Fail open requests on search shard failures (#5072) * Fail open requests on search shard failures ( Signed-off-by: Anshu Agarwal <[email protected]> * Address fail open comments (#5778) [Weighted Shard Routing] Refactor and fix singleton in FailAwareWeightedRouting Signed-off-by: Anshu Agarwal <[email protected]> * remove unintended changes in changelog Signed-off-by: Anshu Agarwal <[email protected]> * remove unintended changes from changelog Signed-off-by: Anshu Agarwal <[email protected]> Signed-off-by: Anshu Agarwal <[email protected]> Co-authored-by: Anshu Agarwal <[email protected]>

…ed shard routing (#5781) * [Backport 2.x] [Weighted Shard Routing] Add support for discovered master and remove local weights in the response #5680 [Weighted Shard Routing] API versioning #5255 [Weighted Shard Routing] Fail open requests on search shard failures #5072 [Weighted Shard Routing] Refactor and fix singleton in FailAwareWeightedRouting #5778 Signed-off-by: Anshu Agarwal <[email protected]>

anshu1106 requested review from a team and reta as code owners November 4, 2022 09:22

anshu1106 marked this pull request as draft November 4, 2022 09:23

anshu1106 changed the title ~~[Draft] Weighted Routing fail open~~ [Weighted Shard Routing] fail open on errors Nov 4, 2022

Fail open changes

02f1fa8

Signed-off-by: Anshu Agarwal <[email protected]>

anshu1106 force-pushed the feature/poc-fail-open branch from d0015fb to 02f1fa8 Compare November 9, 2022 15:05

Refactor code

9e69711

Signed-off-by: Anshu Agarwal <[email protected]>

Anshu Agarwal added 2 commits November 10, 2022 10:16

Merge branch 'main' into feature/poc-fail-open

5ee2b02

Fix test

c0e5b67

Signed-off-by: Anshu Agarwal <[email protected]>

Add integ test with network disruption and refactor code

f3006a5

Signed-off-by: Anshu Agarwal <[email protected]>

gbbafna reviewed Dec 6, 2022

View reviewed changes

Anshu Agarwal added 2 commits December 6, 2022 14:58

Merge branch 'main' into feature/poc-fail-open

683786d

Add log statement

2834af8

Signed-off-by: Anshu Agarwal <[email protected]>

anshu1106 marked this pull request as ready for review December 6, 2022 11:50

Add integ test for search aggregations and flag open flag

4ad2bed

Signed-off-by: Anshu Agarwal <[email protected]>

anshu1106 requested review from anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, setiah, kartg and kotwanikunal as code owners December 20, 2022 05:53

fix test comment

33a04b5

Signed-off-by: Anshu Agarwal <[email protected]>

anshu1106 force-pushed the feature/poc-fail-open branch from 9a13bc9 to 33a04b5 Compare January 9, 2023 15:47

Bukhtawar reviewed Jan 9, 2023

View reviewed changes

Anshu Agarwal added 2 commits January 9, 2023 22:21

Address review comments

ef961bd

Signed-off-by: Anshu Agarwal <[email protected]>

log exception

97ea739

Signed-off-by: Anshu Agarwal <[email protected]>

Address review comments

7df7358

Signed-off-by: Anshu Agarwal <[email protected]>

Bukhtawar reviewed Jan 10, 2023

View reviewed changes

Bukhtawar approved these changes Jan 10, 2023

View reviewed changes

Bukhtawar merged commit e3b5015 into opensearch-project:main Jan 10, 2023

anshu1106 mentioned this pull request Jan 10, 2023

[Weighted Shard Routing] Refactor and fix singleton in FailAwareWeightedRouting #5778

Merged

6 tasks

anshu1106 mentioned this pull request Jan 10, 2023

[Backport 2.x] Add support for API versioning and fail open in weighted shard routing #5779

Closed

6 tasks

anshu1106 mentioned this pull request Jan 10, 2023

[Backport 2.x] Add support for API versioning and fail open in weighted shard routing #5781

Merged

6 tasks

anshu1106 mentioned this pull request Jan 10, 2023

[Backport 2.5] Add support for API versioning and fail open in weighted shard routing #5784

Merged

6 tasks

anshu1106 mentioned this pull request Jan 28, 2023

[BUG][Weighted Routing] weighted shard routing state is not maintained across search requests #6056

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Weighted Shard Routing] Fail open requests on search shard failures #5072

[Weighted Shard Routing] Fail open requests on search shard failures #5072

anshu1106 commented Nov 4, 2022 •

edited

Loading

github-actions bot commented Nov 4, 2022

github-actions bot commented Nov 9, 2022

github-actions bot commented Nov 9, 2022

github-actions bot commented Nov 10, 2022

github-actions bot commented Nov 10, 2022

github-actions bot commented Nov 10, 2022

gbbafna left a comment

gbbafna Dec 5, 2022

github-actions bot commented Dec 6, 2022

Bukhtawar Jan 9, 2023

Bukhtawar Jan 9, 2023 •

edited

Loading

anshu1106 Jan 9, 2023

Bukhtawar Jan 9, 2023

anshu1106 Jan 10, 2023

Bukhtawar Jan 9, 2023

Bukhtawar Jan 9, 2023

github-actions bot commented Jan 9, 2023

github-actions bot commented Jan 9, 2023

github-actions bot commented Jan 9, 2023

github-actions bot commented Jan 9, 2023

Bukhtawar commented Jan 9, 2023

github-actions bot commented Jan 10, 2023

Bukhtawar Jan 10, 2023

Bukhtawar Jan 10, 2023

[Weighted Shard Routing] Fail open requests on search shard failures #5072

[Weighted Shard Routing] Fail open requests on search shard failures #5072

Conversation

anshu1106 commented Nov 4, 2022 • edited Loading

Description

Issues Resolved

Check List

github-actions bot commented Nov 4, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 9, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 9, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 10, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 10, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 10, 2022

Gradle Check (Jenkins) Run Completed with:

gbbafna left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Dec 6, 2022

Gradle Check (Jenkins) Run Completed with:

Choose a reason for hiding this comment

Bukhtawar Jan 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jan 9, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 9, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 9, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 9, 2023

Gradle Check (Jenkins) Run Completed with:

Bukhtawar commented Jan 9, 2023

github-actions bot commented Jan 10, 2023

Gradle Check (Jenkins) Run Completed with:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anshu1106 commented Nov 4, 2022 •

edited

Loading

Bukhtawar Jan 9, 2023 •

edited

Loading