[Segment Replication] Adding segment replication statistics rolled up at index, node and cluster level #9709

Poojita-Raj · 2023-09-01T22:35:18Z

Description

Surfaces existing segment replication statistics -
(1) maxBytesBehind
(2) totalBytesBehind and
(3) maxReplicationLag
at an index, node and cluster level.

Related Issues

Resolves #9417

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-09-01T22:44:06Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24186/
CommitID: 3bef3a3
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

...er/src/internalClusterTest/java/org/opensearch/indices/replication/SegmentReplicationIT.java

server/src/internalClusterTest/java/org/opensearch/indices/stats/IndexStatsIT.java

server/src/main/java/org/opensearch/index/ReplicationStats.java

server/src/main/java/org/opensearch/index/shard/IndexShard.java

github-actions · 2023-09-01T23:04:44Z

Compatibility status:

Checks if related components are compatible with change 3bef3a3

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-09-01T23:07:55Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24188/
CommitID: c0c20a2
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-01T23:10:09Z

Compatibility status:

Checks if related components are compatible with change c0c20a2

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

Signed-off-by: Poojita Raj <[email protected]>

github-actions · 2023-09-03T16:28:21Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24282/
CommitID: 01d4dfe
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-03T16:50:54Z

Compatibility status:

Checks if related components are compatible with change 01d4dfe

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

mch2 · 2023-09-03T21:03:05Z

@Poojita-Raj Bwc tests are failing because of wire compatibility of the new field. We need to PR here with 3_0_0, then on the backport to 2.x flip it to 2_10_0, then make another PR to main to flip to 2_10_0.

Signed-off-by: Poojita Raj <[email protected]>

github-actions · 2023-09-04T06:28:51Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24302/
CommitID: 0e64471
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-04T06:28:55Z

Compatibility status:

Checks if related components are compatible with change 0e64471

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

Poojita-Raj · 2023-09-04T06:38:10Z

Gradle Check (Jenkins) Run Completed with:

* **RESULT:** FAILURE ❌

* **URL:** https://build.ci.opensearch.org/job/gradle-check/24302/

* **CommitID:** [0e64471](https://github.com/opensearch-project/OpenSearch/commit/0e6447118f9a98ca3f1c21f7c92c7c6e2650c775)
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure [a flaky test](https://github.com/opensearch-project/OpenSearch/blob/main/DEVELOPER_GUIDE.md#flaky-tests) unrelated to your change?

Flaky:
#9688
#7934

github-actions · 2023-09-04T07:26:44Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.snapshots.CloneSnapshotIT.testCloneAfterRepoShallowSettingDisabled

URL: https://build.ci.opensearch.org/job/gradle-check/24307/
CommitID: 0e64471
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-09-04T07:27:40Z

Codecov Report

Merging #9709 (7ad9831) into main (63ce832) will decrease coverage by 0.04%.
Report is 1 commits behind head on main.
The diff coverage is 66.07%.

@@             Coverage Diff              @@
##               main    #9709      +/-   ##
============================================
- Coverage     71.07%   71.04%   -0.04%     
- Complexity    57814    57858      +44     
============================================
  Files          4815     4816       +1     
  Lines        272709   272763      +54     
  Branches      39792    39795       +3     
============================================
- Hits         193839   193794      -45     
- Misses        62622    62690      +68     
- Partials      16248    16279      +31

Files Changed	Coverage Δ
...in/java/org/opensearch/index/shard/IndexShard.java	`69.03% <25.00%> (-0.44%)`	⬇️
...ava/org/opensearch/index/engine/SegmentsStats.java	`72.46% <72.72%> (+0.02%)`	⬆️
...in/java/org/opensearch/index/ReplicationStats.java	`77.41% <77.41%> (ø)`
...earch/index/SegmentReplicationPressureService.java	`76.69% <100.00%> (ø)`
...ensearch/index/SegmentReplicationStatsTracker.java	`95.45% <100.00%> (ø)`

... and 450 files with indirect coverage changes

Signed-off-by: Rishikesh Pasham <[email protected]>

github-actions · 2023-09-04T16:53:49Z

Compatibility status:

Checks if related components are compatible with change 7ad9831

Incompatible components

Incompatible components: [https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/neural-search.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-09-04T17:02:56Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24360/
CommitID: 7ad9831
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

mch2 · 2023-09-04T17:17:17Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌

URL: https://build.ci.opensearch.org/job/gradle-check/24360/

CommitID: 7ad9831
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

#9688

github-actions · 2023-09-04T18:05:29Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testSearchAggregationWithNetworkDisruption_FailOpenEnabled

URL: https://build.ci.opensearch.org/job/gradle-check/24367/
CommitID: 7ad9831
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2023-09-04T18:12:38Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-9709-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 aa8367027cddd727cb7da4563187ecfc59dc6ad8
# Push it to GitHub
git push --set-upstream origin backport/backport-9709-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-9709-to-2.x.

dreamer-89 · 2023-09-04T20:29:55Z

@Poojita-Raj @mch2 : Seems like we need manual backport for this change.

… at index, node and cluster level (opensearch-project#9709) * add node stats for segrep Signed-off-by: Poojita Raj <[email protected]> * comment removal Signed-off-by: Poojita Raj <[email protected]> * changelog Signed-off-by: Poojita Raj <[email protected]> * address review comments Signed-off-by: Poojita Raj <[email protected]> * move segrep stats to segments Signed-off-by: Poojita Raj <[email protected]> * fix failures Signed-off-by: Poojita Raj <[email protected]> * uniform response for docrep and segrep segment stats Signed-off-by: Poojita Raj <[email protected]> * remove pri check Signed-off-by: Poojita Raj <[email protected]> * version 3.0 Signed-off-by: Poojita Raj <[email protected]> --------- Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Rishikesh Pasham <[email protected]> Co-authored-by: Rishikesh Pasham <[email protected]>

… at index, node and cluster level (#9709) (#9741) * [Segment Replication] Adding segment replication statistics rolled up at index, node and cluster level (#9709) * add node stats for segrep Signed-off-by: Poojita Raj <[email protected]> * comment removal Signed-off-by: Poojita Raj <[email protected]> * changelog Signed-off-by: Poojita Raj <[email protected]> * address review comments Signed-off-by: Poojita Raj <[email protected]> * move segrep stats to segments Signed-off-by: Poojita Raj <[email protected]> * fix failures Signed-off-by: Poojita Raj <[email protected]> * uniform response for docrep and segrep segment stats Signed-off-by: Poojita Raj <[email protected]> * remove pri check Signed-off-by: Poojita Raj <[email protected]> * version 3.0 Signed-off-by: Poojita Raj <[email protected]> --------- Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Rishikesh Pasham <[email protected]> Co-authored-by: Rishikesh Pasham <[email protected]> * Change bwc version to 2.10 from 3.0 Signed-off-by: Suraj Singh <[email protected]> --------- Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Rishikesh Pasham <[email protected]> Signed-off-by: Suraj Singh <[email protected]> Co-authored-by: Poojita Raj <[email protected]> Co-authored-by: Rishikesh Pasham <[email protected]>

mch2 · 2023-09-04T22:58:25Z

server/src/main/java/org/opensearch/index/ReplicationStats.java

+ * compatible open source license.
+ */
+
+package org.opensearch.index;


Sorry didn't catch this - lets move this to indices.replication package?

… at index, node and cluster level (opensearch-project#9709) * add node stats for segrep Signed-off-by: Poojita Raj <[email protected]> * comment removal Signed-off-by: Poojita Raj <[email protected]> * changelog Signed-off-by: Poojita Raj <[email protected]> * address review comments Signed-off-by: Poojita Raj <[email protected]> * move segrep stats to segments Signed-off-by: Poojita Raj <[email protected]> * fix failures Signed-off-by: Poojita Raj <[email protected]> * uniform response for docrep and segrep segment stats Signed-off-by: Poojita Raj <[email protected]> * remove pri check Signed-off-by: Poojita Raj <[email protected]> * version 3.0 Signed-off-by: Poojita Raj <[email protected]> --------- Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Rishikesh Pasham <[email protected]> Co-authored-by: Rishikesh Pasham <[email protected]> Signed-off-by: Kaushal Kumar <[email protected]>

… at index, node and cluster level (opensearch-project#9709) * add node stats for segrep Signed-off-by: Poojita Raj <[email protected]> * comment removal Signed-off-by: Poojita Raj <[email protected]> * changelog Signed-off-by: Poojita Raj <[email protected]> * address review comments Signed-off-by: Poojita Raj <[email protected]> * move segrep stats to segments Signed-off-by: Poojita Raj <[email protected]> * fix failures Signed-off-by: Poojita Raj <[email protected]> * uniform response for docrep and segrep segment stats Signed-off-by: Poojita Raj <[email protected]> * remove pri check Signed-off-by: Poojita Raj <[email protected]> * version 3.0 Signed-off-by: Poojita Raj <[email protected]> --------- Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Rishikesh Pasham <[email protected]> Co-authored-by: Rishikesh Pasham <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

… at index, node and cluster level (opensearch-project#9709) * add node stats for segrep Signed-off-by: Poojita Raj <[email protected]> * comment removal Signed-off-by: Poojita Raj <[email protected]> * changelog Signed-off-by: Poojita Raj <[email protected]> * address review comments Signed-off-by: Poojita Raj <[email protected]> * move segrep stats to segments Signed-off-by: Poojita Raj <[email protected]> * fix failures Signed-off-by: Poojita Raj <[email protected]> * uniform response for docrep and segrep segment stats Signed-off-by: Poojita Raj <[email protected]> * remove pri check Signed-off-by: Poojita Raj <[email protected]> * version 3.0 Signed-off-by: Poojita Raj <[email protected]> --------- Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Rishikesh Pasham <[email protected]> Co-authored-by: Rishikesh Pasham <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

Poojita-Raj changed the title ~~Adding segment replication statistics rolled up at index, node and cluster level~~ [Segment Replication] Adding segment replication statistics rolled up at index, node and cluster level Sep 1, 2023

mch2 reviewed Sep 1, 2023

View reviewed changes

remove pri check

01d4dfe

Signed-off-by: Poojita Raj <[email protected]>

version 3.0

0e64471

Signed-off-by: Poojita Raj <[email protected]>

mch2 approved these changes Sep 4, 2023

View reviewed changes

Merge branch 'main' into node_stats

7ad9831

Signed-off-by: Rishikesh Pasham <[email protected]>

mch2 merged commit aa83670 into opensearch-project:main Sep 4, 2023

mch2 added the backport 2.x Backport to 2.x branch label Sep 4, 2023

opensearch-trigger-bot bot added the backport-failed label Sep 4, 2023

dreamer-89 mentioned this pull request Sep 4, 2023

[Segment Replication] Adding segment replication statistics rolled up at index, node and cluster level (#9709) #9741

Merged

mch2 reviewed Sep 4, 2023

View reviewed changes

Rishikesh1159 mentioned this pull request Sep 4, 2023

[Segment Replication] Update version back to 2_10_0 in Segmentstats. #9745

Merged

6 tasks

Poojita-Raj mentioned this pull request Oct 11, 2023

[Segment Replication] Aggregate backpressure metrics at node/cluster/index level. #10365

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Segment Replication] Adding segment replication statistics rolled up at index, node and cluster level #9709

[Segment Replication] Adding segment replication statistics rolled up at index, node and cluster level #9709

Poojita-Raj commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

github-actions bot commented Sep 3, 2023

github-actions bot commented Sep 3, 2023

mch2 commented Sep 3, 2023

github-actions bot commented Sep 4, 2023

github-actions bot commented Sep 4, 2023

Poojita-Raj commented Sep 4, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 4, 2023

codecov bot commented Sep 4, 2023 •

edited

Loading

github-actions bot commented Sep 4, 2023

github-actions bot commented Sep 4, 2023

mch2 commented Sep 4, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 4, 2023

opensearch-trigger-bot bot commented Sep 4, 2023

dreamer-89 commented Sep 4, 2023

mch2 Sep 4, 2023

[Segment Replication] Adding segment replication statistics rolled up at index, node and cluster level #9709

[Segment Replication] Adding segment replication statistics rolled up at index, node and cluster level #9709

Conversation

Poojita-Raj commented Sep 1, 2023

Description

Related Issues

Check List

github-actions bot commented Sep 1, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 1, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 1, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 1, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 3, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 3, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

mch2 commented Sep 3, 2023

github-actions bot commented Sep 4, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 4, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

Poojita-Raj commented Sep 4, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 4, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Sep 4, 2023 • edited Loading

Codecov Report

github-actions bot commented Sep 4, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 4, 2023

Gradle Check (Jenkins) Run Completed with:

mch2 commented Sep 4, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 4, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Sep 4, 2023

dreamer-89 commented Sep 4, 2023

mch2 Sep 4, 2023

Choose a reason for hiding this comment

codecov bot commented Sep 4, 2023 •

edited

Loading