[Segment Replication + Remote Store] GA test planning

[mch2]

This issue contains a running list of testing required In preparation for the release of using remote storage with segment replication.

Objective

1. Show trade-offs when compared to document replication.
2. Show trade-offs when compared to segment node-node replication.
3. Learn break points & bottlenecks depending on cluster configuration/workload.

General Hypothesis

1. Segrep + remote store (S&R) will achieve better ingest throughput, search latencies, and replication lag times as replica count grows. This is attributed to primary shards not incurring an incremental network cost per replica.
2. S&R would see higher primary refresh time when compared to node-node replication. This is attributed to primary shards synchronously uploading to remote store in refresh path.
3. Overloaded clusters should result in remote store and/or segment replication backpressure applied.

Performance Benchmark Plan #

Each use case listed below should also be performed with docrep & node-node replication and where possible added to our nightly runs on https://opensearch.org/benchmarks for easy ongoing comparison.

Metrics that should be captured in addition what OSB reports:

• Replication lag - Add to OSB.
• Network Throughput
• IOPS
• Thread pool stats - I think this may already be an optional return from OSB.

All clusters should have 3 dedicated cluster manager nodes.
Small cluster = ~3 nodes
Large cluster = ~40 nodes
Use m5.xlarge node type for consistency.

Use case	Workload(s)	Variations	Notes	Why	Assignee
Ingest append-only	SO	Small cluster (3 nodes, 10 shards). 1 replica case and max replica case	Small cluster here is equivalent to existing nightlies.	General ingest performance
Update heavy / Large dataset	http_logs	large cluster (~10GB per shard) - 1 replica case & max replica case		Validate assertions that segrep performs even better in update/delete cases.
	http_logs	large cluster (~50GB per shard) - 1 replica case & max replica case		Comparison as shards grow in size
	http_logs	Small cluster - (~10GB per shard) 1 replica case & max replica case
	http_logs	Small cluster - (50gb per shard) 1 replica case & max replica case
Concurrent ingest/search with deep replica counts	TBD		No OSB workload exists today that concurrently performs ingest & search.	Validate hypothesis 1 with consistent ingest/search traffic
Zero replicas configured	any			Ensure no downgrade to docrep.

Additional tests (nice to have):
Longevity testing - Longer running test with large cluster.
Architecture comparison (x86 v arm)

Feature Stability and Testing #

This is a list of specific test cases that should be performed either through existing/new ITs or manual.
We are working on enabling S+R across our entire server suite as part of this issue, but are hitting challenges. So in the meantime creating a running list of the most critical.

Feature / Type	Test Case	Notes:
Backpressure
	Primary slow to upload
	Replica slow to fetch
	Slow checkpoint publish
Relocation
	Primary-Primary
	Replica or Primary to Replica
Shard events
	Delete index
	Replica removal
	Replica add
	Primary drop
	Replica drop
	Replica restart
	Primary drop
	Trigger a drop during segment download
Segrep Stats API
	All metrics present
Mem leak / heap analysis		This would be manual, fetch a heap dump for analysis.
	Constant calls to segrep stats API
	Before/after replication events on primary
	before/after replication events on replica
BWC
	Test node-node for no breakage. Add new tests in next release for remote store

Existing test packages to run with SR enabled in :server.
cluster
gateway
index
indexing
indices
ingest
recovery
remotestore
update

[mch2]

Added an issue to opensearch-benchmark for adding replication lag metric - opensearch-project/opensearch-benchmark#339. This may require moving the lag metric to node stats similar to other captured metrics.

[ankitkala]

• Definitely agree with adding a telemetry device in OSB to capture the replication lag. Here is a similar change we did for CCR lag.

Network Throughput
IOPS
Thread pool stats - I think this may already be an optional return from OSB.

These metrics can be published to a different cluster by OSB buy i don't think these are part of the test result.

[mch2]

Splitting this out into individual runs:

Here's what I'm thinking in terms of set up -
All clusters should have 3 dedicated cluster manager nodes.
Small cluster = ~3 nodes
Large cluster = ~40 nodes
Use m5.xlarge node type for consistency.

Use case	Workload(s)	Variations	Notes	Why	Assignee
Ingest append-only	SO	Small cluster (3 nodes, 10 shards). 1 replica case and max replica case	Small cluster here is equivalent to existing nightlies.	General ingest performance
Update heavy / Large dataset	http_logs	large cluster (~10GB per shard) - 1 replica case & max replica case		Validate assertions that segrep performs even better in update/delete cases.
	http_logs	large cluster (~50GB per shard) - 1 replica case & max replica case		Comparison as shards grow in size
	http_logs	Small cluster - (~10GB per shard) 1 replica case & max replica case
	http_logs	Small cluster - (50gb per shard) 1 replica case & max replica case
Concurrent ingest/search with deep replica counts	TBD		No OSB workload exists today that concurrently performs ingest & search.	Validate hypothesis 1 with consistent ingest/search traffic
Zero replicas configured	any			Ensure no downgrade to docrep.

[ankitkala]

For small cluster configuration, let's test with shards in multiple of 3 (currently testing with 3, 6, 12, 24). Similar configuration is being used for remote store perf testing as well

[mch2]

Similar configuration is being used for remote store perf testing as well

Where can we track that testing? I expect there is overlap here given node-node replication is not used with remote store?

[tlfeng]

About the performance testing, I created an dedicated issue #8874 to track.
I will run the performance test locally to satisfy all the requirement in this page.
I plan to post the steps to locally run the benchmark test script that existed in opensearch-build repository there.
While to run these test scenario nightly in CI server, there are a few work needed, I created two issues for that.
opensearch-project/opensearch-benchmark#339 Capture and report replication lag metric
opensearch-project/opensearch-build#3701 Integrate feature of increasing the data size with benchmark test script