Add Benchmark glossary #8190

Naarcha-AWS · 2024-09-05T23:26:55Z

Add Benchmark glossary.

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Archer <[email protected]>

github-actions · 2024-09-05T23:27:30Z

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

IanHoang · 2024-09-11T15:18:45Z

_benchmark/glossary.md

+The following terms are commonly used in OpenSearch Benchmark:
+
+- **Corpora**: A collection of documents.
+- **Latency**: Based on the `target-throughput` set by the user, the total amount of time that the request waits before receiving the response, in addition to any other delays that occur before the request is sent.


Can we rephrase this to:
If target-throughput is disabled (i.e. has no value or a value of 0), latency is equivalent to service time. If target-throughput is enabled (i.e. has a value of 1 or greater), latency is the service time plus the time the request waits in the queue before being sent.

_benchmark/glossary.md

IanHoang

Small changes suggested

_benchmark/glossary.md

Signed-off-by: Naarcha-AWS <[email protected]>

_benchmark/glossary.md

Signed-off-by: Naarcha-AWS <[email protected]>

Naarcha-AWS · 2024-09-12T17:02:25Z

@IanHoang: Feel free to review again.

kolchfa-aws

LGTM. Some clarifying quesions.

kolchfa-aws · 2024-09-19T19:11:12Z

_benchmark/glossary.md

+- **Corpora**: A collection of documents.
+- **Latency**: If `target-throughput` is disabled (has no value or a value of `0)`, latency is equivalent to service time. If `target-throughput` is enabled (has a value of 1 or greater), latency is the service time plus the time the request waits in the queue before being sent.
+- **Metric keys**: The metrics that OpenSearch Benchmark stores, based on the configuration in the [metrics record]({{site.url}}{{site.baseurl}}/benchmark/metrics/metric-records/).
+- **Operations**: In workloads, a list of API requests performed by a workload.


Should we say "API operations for consistency?

kolchfa-aws · 2024-09-19T19:12:33Z

_benchmark/glossary.md

+- **Metric keys**: The metrics that OpenSearch Benchmark stores, based on the configuration in the [metrics record]({{site.url}}{{site.baseurl}}/benchmark/metrics/metric-records/).
+- **Operations**: In workloads, a list of API requests performed by a workload.
+- **Pipeline**: A series of steps occurring before and after a workload is run that determines benchmark results.
+- **Schedule**: In workloads, a list of operations in a specific order.


So, the difference between this and operations is that operations is not necessarily in order but schedule is? Also, is the order time-based, and if so, is it execution time or the time the request was put in the queue or sent?

Not exactly. A schedule is simply a list of two or more operations performed in the order they appear at the time the workload is run. The order the operations in a schedule isn't time-based.

I'll adjust the definition accordingly.

_benchmark/glossary.md

kolchfa-aws · 2024-09-19T19:14:20Z

_benchmark/glossary.md

+- **Summary report**: A report output at the end a test based on the metric keys defined in the workload.
+- **Test**: A single invocation of the OpenSearch Benchmark binary.
+- **Throughput**: The number of operations completed in a given period of time.
+- **Workload**: A collection of one or more benchmarking scenarios that use a specific document corpus to perform a benchmark against your cluster. The document corpus contains any indexes, data files, and operations invoked when the workload runs.


Is a scenario the same as a test?

Yes. Though we'll switch this one to "test" for now. In Benchmark 2.0, which has yet to be released, we are renaming "tests" --- "scenarios".

_benchmark/glossary.md

Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]>

_benchmark/glossary.md

Signed-off-by: Naarcha-AWS <[email protected]>

natebower

@Naarcha-AWS Please see my comments and changes and let me know if you have any questions. Thanks!

_benchmark/glossary.md

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]>

_benchmark/glossary.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Add Benchmark glossary Signed-off-by: Archer <[email protected]> * Add text Signed-off-by: Archer <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Archer <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> (cherry picked from commit 7d3a3f5) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Add Benchmark glossary Signed-off-by: Archer <[email protected]> * Add text Signed-off-by: Archer <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Archer <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: [email protected] <[email protected]>

* Add Benchmark glossary Signed-off-by: Archer <[email protected]> * Add text Signed-off-by: Archer <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Archer <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Noah Staveley <[email protected]>

* Add Benchmark glossary Signed-off-by: Archer <[email protected]> * Add text Signed-off-by: Archer <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Archer <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]>

Naarcha-AWS added 2 commits September 5, 2024 18:24

Add Benchmark glossary

fb59274

Signed-off-by: Archer <[email protected]>

Add text

b7b9df6

Signed-off-by: Archer <[email protected]>

Naarcha-AWS added 3 - Tech review PR: Tech review in progress backport 2.16 labels Sep 5, 2024

Naarcha-AWS requested review from kolchfa-aws, vagimeli, AMoo-Miki, natebower, dlvenable, stephen-crawford and epugh as code owners September 5, 2024 23:26

github-actions bot assigned Naarcha-AWS Sep 5, 2024

IanHoang reviewed Sep 11, 2024

View reviewed changes