Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: Benchmark Jetty 12 & 11 #2782

Draft
wants to merge 27 commits into
base: main
Choose a base branch
from
Draft

WIP: Benchmark Jetty 12 & 11 #2782

wants to merge 27 commits into from

Conversation

iamsanjay
Copy link
Contributor

@iamsanjay iamsanjay commented Oct 19, 2024 •
edited
Loading

SOLR-17503

To benchmark the upgraded version of Jetty

  1. Jumping directly to version 12 adds complexity. I've identified Jetty 11 as the transition point to Jakarta.
  2. The hadoop-auth has been disabled due to the absence of a compatible Jakarta library.
  3. Test cases for the S3 repository are currently failing.
  4. These issues have already been reported in SOLR-16441.

Feel free to modify or use this as needed.

@iamsanjay
Copy link
Contributor Author

iamsanjay commented Oct 19, 2024 •
edited
Loading

CloudIndexing Benchmarking (Jetty 11)

Benchmark               (nodeCount)  (numReplicas)  (numShards)  (preGenerate)  (useSmallDocs)   Mode  Cnt      Score      Error  Units
CloudIndexing.indexDoc            4              1            4          50000            true  thrpt    4  18150.029 ± 1510.437  ops/s
CloudIndexing.indexDoc            4              1            4          50000           false  thrpt    4   1915.014 ±  151.667  ops/s
CloudIndexing.indexDoc            4              3            4          50000            true  thrpt    4   6981.943 ± 9864.875  ops/s

Last one ran but sometimes they it did'nt add to the result

Parameters: (nodeCount = 4, numReplicas = 3, numShards = 4, preGenerate = 50000, useSmallDocs = false)

Result "org.apache.solr.bench.index.CloudIndexing.indexDoc":
1569.041 ±(99.9%) 2608.393 ops/s [Average]
(min, avg, max) = (991.489, 1569.041, 1855.751), stdev = 403.652
CI (99.9%): [≈ 0, 4177.433] (assumes normal distribution)

@iamsanjay
Copy link
Contributor Author

iamsanjay commented Oct 19, 2024 •
edited
Loading

Jetty 11

Benchmark               (maxConnectionPerHost)  (nodeCount)  (numReplicas)  (numShards)  (preGenerate)  (useSmallDocs)   Mode  Cnt      Score      Error  Units
CloudIndexing.indexDoc                       2            4              1            4          50000            true  thrpt    4  18503.170 ± 1687.760  ops/s
CloudIndexing.indexDoc                       2            4              1            4          50000           false  thrpt    4   1961.786 ±  135.141  ops/s
CloudIndexing.indexDoc                       2            4              3            4          50000            true  thrpt    4   6611.327 ± 3970.191  ops/s
CloudIndexing.indexDoc                       2            4              3            4          50000           false  thrpt    4   1644.126 ± 2200.796  ops/s
CloudIndexing.indexDoc                       4            4              1            4          50000            true  thrpt    4  17892.214 ± 1566.016  ops/s
CloudIndexing.indexDoc                       4            4              1            4          50000           false  thrpt    4   1933.051 ±   87.418  ops/s
CloudIndexing.indexDoc                       4            4              3            4          50000            true  thrpt    4   5799.993 ± 6217.784  ops/s
CloudIndexing.indexDoc                       4            4              3            4          50000           false  thrpt    4   1613.783 ± 2331.931  ops/s
CloudIndexing.indexDoc                       8            4              1            4          50000           false  thrpt    4   1900.180 ±  220.971  ops/s
CloudIndexing.indexDoc                       8            4              3            4          50000            true  thrpt    4   6747.636 ± 3621.648  ops/s
CloudIndexing.indexDoc                       8            4              3            4          50000           false  thrpt    4   1553.605 ± 2513.942  ops/s
CloudIndexing.indexDoc                      16            4              1            4          50000            true  thrpt    4  17306.442 ± 1617.436  ops/s
CloudIndexing.indexDoc                      16            4              1            4          50000           false  thrpt    4   1906.139 ±  196.615  ops/s
CloudIndexing.indexDoc                      16            4              3            4          50000            true  thrpt    4   5819.974 ± 7693.510  ops/s
CloudIndexing.indexDoc                      16            4              3            4          50000           false  thrpt    4   1679.356 ± 2227.954  ops/s
CloudIndexing.indexDoc                      32            4              1            4          50000            true  thrpt       16848.384             ops/s
CloudIndexing.indexDoc                      32            4              1            4          50000           false  thrpt    4   1697.644 ±  233.939  ops/s
CloudIndexing.indexDoc                      32            4              3            4          50000            true  thrpt    4   5691.744 ± 3292.040  ops/s
CloudIndexing.indexDoc                      32            4              3            4          50000           false  thrpt        1119.878             ops/s
CloudIndexing.indexDoc                      64            4              1            4          50000            true  thrpt    4  17379.236 ± 1731.692  ops/s
CloudIndexing.indexDoc                      64            4              1            4          50000           false  thrpt        1858.977             ops/s
CloudIndexing.indexDoc                      64            4              3            4          50000            true  thrpt    4   7553.278 ± 5983.650  ops/s
CloudIndexing.indexDoc                     128            4              1            4          50000            true  thrpt    4  17317.630 ± 2131.219  ops/s
CloudIndexing.indexDoc                     128            4              3            4          50000            true  thrpt    4   5798.365 ± 3685.703  ops/s

Entries missing means exceptions.

org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: http://127.0.0.1:40981/solr/testCollection/update
        at org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:529)
        at org.apache.solr.bench.index.CloudIndexing.indexDoc(CloudIndexing.java:138)
        at org.apache.solr.bench.index.jmh_generated.CloudIndexing_indexDoc_jmhTest.indexDoc_thrpt_jmhStub(CloudIndexing_indexDoc_jmhTest.java:239)
        at org.apache.solr.bench.index.jmh_generated.CloudIndexing_indexDoc_jmhTest.indexDoc_Throughput(CloudIndexing_indexDoc_jmhTest.java:109)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:527)
        at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:504)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.io.IOException: cancel_stream_error
        at org.eclipse.jetty.http2.client.http.HttpReceiverOverHTTP2.onReset(HttpReceiverOverHTTP2.java:219)
        at org.eclipse.jetty.http2.client.http.Http
```ChannelOverHTTP2$Listener.onReset(HttpChannelOverHTTP2.java:199)

@iamsanjay
Copy link
Contributor Author

Jetty 12

Benchmark               (nodeCount)  (numReplicas)  (numShards)  (preGenerate)  (useSmallDocs)   Mode  Cnt      Score      Error  Units
CloudIndexing.indexDoc            4              1            4          50000            true  thrpt    4  17428.017 ± 1008.795  ops/s
CloudIndexing.indexDoc            4              1            4          50000           false  thrpt    4   1815.855 ±  106.855  ops/s
CloudIndexing.indexDoc            4              3            4          50000            true  thrpt    4   4861.143 ± 5746.330  ops/s
CloudIndexing.indexDoc            4              3            4          50000           false  thrpt    4   1572.753 ± 1931.068  ops/s

@iamsanjay iamsanjay changed the title WIP: Benchmark Jetty 11 WIP: Benchmark Jetty 12 & 11 Oct 22, 2024
@iamsanjay
Copy link
Contributor Author

iamsanjay commented Oct 22, 2024 •
edited
Loading

Almost worked on Jetty 12!

  1. Though few test cases are failing here and there.
  2. Other than that, able to see the Solr UI.
  3. s3-repository and hadoop-auth is still disabled.
  4. Going to share all the new changes that has to be changed in order to make it work.

This was referenced Oct 26, 2024
Open
Open
@malliaridis malliaridis mentioned this pull request Nov 6, 2024
Draft
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@janhoy janhoy janhoy left review comments

Assignees
No one assigned
Labels
cat:api cat:cli cat:cloud cat:index cat:schema cat:search cat:security client:solrj dependencies Dependency upgrades docker Docker image documentation Improvements or additions to documentation jetty-server module:hadoop-auth module:jwt-auth module:s3-repository module:scripting start-scripts test-framework tests tool:build
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@iamsanjay @janhoy