Add Jedis and Lettuce benchmarks #1

jonathanl-bq · 2023-09-07T03:58:24Z

This PR is based off of Andrew's fork. It measures the average latency, 50, 90, and 99 percentile latencies, and standard deviation for Jedis and Lettuce clients, similar to the existing benchmarks for Python and Node. It still is missing some functionality, like measuring tasks per second and asynchronous tests, which should be implemented in a later PR.

There's also an issue where running ./gradlew test only executes the tests once. Trying to run it a second time will not execute the tests or print any benchmark results. There's probably a Gradle setting to stop it from doing this, but I'm not sure how yet.

Yury-Fridlyand · 2023-09-07T04:08:56Z

Probably, you need to update .gitignore to exclude build and .gradle dirs and *.class files and/or remove them manually.

Yury-Fridlyand · 2023-09-15T03:45:12Z

java/benchmarks/src/test/java/javabushka/client/Benchmarking.java

    static final int SIZE_GET_KEYSPACE = 3750000;
    static final int SIZE_SET_KEYSPACE = 3000000;

+    private static ChosenAction randomAction() {


So test makes 80% SETs, 16% GET_NON_EXISTING and 4% GET_EXISTING - is that correct?

I believe so. It's just like how the Node benchmarks select actions. We can certainly bring this up for discussion if we want to do this a different way, however.

java/benchmarks/build.gradle

Yury-Fridlyand · 2023-09-21T02:33:41Z

java/benchmarks/src/main/java/javabushka/client/BenchmarkingApp.java

+    // create the Options
+    Options options = new Options();
+
+    options.addOption("c", "configuration", true, "Configuration flag [Release]");


add dataSize (-d)

java/benchmarks/src/main/java/javabushka/client/BenchmarkingApp.java

java/benchmarks/src/main/java/javabushka/client/jedis/JedisClient.java

java/benchmarks/src/main/java/javabushka/client/lettuce/LettuceAsyncClient.java

Yury-Fridlyand · 2023-09-21T03:28:45Z

benchmarks/install_and_test.sh

+            runJava=1
+            chosenClients="Lettuce"
+            ;;
+        -lettuce)


Suggested change

-lettuce)

-jedis)

java/benchmarks/src/main/java/javabushka/client/BenchmarkingApp.java

Yury-Fridlyand · 2023-09-21T03:35:27Z

benchmarks/install_and_test.sh

+  cd ${BENCH_FOLDER}/../java
+  echo "./gradlew run --args=\"--resultsFile=${BENCH_FOLDER}/$1 --clients $chosenClients --host $host --port $port\""
+#  ./gradlew run --args="--resultsFile=../$1 --dataSize $2 --concurrentTasks $concurrentTasks --clients $chosenClients --host $host --port $port --clientCount $clientCount $tlsFlag"
+  ./gradlew run --args="--resultsFile=${BENCH_FOLDER}/$1 --clients $chosenClients --host $host --port $port"


reminder for

--dataSize

--concurrentTasks

--clientCount

$tlsFlag

Yury-Fridlyand · 2023-09-30T02:03:32Z

java/.cargo/config.toml

@@ -0,0 +1,3 @@
+[env]
+BABUSHKA_NAME = { value = "BabushkaPy", force = true }


Signed-off-by: acarbonetto <[email protected]>

This reverts commit d9b26a6.

Signed-off-by: acarbonetto <[email protected]>

* Add a java app to run benchmarks --------- Signed-off-by: acarbonetto <[email protected]>

Signed-off-by: acarbonetto <[email protected]>

* Merge Pull Request #5 - Add java pipeline. Also changed: * Merged two projects. * Updated CI. * Fixed tests and updated `junit` version. * Spotless. * Add new gradle tasks. Signed-off-by: Yury-Fridlyand <[email protected]>

* Add sync and async clients both to tests. Signed-off-by: Yury-Fridlyand <[email protected]> * Minor fixes. Signed-off-by: Yury-Fridlyand <[email protected]> --------- Signed-off-by: Yury-Fridlyand <[email protected]>

…onvert to TypeScript. (valkey-io#456) removed duplicated logic and refactored to typescript Signed-off-by: acarbonetto <[email protected]>

* Add Jedis and Lettuce benchmarks * Start ignoring .gradle files * Update gitignore and remove generated files from git Signed-off-by: acarbonetto <[email protected]> * Update gitignore and remove generated files from git Signed-off-by: acarbonetto <[email protected]> * Update gitignore and remove generated files from git Signed-off-by: acarbonetto <[email protected]> * Add benchmarks for GET non-existing * Revert "Update gitignore and remove generated files from git" This reverts commit d9b26a6. * fix redis-rs submodules Signed-off-by: acarbonetto <[email protected]> * Randomize commands in Java benchmarks * rename chooseAction to randomAction * Add a Java benchmarking app (#7) * Add a java app to run benchmarks --------- Signed-off-by: acarbonetto <[email protected]> * Add Readme and update install_and_test script to runJava Signed-off-by: acarbonetto <[email protected]> * Add Readme and update install_and_test script to runJava Signed-off-by: acarbonetto <[email protected]> * Combine java pipeline and java benchmarks (#8) * Merge Pull Request #5 - Add java pipeline. Also changed: * Merged two projects. * Updated CI. * Fixed tests and updated `junit` version. * Spotless. * Add new gradle tasks. Signed-off-by: Yury-Fridlyand <[email protected]> * Add sync and async clients both to tests. (#12) * Add sync and async clients both to tests. Signed-off-by: Yury-Fridlyand <[email protected]> * Minor fixes. Signed-off-by: Yury-Fridlyand <[email protected]> --------- Signed-off-by: Yury-Fridlyand <[email protected]> * Add dataSize option to java benchmark. Signed-off-by: Yury-Fridlyand <[email protected]> --------- Signed-off-by: acarbonetto <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]> Co-authored-by: Jonathan Louie <[email protected]> Co-authored-by: acarbonetto <[email protected]> Co-authored-by: jonathanl-bq <[email protected]>

Signed-off-by: acarbonetto <[email protected]>

Signed-off-by: Yury-Fridlyand <[email protected]>

Yury-Fridlyand · 2023-10-04T01:24:44Z

submodules/redis-rs

* Add option to run tests on multiple clients in concurrency * Common pool of iterations. * Awaiting result from async methods. Signed-off-by: Yury-Fridlyand <[email protected]> * minor fix Signed-off-by: Yury-Fridlyand <[email protected]> * Change while-loop; Spotless Apply Signed-off-by: acarbonetto <[email protected]> --------- Signed-off-by: Yury-Fridlyand <[email protected]> Signed-off-by: acarbonetto <[email protected]> Co-authored-by: acarbonetto <[email protected]>

Signed-off-by: acarbonetto <[email protected]>

* Updated ClientCount to client_count for uniformity for rust.

* Add JSON reporting. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix for #26. Signed-off-by: Yury-Fridlyand <[email protected]> * Update java/benchmarks/src/main/java/javababushka/benchmarks/utils/Benchmarking.java Signed-off-by: Yury-Fridlyand <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]> * Use `Optional`. Signed-off-by: Yury-Fridlyand <[email protected]> * Address PR feedback. Signed-off-by: Yury-Fridlyand <[email protected]> --------- Signed-off-by: Yury-Fridlyand <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]>

Signed-off-by: Yury-Fridlyand <[email protected]>

…java_benchmarks Signed-off-by: Yury-Fridlyand <[email protected]>

Signed-off-by: acarbonetto <[email protected]>

* Add lettuce cluster client when cluster mode enabled --------- Signed-off-by: Andrew Carbonetto <[email protected]>

Signed-off-by: acarbonetto <[email protected]>

* Add Java-client benchmarking app Signed-off-by: acarbonetto <[email protected]> * spotless apply Signed-off-by: acarbonetto <[email protected]> * Update on command line options Signed-off-by: acarbonetto <[email protected]> * Update README Signed-off-by: acarbonetto <[email protected]> * Spotless apply: Signed-off-by: acarbonetto <[email protected]> * Update README example Signed-off-by: acarbonetto <[email protected]> * update commandline defaults for review comments Signed-off-by: acarbonetto <[email protected]> * Remove TLS flag argument from option Signed-off-by: acarbonetto <[email protected]> * Add lettuce clients for benchmarking Signed-off-by: acarbonetto <[email protected]> * Spotless apply Signed-off-by: acarbonetto <[email protected]> * Add Jedis clients Signed-off-by: acarbonetto <[email protected]> * Add to app Signed-off-by: acarbonetto <[email protected]> * Add for-loop for data size list Signed-off-by: acarbonetto <[email protected]> * Add TPS for all async items Signed-off-by: acarbonetto <[email protected]> * spotless apply Signed-off-by: acarbonetto <[email protected]> * Fix TPS calculations Signed-off-by: acarbonetto <[email protected]> * Accept TLS as a flag Signed-off-by: acarbonetto <[email protected]> * Start threads; then wait for results Signed-off-by: acarbonetto <[email protected]> * Add java-jni client Signed-off-by: acarbonetto <[email protected]> * Handle Exceptions from client; add JniSyncClient fixes Signed-off-by: acarbonetto <[email protected]> * Clean up latency and add error checking Signed-off-by: acarbonetto <[email protected]> * Minor fixes. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix result printing. Signed-off-by: Yury-Fridlyand <[email protected]> * Add TPS. Signed-off-by: Yury-Fridlyand <[email protected]> * Remove duplicates. Reorganize and fix imports. Signed-off-by: Yury-Fridlyand <[email protected]> * Int ctor fix. Signed-off-by: Yury-Fridlyand <[email protected]> * Iteration 1. Signed-off-by: Yury-Fridlyand <[email protected]> * Iteration 2: connected! Signed-off-by: Yury-Fridlyand <[email protected]> * Iteration 3: `get` and `set`. Signed-off-by: Yury-Fridlyand <[email protected]> * Iteration 4: benchmark. Signed-off-by: Yury-Fridlyand <[email protected]> * Iteration 5: some fixes. Signed-off-by: Yury-Fridlyand <[email protected]> * Change number of threads in Benchmarking threadpool * Revert "Change number of threads in Benchmarking threadpool" This reverts commit e3f7596. * Add more flushing rules and UT. Signed-off-by: Yury-Fridlyand <[email protected]> * Client clean up. Signed-off-by: Yury-Fridlyand <[email protected]> * Client optimizations. (#37) * Client optimizations. Signed-off-by: Yury-Fridlyand <[email protected]> * minor cleanup. Signed-off-by: Yury-Fridlyand <[email protected]> * Optimize building a command. Signed-off-by: Yury-Fridlyand <[email protected]> * Typo fix. Signed-off-by: Yury-Fridlyand <[email protected]> * Minor rename. Signed-off-by: Yury-Fridlyand <[email protected]> * Clean up Redis close connection Signed-off-by: Andrew Carbonetto <[email protected]> * Clean up Redis close connection Signed-off-by: Andrew Carbonetto <[email protected]> * Minor changes. Signed-off-by: Yury-Fridlyand <[email protected]> * Add todos to closeConnection() Signed-off-by: Andrew Carbonetto <[email protected]> --------- Signed-off-by: Yury-Fridlyand <[email protected]> Signed-off-by: Andrew Carbonetto <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]> * Address PR feedback. Signed-off-by: Yury-Fridlyand <[email protected]> * Rename Signed-off-by: Yury-Fridlyand <[email protected]> * Rename2 Signed-off-by: Yury-Fridlyand <[email protected]> * Fix CI Signed-off-by: Yury-Fridlyand <[email protected]> * More fixes. Signed-off-by: Yury-Fridlyand <[email protected]> * Some changes. Signed-off-by: Yury-Fridlyand <[email protected]> * add null check Signed-off-by: Yury-Fridlyand <[email protected]> * autoflush Signed-off-by: Yury-Fridlyand <[email protected]> * Apply suggestions from code review Signed-off-by: Yury-Fridlyand <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]> * minor changes Signed-off-by: Yury-Fridlyand <[email protected]> --------- Signed-off-by: acarbonetto <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]> Signed-off-by: Andrew Carbonetto <[email protected]> Co-authored-by: acarbonetto <[email protected]> Co-authored-by: Jonathan Louie <[email protected]>

Yury-Fridlyand · 2023-12-15T18:33:44Z

Already merged on upstream

jonathanl-bq requested review from Yury-Fridlyand, acarbonetto and Elen-Ghulam September 7, 2023 03:58

Yury-Fridlyand reviewed Sep 15, 2023

View reviewed changes

Yury-Fridlyand reviewed Sep 16, 2023

View reviewed changes

java/benchmarks/build.gradle Outdated Show resolved Hide resolved

Yury-Fridlyand mentioned this pull request Sep 20, 2023

Combine java pipeline and java benchmarks #8

Merged

Yury-Fridlyand reviewed Sep 21, 2023

View reviewed changes

This was referenced Sep 22, 2023

Add json reporting. #10

Merged

Add sync and async clients both to tests. #12

Merged

acarbonetto force-pushed the java_benchmarks branch from bed8235 to c542597 Compare September 25, 2023 22:45

acarbonetto force-pushed the main branch from c272399 to c7486c3 Compare September 26, 2023 19:36

Yury-Fridlyand reviewed Sep 30, 2023

View reviewed changes

java/.cargo/config.toml Outdated

@@ -0,0 +1,3 @@

[env]

BABUSHKA_NAME = { value = "BabushkaPy", force = true }

Copy link

Yury-Fridlyand Sep 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rename

jonathanl-bq and others added 16 commits October 3, 2023 13:33

Add Jedis and Lettuce benchmarks

06574ea

Start ignoring .gradle files

5990767

Update gitignore and remove generated files from git

6ae93f5

Signed-off-by: acarbonetto <[email protected]>

Update gitignore and remove generated files from git

8203c4d

Signed-off-by: acarbonetto <[email protected]>

Update gitignore and remove generated files from git

e517744

Signed-off-by: acarbonetto <[email protected]>

Add benchmarks for GET non-existing

2a11e9a

Revert "Update gitignore and remove generated files from git"

cbb0dcb

This reverts commit d9b26a6.

fix redis-rs submodules

5f51a5b

Signed-off-by: acarbonetto <[email protected]>

Randomize commands in Java benchmarks

52df672

rename chooseAction to randomAction

84f0efc

Add a Java benchmarking app (#7)

1983974

* Add a java app to run benchmarks --------- Signed-off-by: acarbonetto <[email protected]>

Add Readme and update install_and_test script to runJava

6c1fb45

Signed-off-by: acarbonetto <[email protected]>

Add Readme and update install_and_test script to runJava

46d0cf6

Signed-off-by: acarbonetto <[email protected]>

Combine java pipeline and java benchmarks (#8)

d526f96

* Merge Pull Request #5 - Add java pipeline. Also changed: * Merged two projects. * Updated CI. * Fixed tests and updated `junit` version. * Spotless. * Add new gradle tasks. Signed-off-by: Yury-Fridlyand <[email protected]>

Add sync and async clients both to tests. (#12)

436da8f

* Add sync and async clients both to tests. Signed-off-by: Yury-Fridlyand <[email protected]> * Minor fixes. Signed-off-by: Yury-Fridlyand <[email protected]> --------- Signed-off-by: Yury-Fridlyand <[email protected]>

Move duplicated logic in benchmark JS scripts to a single file, and c…

016f5f6

…onvert to TypeScript. (valkey-io#456) removed duplicated logic and refactored to typescript Signed-off-by: acarbonetto <[email protected]>

Yury-Fridlyand and others added 3 commits October 3, 2023 13:35

Rename jabushka to javababushka (#14)

8664d05

Signed-off-by: acarbonetto <[email protected]>

Add missing renames. (#17)

b15f93e

Signed-off-by: Yury-Fridlyand <[email protected]>

acarbonetto force-pushed the java_benchmarks branch from 7122a23 to b15f93e Compare October 3, 2023 20:35

Yury-Fridlyand reviewed Oct 4, 2023

View reviewed changes

submodules/redis-rs Outdated

Copy link

Yury-Fridlyand Oct 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert?

Yury-Fridlyand and others added 13 commits October 5, 2023 14:01

Update redis-rs to match main branch

d99d27a

Signed-off-by: acarbonetto <[email protected]>

Create clients only once per iteration (#19)

540f49a

Signed-off-by: acarbonetto <[email protected]>

Updated ClientCount to client_count for uniformity.

65090b4

Convert client count to snake case for rust benchmark. (#27)

8a0449d

* Updated ClientCount to client_count for uniformity for rust.

Merge remote-tracking branch 'upstream/main' into java_benchmarks

fe9bb98

Signed-off-by: Yury-Fridlyand <[email protected]>

Typo fix.

d533b7f

Signed-off-by: Yury-Fridlyand <[email protected]>

Typo fix.

9cbc9c2

Signed-off-by: Yury-Fridlyand <[email protected]>

Merge branch 'java_benchmarks' of github.com:Bit-Quill/babushka into …

b00a205

…java_benchmarks Signed-off-by: Yury-Fridlyand <[email protected]>

fix java install_and_test script variables

a62fe92

Signed-off-by: acarbonetto <[email protected]>

Merge branch 'main' into java_benchmarks

30f2f62

Java benchmarks clusters (#34)

05590b0

* Add lettuce cluster client when cluster mode enabled --------- Signed-off-by: Andrew Carbonetto <[email protected]>

acarbonetto force-pushed the main branch from 8a0449d to 90c43fd Compare November 14, 2023 20:50

acarbonetto and others added 2 commits November 14, 2023 14:54

Clean up timer

bcf188c

Signed-off-by: acarbonetto <[email protected]>

Yury-Fridlyand closed this Dec 15, 2023

Yury-Fridlyand deleted the java_benchmarks branch December 15, 2023 18:33

cyip10 mentioned this pull request Jun 28, 2024

Java: new README.md #395

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Jedis and Lettuce benchmarks #1

Add Jedis and Lettuce benchmarks #1

jonathanl-bq commented Sep 7, 2023

Yury-Fridlyand commented Sep 7, 2023

Yury-Fridlyand Sep 15, 2023

jonathanl-bq Sep 15, 2023

Yury-Fridlyand Sep 21, 2023

Yury-Fridlyand Sep 21, 2023

Yury-Fridlyand Sep 21, 2023

Yury-Fridlyand Sep 30, 2023

Yury-Fridlyand Oct 4, 2023

Yury-Fridlyand commented Dec 15, 2023

		@@ -0,0 +1,3 @@
		[env]
		BABUSHKA_NAME = { value = "BabushkaPy", force = true }

Add Jedis and Lettuce benchmarks #1

Add Jedis and Lettuce benchmarks #1

Conversation

jonathanl-bq commented Sep 7, 2023

Yury-Fridlyand commented Sep 7, 2023

Yury-Fridlyand Sep 15, 2023

Choose a reason for hiding this comment

jonathanl-bq Sep 15, 2023

Choose a reason for hiding this comment

Yury-Fridlyand Sep 21, 2023

Choose a reason for hiding this comment

Yury-Fridlyand Sep 21, 2023

Choose a reason for hiding this comment

Yury-Fridlyand Sep 21, 2023

Choose a reason for hiding this comment

Yury-Fridlyand Sep 30, 2023

Choose a reason for hiding this comment

Yury-Fridlyand Oct 4, 2023

Choose a reason for hiding this comment

Yury-Fridlyand commented Dec 15, 2023