Valkey 8.0 multithreading - how to configure it correctly?

[valerymo]

Hello Valkey Community,
Can you please help to configure Valkey for multithreading.

This is my configuration - following was added to original valkey.conf
io-threads 4
io-threads-do-reads yes
events-per-io-thread 0

I tried to run valkey-benchmark with this configuration and without, but have similar results.
In logs I see "io_threads_active:1"m although in valkey.conf is "io-threads 4".
Seems something wrong in my configuration.
Can you please help

There are some logs below.

I tried two tests.

1. No threads
2. 4 treads configured in valkey.conf

• Valkey installed locally on Mac Sonoma arm64.
• Valkey compiled locally
  Valkey 8.0 (Valkey server 7.9.240)
  commit 85a5847 (HEAD -> 8.0, tag: 8.0.0-rc2, origin/8.0)

*** No threads ***

• Server:
  valkey/src ./valkey-server
  11076:C 11 Sep 2024 14:53:38.712 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo
  11076:C 11 Sep 2024 14:53:38.712 * Valkey version=7.9.240, bits=64, commit=85a58478, modified=0, pid=11076, just started
  11076:C 11 Sep 2024 14:53:38.712 # Warning: no config file specified, using the default config. In order to specify a config file use ./valkey-server /path/to/valkey.conf
  11076:M 11 Sep 2024 14:53:38.712 * Increased maximum number of open files to 10032 (it was originally set to 2560).
  11076:M 11 Sep 2024 14:53:38.712 * monotonic clock: POSIX clock_gettime

• another Xterm:
  valkey/src ./valkey-cli INFO |grep thread
  io_threads_active:0
  io_threaded_reads_processed:0
  io_threaded_writes_processed:0
  io_threaded_freed_objects:0
  io_threaded_poll_processed:0
  io_threaded_total_prefetch_batches:0
  io_threaded_total_prefetch_entries:0
  valkey/src git status
  On branch 8.0
  Your branch is up to date with 'origin/8.0'.

nothing to commit, working tree clean
valkey/src uname -a
Darwin vmogilev-mac 23.6.0 Darwin Kernel Version 23.6.0: Mon Jul 29 21:14:04 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T8122 arm64
valkey/src
valkey/src ./valkey-benchmark -t set -P 50 -n 100000000 --threads 4
====== SET ======
100000000 requests completed in 32.01 seconds
50 parallel clients
3 bytes payload
keep alive: 1
host configuration "save": 3600 1 300 100 60 10000
host configuration "appendonly": no
multi-thread: yes
threads: 4

Latency by percentile distribution:
0.000% <= 0.063 milliseconds (cumulative count 50)
50.000% <= 0.735 milliseconds (cumulative count 52612200)
......
99.998% <= 3.103 milliseconds (cumulative count 99998200)
100.000% <= 4.103 milliseconds (cumulative count 100000000)

Summary:
throughput summary: 3123536.00 requests per second
latency summary (msec):
avg min p50 p95 p99 max
0.734 0.056 0.735 0.823 0.863 3.927

*** Multithreading: valkey.conf : 4 threads ******
Following configuration was added to original valkey.conf
io-threads 4
io-threads-do-reads yes
events-per-io-thread 0

valkey/src ./valkey-server ../valkey.conf
11138:C 11 Sep 2024 14:59:59.370 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo
11138:C 11 Sep 2024 14:59:59.370 * Valkey version=7.9.240, bits=64, commit=85a58478, modified=0, pid=11138, just started
11138:C 11 Sep 2024 14:59:59.370 * Configuration loaded
11138:M 11 Sep 2024 14:59:59.370 * Increased maximum number of open files to 10032 (it was originally set to 2560).
11138:M 11 Sep 2024 14:59:59.370 * monotonic clock: POSIX clock_gettime
11138:M 11 Sep 2024 14:59:59.371 # Failed to write PID file: Permission denied
.+^+.
.+#########+.
.+########+########+. Valkey 7.9.240 (85a5847/0) 64 bit
.+########+' '+########+.

• Another xterm:
  valkey/src ./valkey-cli flushall
  OK
  valkey/src ./valkey-cli INFO |grep thread
  io_threads_active:1
  io_threaded_reads_processed:3
  io_threaded_writes_processed:1
  io_threaded_freed_objects:0
  io_threaded_poll_processed:3
  io_threaded_total_prefetch_batches:0
  io_threaded_total_prefetch_entries:0
  valkey/src
  valkey/src ./valkey-benchmark -t set -P 50 -n 100000000 --threads 4
  ====== SET ======
  100000000 requests completed in 52.35 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1
  host configuration "save": 3600 1 300 100 60 10000
  host configuration "appendonly": no
  multi-thread: yes
  threads: 4

Latency by percentile distribution:
0.000% <= 0.039 milliseconds (cumulative count 100)
50.000% <= 1.279 milliseconds (cumulative count 50386350)
.....
100.000% <= 9.103 milliseconds (cumulative count 99999650)
100.000% <= 11.103 milliseconds (cumulative count 100000000)

Summary:
throughput summary: 1910292.62 requests per second
latency summary (msec):
avg min p50 p95 p99 max
1.249 0.032 1.279 1.943 2.375 11.039

valkey/src ./valkey-cli INFO |grep thread
io_threads_active:1
io_threaded_reads_processed:2115746
io_threaded_writes_processed:2000003
io_threaded_freed_objects:99093990
io_threaded_poll_processed:132715
io_threaded_total_prefetch_batches:185015
io_threaded_total_prefetch_entries:1945379
valkey/src

[PingXie]

io-threads-do-reads yes
events-per-io-thread 0

I would suggest leaving them to default, which are yes and 2. Setting events-per-io-thread to 0 is mainly for testing.

How many cores are there on your Mac? It looks like you were running valkey-benchmark with 4 threads too and I don't see any efforts in isolating the two processes (via taskset). There is a possibility that the valkey-server process was not getting enough CPU cycles (you can validate via top).

[valerymo]

Thank you very much for reply. I tried recheck, but used "nice" instead of taskset (althought it's not exactly same, but taskset is missing for MAC). Results seems same when io-threads defined or not in valkey (now - it's the only that changed in default). But I can provide now more details for CPU and systems.
Can you please look if will have time. Thank you for Help!

[valerymo]

[valerymo]

This is result of test in Openshift, in POD (Linux, using taskset )

oc rsh backend-xxxx
$ uname -a
Linux backend-xxxx 5.14.0-427.35.1.el9_4.x86_64 1 SMP PREEMPT_DYNAMIC Fri Aug 30 15:47:10 EDT 2024 x86_64 GNU/Linux
$ nproc
4

Three tests done, for io-threads = 0(1), 2, 3
Example for checking threads num:
$ cat /etc/redis.d/valkey.conf |grep io-thre
$ 3 // or 2 or missing (if io-threads not defined in valkey.conf)
Commands (running in 2 xterms):
1st xterm: taskset -c 0,1 valkey-server ./etc/redis.d/valkey.conf
2nd xterm: taskset -c 2 valkey-benchmark -n 10000000

[ranshid]

@valerymo There are some issues I see with your test:

1. as @PingXie mentioned you should better set the events-per-io-thread to 2 (I think it is the default) otherwise it will caus the io-threads to keep spinning and utilize cpu for no good reason.
2. Your benchmark is using very deep pipelines. The CURRENT implementation of the io-threading is not optimal for deep pipelines since it will only parse the first command on the io-threads (you can see that we are able to consume a complete pipeline in a single syscall via io_threaded_reads_processed). I would suggest use more clients (250) and shorter pipelines (10)
3. your benchmark is using only a single key (no -r is configured). this practically makes our prefetching useless compared to the baseline without io-threads I would suggest to use a value closer to the number of requests like 50M.

[ranshid]

I'd like to ask - what means io_threads_active:1 in INFO (valkey-cli INFO)?

This is a boolean flag indicating the io-threads (in any number of them) where actively handling work.

[valerymo]

Thanks again for reply @ranshid . I did more testing in POD (comparing with and without io-threads defined). Could you please look if will have time. Thank you!

$ nproc
4
$ uname -a
Linux backend-xxxxxxxx xxxx.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Aug 30 15:47:10 EDT 2024 x86_64 GNU/Linux

****** Test 1 - 3 io-threads *****
$ cat valkey.conf
io-threads 3
events-per-io-thread 2
$ taskset -c 0,1,2 valkey-server /etc/redis.d/valkey.conf --port 6380
$ taskset -c 3 valkey-benchmark -t set -n 1000000 -c 250 -P 10 -r
avg min p50 p95 p99 max
18.242 0.248 9.863 61.023 64.415 83.775 -- r not set
14.221 0.160 8.495 50.975 53.471 64.543 -- r 1000
15.088 0.272 7.743 53.791 66.047 93.695 -- r 1000
5.299 0.120 2.231 36.031 50.175 100.159 -- r 1000
15.211 0.168 9.151 51.263 54.079 63.135 -- r 1000000
28.907 0.224 17.151 76.927 89.215 148.351 -- r 50000000

****** Test 2 - NO threads *******
$ valkey-server --port 6380
$ valkey-benchmark -t set -n 50000000 -c 250 -P 10
Summary:
throughput summary: 559916.69 requests per second
latency summary (msec):
avg min p50 p95 p99 max
3.971 1.216 3.759 5.911 7.351 25.679

[ranshid]

@valerymo

Lets try to better align the tests:

****** Test 1 - 3 io-threads *****
$ cat valkey.conf
io-threads 3
events-per-io-thread 2
$ taskset -c 0,1,2 valkey-server /etc/redis.d/valkey.conf --port 6380
$ taskset -c 3 valkey-benchmark -t set -n 1000000 -c 250 -P 10 -r 50000000

****** Test 2 - NO threads *******
$ taskset -c 0,1,2 valkey-server /etc/redis.d/valkey.conf --port 6380
$ taskset -c 3 valkey-benchmark -t set -n 50000000 -c 250 -P 10 -r 50000000

[valerymo]

@ranshid Thank you for reply and help. This is retest results

*** Tests - Compare Valkey with different configuration. In POD ***

• Test 1 - Three io-threads defined in valkey.conf
• Test 2 - No io-threads defined in valkey.conf
• Same commands used for both tests. Benchmark command repeted 3 times in both tests.
  $ taskset -c 0,1,2 valkey-server /etc/redis.d/valkey.conf --port 6380
  $ taskset -c 3 valkey-benchmark -t set -n 1000000 -c 250 -P 10 -r 50000000
• Pod is recreated for each test, so both tests are on "fresh" system.
• 4 cores
• OS Linux x86_64

===> Test 1: three io-threads
$ oc rsh backend-xxxx1
$ cat /etc/redis.d/valkey.conf |grep thread
io-threads 3
events-per-io-thread 2
$ date
Tue Sep 24 13:41:15 UTC 2024
$
avg min p50 p95 p99 max
13.244 0.104 7.175 51.743 56.767 76.031
14.873 0.224 8.327 55.391 63.743 77.119
14.881 0.128 8.751 53.087 60.319 88.255

===> Test 2 - NO threads
$ oc rsh backend-xxxx2
$ cat /etc/redis.d/valkey.conf |grep thread
$ date
Tue Sep 24 13:51:47 UTC 2024
$
avg min p50 p95 p99 max
6.315 1.720 4.919 23.135 29.279 36.895
6.991 1.832 5.471 22.687 28.559 32.431
7.655 2.008 7.223 11.103 18.479 32.479

[valerymo]

same in table view:

[ranshid]

@valerymo

1. I noticed this is an X86 machine. are there 4 physical cores or threads? (are you enabling hyperthreading?)
2. can you please tell me how your irq are pinned/configured?(eg smp_affinity, irqbalance etc...)?

[valerymo]

@ranshid Thank you for reply and help

1. I did tests/benchmarking on two systems:
   • Openshift 4.16 - POD with Linux x86_64 - nproc: 4
   • and on prem - on my MAC Sonoma arm64 with 8 cores.
     Results are similar on both systems - minimum latency if better when io-threads are configured,
     but average latency and other latency metrics are almost same, and max letency even can be worse.
     It's on both systems! Same behavior.
     There are several configuration for io-therads were tried:
     for 4 cores (x86_64) - no(1), 2 and 3
     for 8 cores (MAC arm64) - no(1), 3, 6.
     If compare MAC with 8 cores and POD/Linux x86_64 with 4 cores, - of causue results on MAC is much better,
     but again - same behavior for 1 or more threads - minimum latency is better, other - not
2. I used same configuration for Valkey as was used in Redis (redis.conf/valkey.conf), but I also used original valkey.conf.
   The only changes - definitions of the io-threads.
3. Can you please send me expected benchmarking testing results for several io-threads vs no(1) thread,
   and also your testing system details, and valkey.conf file used for your benchmarking.

Thanks very much for your help

[ranshid]

@valerymo

I tried to reproduce your benchmark on the 16 core ARM machine (AWS graviton2 machine)
I got a much better results with io-threads:

with io-threads:

Summary:
  throughput summary: 742390.50 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
        2.029     0.544     1.599     2.759     3.727    82.879

without:

Summary:
  throughput summary: 474833.81 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
        4.997     2.344     4.703     7.175     8.335    40.607

Can you maybe share the complete configuration file?

[valerymo]

Thank you @ranshid , Can you please send me commands you used and valkey.conf.
Your results show min - similar as for me, Max is same as for me - worse. x3 for p50 - yes, it's better
How many io-threads?
If you can't send me your valkey.conf, can you please tell me if it's differ from codebase.
Thanks again for your assistence.

[ranshid]

Well in this case I used the same as you indicated in the LAST case like:

cat /etc/redis.d/valkey.conf |grep thread
io-threads 3
events-per-io-thread 2

so my config file is:

io-threads 3
events-per-io-thread 2

I used the following commands:
with io-threads:

nohup taskset -c 0,1,2 ./src/valkey-server ~/valkey.conf --port 6379 &
taskset -c 3 ./src/valkey-benchmark -t set -n 1000000 -c 250 -P 10 -r 50000000

without io-threads:

nohup taskset -c 0,1,2 ./src/valkey-server ~/valkey.conf --port 6379 &
taskset -c 3 ./src/valkey-benchmark -t set -n 1000000 -c 250 -P 10 -r 50000000

[valerymo]

Thank you! My valkey.conf, - is original + io-threads definition:
~/go/valkey git diff valkey.conf
diff --git a/valkey.conf b/valkey.conf
.....
-# io-threads 4
+io-threads 3
+events-per-io-thread 2

.....
I had good performance results in absolute values on my MAC in short test, but 1 thread also worked perfect, no difference compating with io-threads. I could not find "taskset" for MAC, but I will try more and will update. Just wanted to summarize as it seems to me meanwhile -

• Valkey multithreading performence comparing to 1 thread is dependent to system; I didn't succed to see in my systems that it's working better than 1 thread. Tested in Openshift POD Linux x86_64 - 4 cores, and MAC arm64 8 cores.
• You sent me results of your testing on 16 core ARM machine (AWS graviton2 machine) - 3 io-threads are working better than 1 thread (except max latency).
  Thank you very much @ranshid

[melroy89]

Sorry but... io-threads-do-reads is even missing from the example config??

Is this option even supported in valkey? If so, why isn't it documented.

[zuiderkwast]

It was removed from documentation after #758. Was it a mistake?

Anyway, after that rewrite of the I/O threading, this config makes no sense, except maybe for testing. The default was changed to "yes" (enabled), but it is still possible to disable by setting it explicitly to "no". I guess this can cause some confusion to users. Disabling will be bad for performance because it will not let the I/O threads do all the I/O.

[melroy89]

It's indeed confusing.

1. Either remove the whole option from the code. And thus no need for documentation. OR:
2. Introduce the option back in the configuration, just as a comment is fine. And showing the default value "yes": # io-threads-do-reads yes with additional text.

[zuiderkwast]

#1116