Akka.NET v1.4.19: ChannelExecutor performance data

[Aaronontheweb]

In Akka.NET v1.4.19 we will be introducing an opt-in feature, the ChannelExecutor - a new dispatcher type that re-uses the same configuration as a ForkJoinDispatcher but runs entirely on top of the .NET ThreadPool and is able to take advantage of dynamic thread pool scaling to size / resize workloads on the fly.

You can read the initial development pull request, commentary, and benchmark results here: #4882

During its initial development and benchmarks, we observed the following:

1. The ChannelExecutor tremendously reduced idle CPU and max busy CPU even during peak message throughput, primarily as a result of dynamically shrinking the total ThreadPool to only the necessary size. This resolves one of the largest complaints large users of Akka.NET have today. However, in order for this setting to be effective ThreadPool.SetMin(0,0) must also be set. We are considering doing this inside the ActorSystem.Create method (edit: this is now done inside ActorSystem.Create automatically), but if you those settings don't work for you you can easily override them by simply calling ThreadPool.SetMin(yourValue, yourValue) again after ActorSystem.Create has exited.
2. The ChannelExecutor actually beat the ForkJoinDispatcher and others on performance even in environments like Docker and bare metal on Windows.

We are looking to gather more feedback from real-world use from users who require high performance from their Akka.NET applications.

If you could please comment on this discussion thread with the following data:

• Your dispatcher configurations - https://getakka.net/articles/actors/dispatchers.html will show you how to enable the ChannelExecutor everywhere
• Your observed performance (i.e. throughput, CPU consumption) before and your observed performance after; and
• Your general degree of satisfaction with the feature.

This information would be very helpful for us in determining how best to performance-tune Akka.NET and its dispatching system in the future. Thank you for your support!

[Zetanova]

I made now some test with a new ChannelExecutor and looks a bit better.
Main benefit would be that it is running without ThreadPool.SetMinThreads(0, 0);

see my commit: Zetanova@cc7e3b1

It will run without the ThreadPool.SetMinThreads manipulation with 1-2 Threads and idling it should have very low CPU.

[Zetanova]

I make a new commit for the ChannelTaskScheduler implemntation
https://github.com/Zetanova/akka.net/tree/channeltaskscheduler

It uses a Timer and does have view magic numbers to limit new workers to max +2 for each 250ms

[Aaronontheweb]

Got this using your latest:

[ERROR][5/10/2021 11:08:58 PM][Thread 0006][Cluster (akka://ClusterSys)] Failed to startup Cluster. You can try to increase 'akka.actor.creation-timeout'.
Cause: Akka.Actor.AskTimeoutException: Timeout after 00:00:20 seconds
   at Akka.Actor.Futures.Ask(ICanTell self, Func`2 messageFactory, IActorRefProvider provider, Nullable`1 timeout, CancellationToken cancellationToken)
   at Akka.Actor.Futures.Ask[T](ICanTell self, Func`2 messageFactory, Nullable`1 timeout, CancellationToken cancellationToken)
   at Akka.Cluster.Cluster.GetClusterCoreRef()

Both nodes timed out this way while attempting to start up the /system actors.

[Zetanova]

I tested it now four times without issues: https://github.com/Zetanova/akka.net/tree/channeltaskscheduler
2x with old bins and 2x with new bins with/without ThreadPool.SetMinThreads(0, 0)

Could u please test again?

message throughput should not be the only perf measurement.
What we would need in the ClusterPingPongTest some Timings of the ping with Min,Max,Mean
and a new test to measure reaction time when a single message is sent in a view seconds interval
like an icmp ping.

I don't think that the 'channel' makes better msg/s but it is a clean way to custom limit the parallelization
without global hacks like ThreadPool.SetMinThreads(0, 0)

current channelexecutor with ThreadPool.SetMinThreads(0, 0):

OSVersion:                         Unix 5.4.72.2                                          
ProcessorCount:                    6                                                      
Node Count:                        2                                                      
Active Connections:                1                                                      
Msgs sent/received per actor:      200000  (2e5)                                          
Is Server GC:                      True                                                   
Thread count per node:             30                                                     
                                                                                          
Connections, Actors/node, Total [actor], Total [msg], Msgs/sec, Total [ms], Threads       
       1,        5,             10,    1,000,000,     37,284,   26820.49,    5 -> 41      
       1,       10,             20,    2,000,000,     46,365,   43135.61,    4 -> 32      
       1,       15,             30,    3,000,000,     48,327,   62077.03,    2 -> 32      
       1,       20,             40,    4,000,000,     50,884,   78609.09,    3 -> 31      
       1,       25,             50,    5,000,000,     51,720,   96673.80,    3 -> 31      
       1,       30,             60,    6,000,000,     54,878,  109331.54,    2 -> 31      
Benchmark complete                                                                        

current channelexecutor WITHOUT ThreadPool.SetMinThreads(0, 0):

OSVersion:                         Unix 5.4.72.2                                               
ProcessorCount:                    6                                                           
Node Count:                        2                                                           
Active Connections:                1                                                           
Msgs sent/received per actor:      200000  (2e5)                                               
Is Server GC:                      True                                                        
Thread count per node:             29                                                          
                                                                                               
Connections, Actors/node, Total [actor], Total [msg], Msgs/sec, Total [ms], Threads            
       1,        5,             10,    1,000,000,     39,441,   25353.87,    3 -> 38           
       1,       10,             20,    2,000,000,     48,531,   41210.76,    3 -> 32           
       1,       15,             30,    3,000,000,     49,923,   60092.18,    3 -> 31           
       1,       20,             40,    4,000,000,     53,215,   75165.50,    3 -> 31           
       1,       25,             50,    5,000,000,     52,670,   94929.02,    3 -> 31           
       1,       30,             60,    6,000,000,     53,453,  112246.43,    3 -> 31           
Benchmark complete                                                                             

channeltaskscheduler with ThreadPool.SetMinThreads(0, 0):

OSVersion:                         Unix 5.4.72.2
ProcessorCount:                    6
Node Count:                        2
Active Connections:                1
Msgs sent/received per actor:      200000  (2e5)
Is Server GC:                      True
Thread count per node:             30

Connections, Actors/node, Total [actor], Total [msg], Msgs/sec, Total [ms], Threads
       1,        5,             10,    1,000,000,     40,136,   24914.86,    4 -> 38
       1,       10,             20,    2,000,000,     47,308,   42276.12,    3 -> 32
       1,       15,             30,    3,000,000,     52,970,   56635.26,    2 -> 31
       1,       20,             40,    4,000,000,     56,046,   71369.06,    3 -> 31
       1,       25,             50,    5,000,000,     55,169,   90629.47,    3 -> 31
       1,       30,             60,    6,000,000,     47,900,  125260.83,    2 -> 32
Benchmark complete

channeltaskscheduler WITHOUT ThreadPool.SetMinThreads(0, 0):

OSVersion:                         Unix 5.4.72.2                                             
ProcessorCount:                    6                                                         
Node Count:                        2                                                         
Active Connections:                1                                                         
Msgs sent/received per actor:      200000  (2e5)                                             
Is Server GC:                      True                                                      
Thread count per node:             30                                                        
                                                                                             
Connections, Actors/node, Total [actor], Total [msg], Msgs/sec, Total [ms], Threads          
       1,        5,             10,    1,000,000,     35,831,   27908.32,    3 -> 38         
       1,       10,             20,    2,000,000,     46,892,   42650.30,    3 -> 32         
       1,       15,             30,    3,000,000,     52,091,   57591.42,    3 -> 31         
       1,       20,             40,    4,000,000,     53,987,   74090.84,    3 -> 31         
       1,       25,             50,    5,000,000,     52,733,   94817.13,    3 -> 31         
       1,       30,             60,    6,000,000,     50,822,  118057.05,    3 -> 31         
Benchmark complete                                                                           

[Zetanova]

I fixed some bugs in my Priority ChannelTaskScheduler.
It consume in idle a bit more then a idle mssql instance
On load it will prio the system workload.
Runs good/better and stable on my cluster.

It can be packed in an extension and no changes to akka are required.

Should I just make a nuget package under what Prefix/Name ?

[Aaronontheweb]

Go ahead and create an Akka.Experimental.* package - that would be a good place to review it.

Sorry for the delay in reviewing this - I'm going to be out of the office for a little bit starting on Monday for a paternity leave and I got way behind in reviewing Akka.NET issues over the past few weeks while I was getting ready for it.

[Aaronontheweb]

Unrelated, but just found this delightful gem in the .NET Core repos:

dotnet/runtime#51935 (comment)

Looks like hill-climbing is affected by a signed integer overflow bug...

[Zetanova]

I tried now the channelexecutor of v1.4.20
I reduces the idle cpu of an 6-node drastically from >16% each to 6-8% each (mssql and eventstore have 2-4%)

But now i am getting "heartbeat interval is growing too large ... >2700ms" warnings
and got one forced disassociated, both an higher load/work.

[Zetanova]

ChannelTaskScheduler Extension

• Priority: High, Normal, Low, Idle
• Singleton

New Branch: https://github.com/Zetanova/akka.net/tree/channel-priority-task-scheduler

Idle CPU and Internal-Dispatcher problem should be resolved.
Internal-Dispatcher and remote-dispatcher is running on High,
Lower Priorities then High will only afterwards be processed.

custom config:

akka.channel-scheduler {
    parallelism-min = 2
    parallelism-factor = 1
    parallelism-max = 64,
    work-max = 10
}

akka.actor.default-dispatcher = {
    executor = channel-executor
    channel-executor.priority = "normal"
}

akka.actor.internal-dispatcher = {
    executor = channel-executor   
    throughput = 5
    channel-executor.priority = "high"
}

akka.remote.default-remote-dispatcher {
    type = Dispatcher
    executor = channel-executor
    channel-executor.priority = "high"
}

akka.remote.backoff-remote-dispatcher {
  executor = channel-executor
  channel-executor.priority = "low"
}

PriorityChannelTaskScheduler WITHOUT ThreadPool.SetMinThreads(0, 0):

OSVersion:                         Unix 5.4.72.2                                         
ProcessorCount:                    6                                                     
Node Count:                        2                                                     
Active Connections:                1                                                     
Msgs sent/received per actor:      200000  (2e5)                                         
Is Server GC:                      True                                                  
Thread count per node:             30                                                    
                                                                                         
Connections, Actors/node, Total [actor], Total [msg], Msgs/sec, Total [ms], Threads      
       1,        5,             10,    1,000,000,     28,961,   34528.35,    4 -> 39     
       1,       10,             20,    2,000,000,     43,516,   45959.22,    3 -> 31     
       1,       15,             30,    3,000,000,     45,172,   66412.19,    2 -> 31     
       1,       20,             40,    4,000,000,     45,687,   87550.86,    3 -> 31     
       1,       25,             50,    5,000,000,     46,271,  108057.41,    3 -> 31     
       1,       30,             60,    6,000,000,     39,694,  151153.20,    3 -> 31     
Benchmark complete          

It is a bit worse then before.
I set the channels options SingleWriter = false don't know why it worked before without error.

[Aaronontheweb]

I'll need to run that PR on my machine to compare it to the baseline, which is about to change with the ThreadPool.SetMinThreads setting being reverted.

So we'd have to have a fixed number of channels / channel registrations for this to work? A structure similar to what we do in the mailbox for prioritizing ISystemMessage ahead of user-land messages?

[Zetanova]

There are 4 channels for 4 priorities.
High => Executed until none left for system messages
Normal => normal user-land
Low => sometimes executed or when there where no high/normal work in batch
Idle => Only executed if no high, normal or low work was done.

It is now a singleton extensions and all workers are shared between all dispatchers.

There can be some optimizations for sure and i don't thing that it will improve max msg/s
But it will solve the idle, "reaction under idle" and "reaction under load" problems
and should have no problem with low and high cpu/thread count.

[Zetanova]

@Aaronontheweb Pls, could u run/test the singleton scheduler?
Branch: https://github.com/Zetanova/akka.net/tree/channel-priority-task-scheduler

Is there a why to add a custom dispatcher to akka.net without forking the lib?

[Aaronontheweb]

Yep, I'll give that a try. Sorry - been tied up working on our own Akka.NET applications recently!

[chris-sung]

The ChannelExecutor tremendously reduced idle CPU and max busy CPU even during peak message throughput, primarily as a result of dynamically shrinking the total ThreadPool to only the necessary size. This resolves one of the largest complaints large users of Akka.NET have today. However, in order for this setting to be effective ThreadPool.SetMin(0,0) must also be set. We are considering doing this inside the ActorSystem.Create method (edit: this is now done inside ActorSystem.Create automatically), but if you those settings don't work for you you can easily override them by simply calling ThreadPool.SetMin(yourValue, yourValue) again after ActorSystem.Create has exited.

I think this is too risky. I'm running a very large-scale service and have messed the service up with ThreadPool a lot, this kind of undocumented, unexpected "hidden" changes will cost a lot to discover the problem in a production environment.

Also, for other use cases, like, If someone combines their ASP.net service and actor system that would be a disaster. this is a typical pattern of rapidly growing small to medium scale service. I've been there. :'(

The min thread count benefits from reducing scheduling overhead (also there a lock inside this scheduler..) by aggressively reducing thread count but also this means thread pool will create a thread on demand, which will cause allocation overhead and lock contention.

If this is a such problem, how about creating your own dedicated thread pool for this?

[Aaronontheweb]

Alright, we'll roll this back for v1.4.21.

[Aaronontheweb]

It also seems that 1.4.20 doesn't play nicely with the default akka settings

Which settings does v1.4.20 not play nicely with?

[boris-rg]

the default dispatcher settings.. although, using the new channel-executor settings did not work either.. so something seems generally broken

[Aaronontheweb]

The only default HOCON settings we changed were in v1.4.18. Haven't touched that file since.

The other dispatcher changes we made were just to add channel-executor as a config option for dispatcher loading itself: https://github.com/akkadotnet/akka.net/pull/4882/files

We did change the remote.conf to use a higher thread count for the fork-join-executor but any user values would still override that.

[boris-rg]

not sure, might have been a red herring.. after rolling back to 1.4.18 the cluster still decides to down random nodes.. I'll play around with it some more

[Aaronontheweb]

Reverting the ThreadPool.SetMinThreads(0,0) call here #5059

[Aaronontheweb]

I have finally gotten around to trying @Zetanova's ChannelDispatcher implementation and I am very pleased with the results on .NET 5

Dev

OSVersion:                         Microsoft Windows NT 10.0.19041.0
ProcessorCount:                    16
ClockSpeed:                        0 MHZ
Actor Count:                       32
Messages sent/received per client: 200000  (2e5)
Is Server GC:                      True
Thread count:                      111

Num clients, Total [msg], Msgs/sec, Total [ms]
         1,  200000,    157233,    1272.65
         5, 1000000,    212766,    4700.54
        10, 2000000,    212970,    9391.40
        15, 3000000,    210956,   14221.79
        20, 4000000,    211663,   18898.04
        25, 5000000,    210731,   23727.46
        30, 6000000,    210726,   28473.71

@Zetanova / ChannelDispatcher

OSVersion:                         Microsoft Windows NT 10.0.19041.0
ProcessorCount:                    16                               
ClockSpeed:                        0 MHZ                            
Actor Count:                       32                               
Messages sent/received per client: 200000  (2e5)                    
Is Server GC:                      True                             
Thread count:                      109                              
                                                                    
Num clients, Total [msg], Msgs/sec, Total [ms]                      
         1,  200000,    152672,    1310.75                          
         5, 1000000,    197668,    5059.51                          
        10, 2000000,    195008,   10256.66                          
        15, 3000000,    195810,   15321.39                          
        20, 4000000,    197678,   20235.66                          
        25, 5000000,    195580,   25565.47                          
        30, 6000000,    201492,   29778.18                          

N.B. @Zetanova's branch doesn't benefit from the significant performance optimizations introduced to Akka.Remote in Akka.NET v1.4.21 and he hasn't been able to update that branch yet due to #5185 - that probably explains the performance gap between the two thus far.

[Aaronontheweb]

Welp, I'm an idiot. I had assumed the benchmarks were updated to run the channel-dispatcher but they were not. I need to go redo this :p

So the difference in my two performance numbers is solely explained by the v1.4.21 performance changes.

[Aaronontheweb]

Using config:

akka {
  channel-scheduler{
    parallelism-min = 2
    parallelism-factor = 1
    parallelism-max = 64
  }
  actor.provider = remote
  loglevel = ERROR
  suppress-json-serializer-warning = on
  log-dead-letters = off

  remote {
    log-remote-lifecycle-events = off

    dot-netty.tcp {
        port = 0
        hostname = ""localhost""
    }
    
  }
}

akka.actor.default-dispatcher = {
    executor = channel-executor
    channel-executor.priority = normal
}

akka.actor.internal-dispatcher = {
    executor = channel-executor
    throughput = 5
    channel-executor.priority = high
}


akka.remote.default-remote-dispatcher {
    type = Dispatcher
    executor = channel-executor
    channel-executor.priority = high
    fork-join-executor {
      parallelism-min = 2
      parallelism-factor = 0.5
      parallelism-max = 16
    }
}

akka.remote.backoff-remote-dispatcher {
  executor = channel-executor
  channel-executor.priority = low
}

I got these numbers:

OSVersion:                         Microsoft Windows NT 10.0.19041.0
ProcessorCount:                    16
ClockSpeed:                        0 MHZ
Actor Count:                       32
Messages sent/received per client: 200000  (2e5)
Is Server GC:                      True
Thread count:                      47

Num clients, Total [msg], Msgs/sec, Total [ms]
[ERROR][8/9/2021 6:08:10 PM][Thread 0004]
         1,  200000,     52549,    3806.07
         5, 1000000,     95521,   10469.23
        10, 2000000,    108150,   18493.45
        15, 3000000,    112516,   26663.22
        20, 4000000,    115205,   34721.09
        25, 5000000,    115878,   43149.61
        30, 6000000,    120137,   49943.35

[Aaronontheweb]

What sorts of values are you getting @Zetanova ?

[Zetanova]

I will make the Akka.Experimental.ChannelScheduler Lib as discussed soon,
it will be a standalone extension.

There are view little changes to the code.
One big one is "parallelism-min = 4" till akka 1.4.21
4 active threads are required to start a cluster node.

I am using it already in production and it decreased the idle CPU load
and reduced the "cluster lag warnings" a lot.

[Aaronontheweb]

Please include in your branch / NuGet package some default configuration - that would be very helpful/

[to11mtm]

Running into some issues trying this with Akka.Persistence.Linq2db Perf tests.

            akka.persistence {{
                dispatchers {{
    channel-executor = {{
        executor = channel-executor
        fork-join-executor {{ #channelexecutor will re-use these settings
          parallelism-min = 2
          parallelism-factor = 1
          parallelism-max = 16
        }}
    }}
}}

If I set my plugin-dispatcher (and materializer-dispatcher) to this, I see a few things happening (I've played with other variations including min/max of (1,4)/8 and permutations thereof)

• Recovery benchmarks all fail with timeout.
• Persist Single benchmark fails with timeout, the numbers that I do sometimes get before failure look like a good improvement.
• PersistAll benchmark fails with timeout...sometimes
• PersistAsync and PersistAsync pass. hmmmmmmmmmm
• PersistGroup throughputs look pretty good. Sometimes better, sometimes worse. I'd say they are 'within margin' since docker VMs for db benchies is kinda inconsistent (at least on my box).

What's really throwing me off on this, is that default-plugin-dispatcher is a dedicated single thread. We don't see issues when we use default-dispatcher either.

For whatever it is worth, we are using a wide variety of async within Akka Streams pipelines, as well as TaskCompletionSource. Sometimes we might be a bit mean to it. I know we do some questionable things on the Read pipeline (maybe we should rethink that). But the Single write pipeline is not different enough from the multi-write to really explain what's happening there.

[Zetanova]

@Aaronontheweb
I made now the repo at https://github.com/Zetanova/Akka.Experimental.ChannelTaskScheduler

I am using it with akka 1.4.21 prodction without issues.

Maybe or maybe not the throughput is higher
But the idle CPU is for sure lower.
Reaction time and work load increase is very low

On my manuel tests, most of the time a single worker will execute everything.

[ismaelhamed]

@Aaronontheweb what's the status of the ChannelExecutor, given this thread is rather old?

[Aaronontheweb]

The status of it is that we like the concept and the results it produces in terms of lower CPU consumption, but throughput is still a concern. We're going to revisit some of the design decisions here to see if we can improve throughput.