SignalR.Kafka

An Apache Kafka backplane for ASP.NET Core SignalR

This project is largely based off of a fork of the SignalR Core Redis provider. It uses Kafka as the backplane to send SignalR messages between multiple servers, allowing for horizontal scaling of the SignalR implementation. The Confluent Kafka dotnet client is leveraged for publishing and consuming messages.

Kafka Configuration

Kafka topics may be created automatically if they don't yet exist depending on configuration. The default topic configuration uses 10 partitions and a replication factor of 1 unless specified in the startup options.

Topics may also be manually created in Kafka prior to running. The following schema is used. A partitioning strategy may be designed based off the key information provided for each topic.

• (prefix-)ack: Acknowledge group management messages: Key is server unique name
• (prefix-)group-mgmt: Group management messages (add/removal of connection from group): Key is server unique name
• (prefix-)send-all: Messages intended for all connected clients: No key is used, messages will be delivered round robin to partitions
• (prefix-)send-conn: Messages intended for a specific client connection. Key is connection id
• (prefix-)send-group: Messages intended for a specific group. Key is group name
• (prefix-)send-user: Messages intended for a specific user. Key is user id

Usage

1. Install the SignalR.Kafka NuGet package.
2. In ConfigureServices in Startup.cs, configure SignalR with .AddKafka():

.AddSignalR()
.AddKafka((options) =>
{
    options.ConsumerConfig = new ConsumerConfig
    {
        GroupId = $"{Environment.MachineName}_{Guid.NewGuid():N}",
        BootstrapServers = bootstrapServers,
        AutoOffsetReset = AutoOffsetReset.Latest,
        EnableAutoCommit = true
    };
    options.ProducerConfig = new ProducerConfig
    {
        BootstrapServers = bootstrapServers,
        ClientId = $"{Environment.MachineName}_{Guid.NewGuid():N}"
    };
});

The configuration for producer and consumer must be specified with options.ConsumerConfig and options.ProducerConfig. Configuration for an admin connection may optionally be provided to define connection options used by the AdminClient for topic creation. Topic creation will only be attempted if this configuration is provided:

.AddKafka((options) =>
{
    options.AdminConfig = new AdminConfig
    {
        BootstrapServers = bootstrapServers
    };
});

A topic prefix may be configured thru the KafkaTopicConfig object to allow for multiple instances of the schema on a single Kafka deployment:

.AddKafka((options) =>
{
    options.KafkaTopicConfig = new KafkaTopicConfig(topicPrefix: "my-prefix");
});

The KafkaTopicConfig may also be used for specifying initial topic creation specifications for each topic in the schema:

.AddKafka((options) =>
{
    options.KafkaTopicConfig = new KafkaTopicConfig(
        ackSpecification: new KafkaTopicSpecification
        {
            ReplicationFactor = 1,
            NumPartitions = 10
        },
        groupManagementSpecification: new KafkaTopicSpecification
        {
            ReplicationFactor = 1,
            NumPartitions = 10
        });
});

Performance Considerations

The Confluent Kafka client producer accumulates messages internally and sends them in batches. This supports high overall throughput, but sacrifices latency for individual message delivery. The latency and buffer sizes are configurable using Kafka ProducerConfig. See the Confluent documentation for more information.

By default, produce is called asyncrounously and the SignalR action (Send) returns prior to message delivery to the Kafka server. This allows for the highest throughput but comes with some risk that message delivery to the server may fail silently from client perspective (exceptions will still be logged). The configuration options allow changing the behavior to syncronously await produce call completion:

.AddKafka((options) =>
{
    options.AwaitProduce = true;
});

Any exceptions during the produce operation should bubble up to the client when AwaitProduce is true. However, this configuration will dramatically reduce throughput. See the Confluent documentation for more information on typical producer usage patterns.