Support Redis 7 sharded PUB/SUB

[EmandM]

Our team are looking into solutions for automating adding redis capacity with no downtime (we are less concerned with automatic downscaling, but that would also be nice to have). We are wondering if this is something anyone has experience or thoughts on? All the examples in this repository seem to assume all redis capacity is known on centrifuge startup.

We are also wondering if you have run benchmarks with the different configurations of redis broker found here https://centrifugal.github.io/centrifugo/server/engines/#redis-engine or if you have any opinions on which performs better.

[FZambia]

Hello, are you looking for a way to reconfigure Redis engine without application restart? If you can afford restart then things should work out of the box.

You are pointing to the old Centrifugo doc btw - make sure it's intentional.

Regarding Redis performance - it can vary depending on your load profile, impossible to say - usually one Redis instance can handle several thousand requests per second and more (up to several hundred thousand). But some ops are heavier than others.

I expect separate client-side sharded non-cluster Redis setup to perform better. Though Redis cluster may he simplier to operate in terms of HA setup. It's possible to shard over different Redis clusters too.

[FZambia]

Also, Centrifuge library has some Redis-related benchmarks in https://github.com/centrifugal/centrifuge/blob/master/broker_redis_test.go

[EmandM]

We are looking for ways to reconfigure Redis engine without application restart. We cannot really afford to restart the system to add capacity.

[FZambia]

I suppose the simplest solution in your case is using Redis Cluster. Redis Cluster can scale independently from the WebSocket application - no app reconfiguration required. Centrifuge already supports it. In the future we can also add support for sharded pubsub added recently in Redis 7.0 - this improves PUB/SUB scalability in Redis Cluster.

Regarding performance – most probably cluster will perform worse than single standalone Redis instance in terms of raw latencies, but it scales for non PUB/SUB ops. To scale PUB/SUB in Redis Cluster we need to support sharded pubsub mentioned above (or use client-side sharding between different Redis Clusters - already available but requires manual re-configuration). Though maybe PUB/SUB throughput which can be achieved with cluster without sharded pubsub will be more than enough for you. Unfortunately I can't provide any numbers at the moment - you may test yourself for your workload.

[FZambia]

Also, https://centrifugal.dev/blog/2020/02/10/million-connections-with-centrifugo can give some insights to estimate numbers possible to achieve with Centrifuge and Redis for some artificial workload. I suppose you have already seen it though. And it does not involve Redis Cluster.

[EmandM]

Sorry for the slow reply. The pubsub sharding looks like it will be exactly what we need. Thank you for the heads up about that!

[tony-pang]

Hi @FZambia, tyvm for the suggestions. Any idea when Redis 7 will be supported? thanks :)

[FZambia]

@tony-pang hello, you mean sharded pubsub right? Not in the near future I suppose, except someone comes with pr.

I am currently spend all the time for the next iteration of Centrifuge client connectors (new API, standardized states, adopting client protocol v2 recently introduced in experimental form). Also our own bidirectional emulation layer.

Wondering whether you already met PUB/SUB performance bottleneck of single Redis Cluster. Or maybe you have estimations that Redis Cluster without sharded PUB/SUB won't fit your use case – would be awesome to have a real use case to solve with sharded PUB/SUB support.

[tony-pang]

no worries, I'm asking because you mentioned in a previous response. We have not met any bottleneck in this regard yet. I believe we will reach there eventually, but not in near future. When we do have this issue, we can further discuss this topic. Again, thank you very much for the suggestions

[FZambia]

In #262 I added sharded PUB/SUB support, but then before merging made it non-public. The problem is that enabling sharded PUB/SUB results into using different Redis channels and keys compared to what we have in basic Redis Cluster scenario. It also requires setting a reasonable number of cluster slots Centrifuge will use for channels and history – which I think requires more thinking how to document right. At this point though it seems that it's the only viable approach to have sharded PUB/SUB as we need to deal with millions of channels – so we need a stateful PUB/SUB connection which operates with subset of channels belonging to the same slot of Redis Cluster.

If someone have a real use case where sharded PUB/SUB is required - please write to me and we can discuss. Or probably I'll make sharded PUB/SUB public a bit later if I'll be confident the implementation is OK.

[EmandM]

Our team was hoping to use sharded PUB/SUB. We've switched to another option for now. We decided on an approach with multiple deployments that allowed us to add capacity without downtime. This means don't need sharded PUB/SUB right now, though we might still be interested in exploring it in the future. For now we're happy to wait until the implementation is more stable. Thank you so much for working on this!

[ekaterinamzr]

@FZambia, Hi, is there any news on this issue?

[FZambia]

@ekaterinamzr hello, mostly same state as described in #228 (comment)

Some new insights with benchmarks can be found in this discussion - centrifugal/centrifugo#757

I don't want to proceed without working closely with a real use case regarding this feature.