Watchdog keeps connecting to the nodes which were removed from the Redis cluster.

[Eddie9922]

I removed a node from Redis cluster, however Watchdog keep trying to connect to the removed nodes. My assumption is that the connection to the removed nodes should be closed after the topology refresh. We are using our own topology refresh logic, we override the loadViews function. I feel we might miss something. Can someone point me to the logic in the open source code? Thanks!

[tishun]

Hey @Eddie9922
Have you checked the node filtering feature (see #1942)? Does this help?

[Eddie9922]

This is where the removed nodes' connection should be closed. I added some log, I do saw the removed nodes were closed.

[PooledClusterConnectionProvider] found stale key ConnectionKey [connectionIntent=WRITE, nodeId='null', host='10.94.51.169', port=25692]

However, later the watchdog will still try to reconnect it. The below log is from here, I changed it a bit.

[Time:2024-09-25T03:08:02.250889][ConnectionWatchdog][channel=0x6ff5ce98, /10.156.110.32:51966 -> 10.94.51.169/10.94.51.169:25692, last known addr=10.94.51.169/10.94.51.169:25692] Cannot reconnect to [10.94.51.169:25692]: finishConnect(..) failed: Connection refused: /10.94.51.169:25692` 

I also noticed this only happen when I replace a primary node in the cluster.

Lettuce version: 6.2
Redis version: 6.0

[tishun]

I removed a node from Redis cluster

Can you elaborate on what you mean by that. Is the node present in the results if you call CLUSTER NODES, or is it removed from the topology altogether? What actions do you partake in order to remove the node?

We are using our own topology refresh logic, we override the loadViews function.

Are you at liberty to show us this topology refresh logic? A (minimal) way to reproduce the issue would help a lot.

I do saw the removed nodes were closed.

I am not sure that closing the connection is enough. If we have the node still listed in the topology view the driver would simply attempt to establish a new connection.

Lettuce version: 6.2

There might be some issue with older versions that I am not aware of (e.g. #1909 which should be fixed in 6.2).
Have you tried updating the driver just in case?

I believe we need to verify the list of nodes after the topology has changed.

• First verify that sending CLUSTER NODES does not list the removed node from another client, e.g. redis-cli
• Second verify that - after the custom topology refresh logic is applied - the node is not present in the Partitions view