How to handle reconnection on node teardown with multiple nodes scenario

[giovanni-bertoncelli]

Hi, I was wondering if it does exists any sort of mechanism that allows to recover a client connection when a backend replica/node gets torn down. In a orchestrator-oriented environment (like with kubernetes) each socket.io replica (even with active clients) may be torn down in any moment or at least when the scaling policy decides. I noticed that this particular scenario triggers (correctly) a transport close disconnect event client side. In this case the client probably believes that the server is no more reachable and the connection dies permanently.
Is there any connection recovery mechanism for this scenario or is it charged to only the client to retry connecting? It may be a good idea to consider this scenario for some possibile evolution of socket.io?
I've seen this page: https://socket.io/docs/v4/connection-state-recovery, but I do not think it may be useful in this case.
Thank you!

[giuliopulina]

Hi @giovanni-bertoncelli, I have a very limited experience in socket.io but I'm doing some experiments with connection state recovery, using a cluster of Socket.io servers (Node.js with Haproxy), connected to a Mongodb cluster through the Mongo adapter.

In my simple project, connection state recovery works only in case of a temporary disconnection between client and server: in this situation, the socket.id remains the same, socket object is marked as recovered and the messages that have been sent to the client, while it was not was connected, are correctly received upon reconnection.
However, tearing down a Socket.io node doesn't result in the same behavior: a new connection with another node is created, and the messages are lost. Recovering missing events for such a case seems to be covered in delivery-guarantees_from-server-to-client docs page, but it's not clear to me how this strategy integrates with connection state recovery. (EDIT: this tutorial https://socket.io/docs/v4/tutorial/step-7 clarifies a lot of things!)

In any case, it's really possible that I'm not doing everything correctly.
I have to say that I found the docs about connection state recovery a bit unclear. It's not clear which type of recovery are supported and which are not, and some other details are missing (for example, I couldn't understand what the flag 'skipMiddlewares' is about).

One last note: this remark is very important. The recovery of a client that has not been 'initialized' with at least one message is not supported.

[giovanni-bertoncelli]

Hi @giuliopulina, thank you for your reply. However my problem is not on client disconnection but instead when a server instance gets torn down. But for now I think the only way is to restore the connection manually on client side.

[giuliopulina]

If I understood well: you have clients connected to a node that get torn down (maybe due to autoscaling policies, maybe because the node becomes unhealthy).
So, now you have clients disconnected, probably getting tons of errors in the JS console (or in the UI), and not able to use your application. Am I correct?

In this scenario, if you don't need to synchronize the state of the client upon reconnection (i.e. you can live with lost messages), reconnection upon "transport close" disconnection is just a matter of configuration (see 'reconnection*' options in https://socket.io/docs/v4/client-options/. Also check disconnect docs for details about disconnect events.)

Obviously, you have to configure a cluster of nodes with sticky session (https://socket.io/docs/v4/using-multiple-nodes) but I think you are already doing this.

Or maybe I'm missing something?

[giovanni-bertoncelli]

No, thank you. That's it! When I wrote the discussion I probably didn't have a clear vision of the problem 👍🏻