[Spike] Pre-requisite for fault-tolerant client-server

[mshabarov]

Describe your motivation

This is a pre-requisite investigation task before we can start #20348.
The description there is not enough to start the development and it costs too many story points from the team's opinion, thus needs a decomposition.

The outcome of this investigation should be:

• Describing and understanding what are the cases might be, when message ID is unexpected. This should give enough technical details to deduct what and where exactly we have to change in framework . This ticket can be considered as a source of real observations Unexpected message id from the client (sync ids different) #12640, but not only limited to those.
• A list of enhancements with <= 8 story points that could address the cases, where the network is a culprit, i.e. when the messages are lost. The cases, caused by misconfiguration of session replication, should be described, but out of scope of further enhancements in framework. Though they should be properly documented.
• A testing procedure needs to be ready, at least the idea how we can test these enhancements and guarantee that they cover real cases.

[caalador]

Cases seen in reported logs:

• Ignore old message from client
  • Server receives previous message id with the same content hash as last processed
  • Server should resend last response to sync state on client.
• Unexpected id from client
  • If the client id is bigger than the expected there has been a dropped client request. (client updates the id on preparePayload)
  • Client should be requested to resend previous message and server should sync the id to current request id.
• Server expected much bigger than the client request?
  • Resynchronization should probably happen as the gap is too big and why is the client requesting with much too old id?

[caalador]

As the server always answers the client, the client should probably queue the sends as it does for one single pushPendingMessage when using bidirectional transport, but not for xhr. Also send(JsonObject payload) is public and doesn't do any queueing so that sendInvocationsToServer would purge them instead of pushing it out immediately where we could possibly get faulty sync id from the client.