Add: Idempotency strategy

[mtrezza]

Issue

Several issues* have been opened over the years regarding idempotency in case of:

• unreliable network connections
• blocking ISPs
• network retry mechanisms that are out of reach for the application layer
• app network interface access management specifically on mobile OS

The basic problem is that the client intends to send a request to the server and for different reasons the server receives the request multiple times. That leads to objects being saved multiple times, Cloud Code function being executed multiple times, etc.

None of the vast discussions resulted in a PR, due to the complexity of the issue, which led to some people using customized solutions, mostly with Cloud Code triggers. There are several possible solutions which primarily differ in their tradeoffs between cost and efficacy and the desired efficacy varies depending on the use case for Parse Server. It should be noted that this issue is not specific to Parse Server but a common networking issue.

Suggested solution

Process:

1. Client generates request UUID
2. Client sends UUID in request header
3. Server tries to write UUID into a table where UUID is a unique field
4. If the UUID write succeeds, the server processes the request
5. If the UUID write fails due to duplicate key, the server does not process the request and sends a 4xx response that signals to the SDK that the request has been discarded. It is the developer's responsibility to handle such a response gracefully.

Example:

1. Clients sends request to save a new chat message object
2. Server receives the request and creates the object
3. Server receives another identical request and sends a 400 response with Parse Error DuplicateRequest
4. Client gets response and knowns something went wrong, therefore it reloads the chat message list from the server to account for any inconsistencies

Context:

• The feature is optional, off by default (due to the cost overhead for data transfer, processing)
• The client has an option to send a request UUIDs
• The server has an option to process request UUIDs
• Requests without UUID are always processed (for backwards compatibility)
• The TTL of UUID entries in the lookup table is configurable, 5 mins by default
• The DB table table is in cache or in the DB in a private class, not displayed in Parse Dashboard
• Identical request can arrive within milliseconds, therefore a lookup in a table for already processed requests can be too slow. The solution is to use a table with a unique field where DB itself does not allow duplicate values. This is possible in MongoDB with a unique index. This also requires only 1 write request in the majority of cases where the request is not a duplicate.
• Ideal idempotent behavior would be to send a 2xx response on repeated requests, however since responses are not cached, we cannot easily reconstruct a response, for example a created objectId.

Alternative solutions

• Changing the number of network retries and timeout settings of the client can mitigate the issue but is ineffective in some scenarios.

Additional context

*Similar issues:

• Duplicate records for the a single saveInBackground Call because Internet provider block response #4006
• ParseObject saved multiple times on slow internet connection Parse-SDK-Android#764

Similar solutions:

• https://github.com/panda-clouds/idempotency

TODOs

• Changes in Parse Server Add idempotency #6748

[mtrezza]

To give an update about the current status of this feature request:

The heavy lifting is done as the necessary changes are implemented in Parse Server. The next step would be to add the x-parse-request-id header with a request UUID to each of the client SDKs. The SDK should have an initialization option whether to send the header.

Since we have a number of SDKs, whoever feels like adding this to an SDKs please feel free to go ahead and submit a PR. Examples using the REST API are test cases in the Idempotency.spec.js.

Things to consider:

• The Parse JS SDK will be the first SDK to implement the option to send the header, so other SDKs should name the options accordingly for consistency.
• By default, the option to send the header should be disabled.
• The UUID of a request should not change when automatically retrying the same request due to network issues.
• Parse Server does not enforce idempotency for GET requests. Idempotency is enforced only on POST, PUT routes as GET, DELETE are already idempotent. The header therefore should only be sent on POST and PUT requests.

[mtrezza]

I am closing this as the Parse Server related PR is finished.

Practically speaking, the client SDKs for Android and iOS are still lacking the PRs to send the request ID, as has already been done with the JS SDK. These PRs should ideally contain a reference to his issue for readability.