DataStore bug: If a newly-created item fails to get synced on the first try because of a network error, it never gets synced properly after that.

[dragosiordachioaia]

Before opening, please confirm:

• I have searched for duplicate or closed issues and discussions.
• I have read the guide for submitting bug reports.
• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

JavaScript Framework

React

Amplify APIs

DataStore

Amplify Categories

auth, api

Environment information

# Put output below this line

System:
    OS: macOS 12.1
    CPU: (8) arm64 Apple M1
    Memory: 129.13 MB / 16.00 GB
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 16.13.2 - /usr/local/bin/node
    npm: 8.5.4 - ~/repos/inspections-tool/node_modules/.bin/npm
  Browsers:
    Chrome: 99.0.4844.83
    Firefox: 96.0.2
    Safari: 15.2
  npmPackages:
    @ant-design/icons: ^4.2.2 => 4.7.0 
    @aws-amplify/datastore: ^3.7.9 => 3.7.9 
    @elastic/elasticsearch: ^8.0.0 => 8.1.0 
    @testing-library/jest-dom: ^5.16.1 => 5.16.2 
    @testing-library/react: ^12.1.2 => 12.1.4 
    @testing-library/user-event: ^13.5.0 => 13.5.0 
    antd: ^4.18.5 => 4.19.1 
    aws-amplify: ^4.3.17 => 4.3.17 
    aws-sdk: ^2.1070.0 => 2.1092.0 
    axios: ^0.25.0 => 0.25.0 (0.21.4)
    bir-vue:  0.1.0 
    browser-image-resizer: ^2.2.1 => 2.2.1 
    classnames: ^2.3.1 => 2.3.1 
    elasticsearch: ^15.5.0 => 15.5.0 
    fuzzysort: ^1.1.4 => 1.2.1 
    http-aws-es: ^6.0.0 => 6.0.0 
    i: ^0.3.7 => 0.3.7 
    idb-keyval: ^6.1.0 => 6.1.0 
    js-cookie: ^3.0.1 => 3.0.1 (2.2.1)
    lodash: ^4.17.21 => 4.17.21 
    md5: ^2.3.0 => 2.3.0 
    moment: ^2.29.1 => 2.29.1 
    npm: ^8.4.1 => 8.5.4 
    react: ^17.0.2 => 17.0.2 
    react-device-detect: ^2.1.2 => 2.1.2 
    react-dom: ^17.0.2 => 17.0.2 
    react-helmet: ^6.1.0 => 6.1.0 
    react-jsonschema-form: ^1.8.1 => 1.8.1 
    react-responsive-carousel: ^3.2.22 => 3.2.23 
    react-router-dom: ^6.2.1 => 6.2.2 
    react-scripts: 5.0.0 => 5.0.0 
    react-webcam: ^6.0.1 => 6.0.1 
    sass: ^1.49.0 => 1.49.9 
    shortid: ^2.2.16 => 2.2.16 
    urlencode: ^1.1.0 => 1.1.0 
    use-debounce: ^7.0.1 => 7.0.1 
    uuid: ^8.3.2 => 8.3.2 (3.4.0, 3.3.2)
    web-vitals: ^2.1.3 => 2.1.4 
    webpack: ^5.69.1 => 5.70.0 
  npmGlobalPackages:
    @aws-amplify/cli: 7.6.22
    aws-azure-login: 3.4.0
    corepack: 0.10.0
    npm: 8.4.0
    serve: 13.0.2

Describe the bug

Creating items in DataStore fails if there is bad connectivity. When it comes to the availability of the internet connection, there are 3 main scenarios:

1. completely offline - DataStore works correctly
2. online with good connection - DataStore works correctly
3. online with spotty/slow/high-latency connection - the issue manifests itself

In short, if a "create" mutation has already started processing and then it fails for a network-related reason and then the internet connection goes offline altogether, when the connection returns, there is an Unrecoverable DataStore error message from DataStore with a ConditionalCheckFailedException coming from DynamoDB. At this point, the mutation is removed from the queue and it never gets retried, which means that item is never going to be properly synchronised.

I've looked in DynamoDB and the item does get created, which explains the ConditionalCheckFailedException we get back.

Expected behavior

If there is spotty connectivity, I expect that when good connectivity is re-established, DataStore would perform a sync that lets it know the item has actually been successfully synchronised, which should make it update its local database with the correct state.

Reproduction steps

1. Create a high-latency network profile in Google Chrome and switch to it
2. Trigger a "create" mutation from the front-end.
3. After a few seconds, switch to the "Offline" network profile in Google Chrome.
4. Again after a few seconds, switch to the "No throttling" network profile in Google Chrome.
5. There is now an Unrecoverable DataStore error message from DataStore with a ConditionalCheckFailedException coming from DynamoDB.
6. The newly-created item doesn't gets a value set for its updatedAt field.

Code Snippet

// Put your code below this line.

Log output

No response

aws-exports.js

No response

Manual configuration

No response

Additional configuration

No response

Mobile Device

No response

Mobile Operating System

No response

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response

[david-mcafee]

May possibly be fixed by this PR. Testing with the latest release.

[dragosiordachioaia]

@david-mcafee I already have that change and unfortunately it doesn't fix it. I've even tried to make a fork of the DataStore repo to completely remove code that dequeues items and all that resulted in was DataStore kept retrying the item forever, with DynamoDB returning the same error.

[dragosiordachioaia]

I've just realised that the "Unrecoverable DataStore error" message is a log I had added to my own code, but that doesn't change the underlying issue. Just putting this here because it's probably confusing to see that error message in the issue description, since the wording isn't actually coming from within DataStore itself.

[svidgen]

@dragosiordachioaia We're seeing this from at least one other customer as well. But, any additional information you can provide would be very helpful.

Is there any chance you can provide scrubbed request and response bodies for me to look at?

[chrisbonifacio]

Hi @dragosiordachioaia and @jameswillis99 👋 We haven't been able to reproduce this issue with the reproduction steps given on the latest version of aws-amplify. Going to close this as stale, but if you are experiencing this issue please let us know or provide any additional context/info you might've gathered since. We can re-open this issue once we have more information.