Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DataStore] ReferenceError: Property 'err' doesn't exist #12824

Open
3 tasks done
dmitryusikriseapps opened this issue Jan 10, 2024 · 20 comments
Open
3 tasks done

[DataStore] ReferenceError: Property 'err' doesn't exist #12824

dmitryusikriseapps opened this issue Jan 10, 2024 · 20 comments
Assignees
Labels
DataStore Related to DataStore category pending-maintainer-response Issue is pending a response from the Amplify team. question General question React Native React Native related issue

Comments

@dmitryusikriseapps
Copy link

dmitryusikriseapps commented Jan 10, 2024

Before opening, please confirm:

JavaScript Framework

React Native

Amplify APIs

Authentication, DataStore, Storage

Amplify Version

v6

Amplify Categories

auth, storage, api

Backend

None

Environment information

# Put output below this line
  System:
    OS: macOS 14.2.1
    CPU: (10) arm64 Apple M1 Max
    Memory: 1.15 GB / 32.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 16.19.1 - ~/.nvm/versions/node/v16.19.1/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 9.8.0 - ~/.nvm/versions/node/v16.19.1/bin/npm
    Watchman: 2023.09.25.00 - /opt/homebrew/bin/watchman
  Browsers:
    Chrome: 120.0.6099.199
    Safari: 17.2.1
  npmPackages:
    @abelmkr/abelmkr-app-config: 1.7.0 => 1.7.0 
    @abelmkr/amplify: 6.6.5 => 6.6.5 
    @abelmkr/appsync-data-utils: 6.6.5 => 6.6.5 
    @abelmkr/appsync-job-rate-transformer: 6.6.5 => 6.6.5 
    @abelmkr/general-shared-utils: 2.3.1 => 2.3.1 
    @abelmkr/job-rate-calculator: 3.5.1 => 3.5.1 
    @abelmkr/job-rate-types: 3.5.1 => 3.5.1 
    @abelmkr/money: 3.4.1 => 3.4.1 
    @aws-amplify/react-native: 1.0.10 => 1.0.10 
    @azure/core-asynciterator-polyfill: ^1.0.2 => 1.0.2 
    @babel/core: ^7.21.3 => 7.21.3 (7.20.5)
    @babel/preset-typescript: ^7.21.0 => 7.21.0 (7.18.6)
    @commitlint/cli: ^17.4.4 => 17.4.4 
    @elevai/commitlint-config-github: ^0.1.1 => 0.1.1 
    @elevai/commitlint-plugin-github: ^0.1.1 => 0.1.1 
    @expo/config-plugins: ~6.0.0 => 6.0.1 
    @formatjs/intl-datetimeformat: ^6.5.1 => 6.5.1 
    @formatjs/intl-displaynames: ^6.2.6 => 6.2.6 
    @formatjs/intl-getcanonicallocales: ^2.1.0 => 2.1.0 
    @formatjs/intl-listformat: ^7.1.9 => 7.1.9 
    @formatjs/intl-locale: ^3.1.1 => 3.1.1 
    @formatjs/intl-numberformat: ^8.3.5 => 8.3.5 
    @formatjs/intl-pluralrules: ^5.1.10 => 5.1.10 
    @formatjs/intl-relativetimeformat: ^11.1.10 => 11.1.10 
    @gorhom/bottom-sheet: ^4.4.5 => 4.4.5 
    @hookform/resolvers: ^2.9.11 => 2.9.11 
    @hookform/resolvers/ajv:  1.0.0 
    @hookform/resolvers/class-validator:  1.0.0 
    @hookform/resolvers/computed-types:  1.0.0 
    @hookform/resolvers/io-ts:  1.0.0 
    @hookform/resolvers/joi:  1.0.0 
    @hookform/resolvers/nope:  1.0.0 
    @hookform/resolvers/superstruct:  1.0.0 
    @hookform/resolvers/typanion:  1.0.0 
    @hookform/resolvers/vest:  1.0.0 
    @hookform/resolvers/yup:  1.0.0 
    @hookform/resolvers/zod:  1.0.0 
    @react-native-async-storage/async-storage: ~1.17.3 => 1.17.11 
    @react-native-community/datetimepicker: 6.7.3 => 6.7.3 
    @react-native-community/masked-view: 0.1.11 => 0.1.11 
    @react-native-community/netinfo: 9.3.7 => 9.3.7 
    @react-native-community/viewpager: 5.0.11 => 5.0.11 
    @react-navigation/material-bottom-tabs: ^6.2.15 => 6.2.15 
    @react-navigation/native: ^6.1.6 => 6.1.6 
    @react-navigation/stack: ^6.3.16 => 6.3.16 
    @reduxjs/toolkit: ^1.9.3 => 1.9.3 
    @reduxjs/toolkit-query:  1.0.0 
    @reduxjs/toolkit-query-react:  1.0.0 
    @sentry/types: 7.20.1 => 7.20.1 (6.19.2, 6.19.7)
    @shopify/flash-list: 1.4.0 => 1.4.0 
    @swc/core: ^1.3.56 => 1.3.56 
    @swc/jest: ^0.2.26 => 0.2.26 
    @testing-library/jest-native: ^5.4.2 => 5.4.2 
    @testing-library/react-hooks: ^8.0.1 => 8.0.1 
    @testing-library/react-native: ^12.0.0 => 12.0.0 
    @types/bad-words: ^3.0.1 => 3.0.1 
    @types/color: ^3.0.3 => 3.0.3 
    @types/deep-equal: ^1.0.1 => 1.0.1 
    @types/glob: ^8.1.0 => 8.1.0 
    @types/jest: ^29.5.0 => 29.5.0 
    @types/node: ^18.15.3 => 18.15.3 (18.11.9)
    @types/prompts: ^2.4.3 => 2.4.3 
    @types/react: ~18.0.27 => 18.0.28 (18.0.25)
    @types/react-native: ~0.71.3 => 0.71.3 
    @types/react-redux: ^7.1.25 => 7.1.25 
    @types/redux: ^3.6.0 => 3.6.0 
    @types/redux-persist: ^4.3.1 => 4.3.1 
    @types/sinon: ^10.0.13 => 10.0.13 
    @types/uuid: ^8.3.4 => 8.3.4 (9.0.7)
    @typescript-eslint/eslint-plugin: ^5.55.0 => 5.55.0 
    @typescript-eslint/parser: ^5.55.0 => 5.55.0 (5.44.0)
    @zeplin/cli: ^2.0.1 => 2.0.1 
    @zeplin/cli-connect-react-plugin: ^1.1.1 => 1.1.1 
    HelloWorld:  0.0.1 
    aws-amplify: 6.0.11 => 6.0.11
    aws-amplify/adapter-core:  undefined ()
    aws-amplify/analytics:  undefined ()
    aws-amplify/analytics/kinesis:  undefined ()
    aws-amplify/analytics/kinesis-firehose:  undefined ()
    aws-amplify/analytics/personalize:  undefined ()
    aws-amplify/analytics/pinpoint:  undefined ()
    aws-amplify/api:  undefined ()
    aws-amplify/api/server:  undefined ()
    aws-amplify/auth:  undefined ()
    aws-amplify/auth/cognito:  undefined ()
    aws-amplify/auth/cognito/server:  undefined ()
    aws-amplify/auth/enable-oauth-listener:  undefined ()
    aws-amplify/auth/server:  undefined ()
    aws-amplify/datastore:  undefined ()
    aws-amplify/in-app-messaging:  undefined ()
    aws-amplify/in-app-messaging/pinpoint:  undefined ()
    aws-amplify/push-notifications:  undefined ()
    aws-amplify/push-notifications/pinpoint:  undefined ()
    aws-amplify/storage:  undefined ()
    aws-amplify/storage/s3:  undefined ()
    aws-amplify/storage/s3/server:  undefined ()
    aws-amplify/storage/server:  undefined ()
    aws-amplify/utils:  undefined ()
    axios: ^1.6.5 => 1.6.5 (0.21.4)
    babel-loader: ^9.1.2 => 9.1.2 
    babel-preset-expo: ^9.3.0 => 9.3.0 (9.3.2)
    bad-words: ^3.0.4 => 3.0.4 
    color: ^4.2.3 => 4.2.3 (3.2.1)
    dayjs: ^1.11.7 => 1.11.7 (1.11.6)
    deep-equal: ^2.2.0 => 2.2.0 
    dpdm: ^3.13.1 => 3.13.1 
    eslint: ^8.36.0 => 8.36.0 (8.28.0)
    eslint-config-prettier: ^8.7.0 => 8.7.0 
    eslint-import-resolver-typescript: ^3.5.3 => 3.5.3 
    eslint-plugin-destructuring: ^2.2.1 => 2.2.1 
    eslint-plugin-import: ^2.27.5 => 2.27.5 
    eslint-plugin-jest: ^27.2.1 => 27.2.1 
    eslint-plugin-jest-formatting: ^3.1.0 => 3.1.0 
    eslint-plugin-prettier: ^4.2.1 => 4.2.1 
    eslint-plugin-react: ^7.32.2 => 7.32.2 
    eslint-plugin-react-hooks: ^4.6.0 => 4.6.0 
    eslint-plugin-react-native: ^4.0.0 => 4.0.0 
    eslint-plugin-simple-import-sort: ^10.0.0 => 10.0.0 
    eslint-plugin-sonarjs: ^0.18.0 => 0.18.0 
    expo: ^48.0.21 => 48.0.21 
    expo-app-loading: ~2.1.1 => 2.1.1 
    expo-application: ~5.1.1 => 5.1.1 
    expo-background-fetch: ~11.1.1 => 11.1.1 
    expo-constants: ~14.2.1 => 14.2.1 
    expo-crypto: ~12.2.1 => 12.2.1 
    expo-dev-client: ~2.2.1 => 2.2.1 
    expo-device: ~5.2.1 => 5.2.1 
    expo-image-manipulator: ~11.1.1 => 11.1.1 
    expo-image-picker: ~14.1.1 => 14.1.1 
    expo-linking: ~4.0.1 => 4.0.1 
    expo-localization: ~14.1.1 => 14.1.1 
    expo-location: ~15.1.1 => 15.1.1 
    expo-notifications: ~0.18.1 => 0.18.1 
    expo-sensors: ~12.1.1 => 12.1.1 
    expo-splash-screen: ~0.18.1 => 0.18.2 (0.17.5)
    expo-sqlite: ~11.1.1 => 11.1.1 
    expo-status-bar: ~1.4.4 => 1.4.4 
    expo-system-ui: ~2.2.1 => 2.2.1 
    expo-task-manager: ~11.1.1 => 11.1.1 
    expo-updates: ~0.16.4 => 0.16.4 
    expo-web-browser: ~12.1.1 => 12.1.1 
    geolib: ^3.3.3 => 3.3.3 
    husky: ^8.0.3 => 8.0.3 
    i18next: ^22.4.12 => 22.4.12 
    jest: ^29.2.1 => 29.5.0 
    jest-expo: ^48.0.2 => 48.0.2 
    jest-fetch-mock: ^3.0.3 => 3.0.3 
    json-diff: ^1.0.3 => 1.0.3 
    libphonenumber-js: ^1.10.24 => 1.10.24 
    libphonenumber-js/build:  undefined ()
    libphonenumber-js/core:  undefined ()
    libphonenumber-js/max:  undefined ()
    libphonenumber-js/max/metadata:  undefined ()
    libphonenumber-js/min:  undefined ()
    libphonenumber-js/min/metadata:  undefined ()
    libphonenumber-js/mobile:  undefined ()
    libphonenumber-js/mobile/examples:  undefined ()
    libphonenumber-js/mobile/metadata:  undefined ()
    lint-staged: ^13.2.0 => 13.2.0 
    patch-package: ^6.5.1 => 6.5.1 
    prettier: ^2.8.4 => 2.8.5 (2.8.0)
    prettier-eslint: ^15.0.1 => 15.0.1 
    prompts: ^2.4.2 => 2.4.2 
    prop-types: ^15.8.1 => 15.8.1 
    react: 18.2.0 => 18.2.0 
    react-hook-form: 7.43.6 => 7.43.6 
    react-i18next: ^12.2.0 => 12.2.0 
    react-native: 0.71.14 => 0.71.14 
    react-native-animatable: ^1.3.3 => 1.3.3 
    react-native-gesture-handler: ~2.9.0 => 2.9.0 
    react-native-get-random-values: ~1.9.0 => 1.9.0 
    react-native-google-places-autocomplete: ^2.5.1 => 2.5.1 
    react-native-maps: 1.3.2 => 1.3.2 
    react-native-modal: 13.0.1 => 13.0.1 
    react-native-paper: ~5.4.1 => 5.4.1 
    react-native-reanimated: ~2.14.4 => 2.14.4 
    react-native-safe-area-context: 4.5.0 => 4.5.0 
    react-native-screens: ~3.20.0 => 3.20.0 
    react-redux: ^8.0.5 => 8.0.5 
    redux: ^4.2.1 => 4.2.1 (4.2.0)
    redux-persist: ^6.0.0 => 6.0.0 
    redux-persist/integration/react:  undefined ()
    sanitize-filename: ^1.6.3 => 1.6.3 
    sentry-expo: ~4.2.0 => 4.2.0 
    snapshot-diff: ^0.10.0 => 0.10.0 
    sort-json: ^2.0.1 => 2.0.1 
    ts-node: ^10.9.1 => 10.9.1 
    typescript: ^4.9.4 => 4.9.5 (4.9.3, 5.2.2)
    use-debounce: ^9.0.3 => 9.0.3 
    uuid: ^9.0.0 => 9.0.0 (8.3.2, 3.4.0, 7.0.3)
    yup: ^1.0.2 => 1.0.2 
  npmGlobalPackages:
    @aws-amplify/cli: 11.0.3
    corepack: 0.15.1
    detox-cli: 20.0.0
    eas-cli: 3.10.0
    expo-cli: 6.3.10
    npm: 9.8.0
    pnpm: 8.6.6
    serve: 14.2.1
    turbo: 1.10.8

Describe the bug

As soon as some DataStore mutations are called with a break near 1-2 seconds between them, then an unhandled promise is thrown. Then if the application is reloaded, the warning will still be shown. If the current user is signed out and authenticated one more time, then an uncaught error will be thrown. Everything was working well using the 5th version of the aws-amplify library.
Simulator Screenshot - iPhone 15 Plus - 2024-01-10 at 11 30 34
Simulator Screenshot - iPhone 15 Plus - 2024-01-10 at 11 45 16

Expected behavior

No warnings or errors should be thrown.

Reproduction steps

  1. Run a few mutations one by one with a break between then 1-2 seconds.
  2. See a warning which is described above.

Code Snippet

// Put your code below this line.

Log output

// Put your logs below this line


aws-exports.js

No response

Manual configuration

No response

Additional configuration

No response

Mobile Device

iPhone 15 Plus

Mobile Operating System

iOS 17.0

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response

@dmitryusikriseapps dmitryusikriseapps added the pending-triage Issue is pending triage label Jan 10, 2024
@dmitryusikriseapps
Copy link
Author

dmitryusikriseapps commented Jan 10, 2024

It happens during this DataStore event:

 LOG  2024-01-10 15:40:35.448+02:00 [DATASTORE / ApplicationStack_Synced] DataStore event received:
 LOG  {
  "event": "outboxStatus",
  "data": {
    "isEmpty": false
  }
}

@chrisbonifacio chrisbonifacio self-assigned this Jan 10, 2024
@danrivett
Copy link

danrivett commented Jan 10, 2024

Update: The error I report below, we now believe may be a separate DataStore bug. So read the following accordingly.

A bit more information on this as I'm working with @dmitryusikriseapps on this:

When I attempt to save a model type (Worker in our example) through DataStore in our RN app, it works fine for me as expected.

When I tried to use the app to update a Worker's name twice in fairly quick succession, in my testing I couldn't recreate the error above that way.

However when I updated the code to make two back to back DataStore.save() calls I consistently get an error, but it's slightly different from the one reported above. I get:

 WARN  [WARN] 17:47.730 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]

For reference, I updated our code to do:

let updatedWorker = await DataStore.save(
      DataStoreModels.Worker.copyOf(remoteWorker, updated => { ... })
);

// Duplicate the update here to show the error:
updatedWorker = await DataStore.save(
      DataStoreModels.Worker.copyOf(remoteWorker, updated => { ... })
);

Interestingly if I add a sleep in between the two DataStore.save() calls I still get the error, no matter how long I sleep for (I tried up to 15s):

const sleep = (ms: number): Promise<void> => {
  return new Promise(resolve => {
    setTimeout(resolve, ms);
  });
};

log(`updateCurrentWorker(): Saving to DataStore`);
let updatedWorker = await DataStore.save(
      DataStoreModels.Worker.copyOf(remoteWorker, updated => { ... })
);

await sleep(15000);

log(`updateCurrentWorker(): Saving to DataStore`);
updatedWorker = await DataStore.save(
      DataStoreModels.Worker.copyOf(remoteWorker, updated => { ... })
);

Perhaps importantly I can confirm that the DataStore outbox is evented as empty before the 2nd mutation and we still get the problem, so it doesn't seem related to that at least.

  LOG  2024-01-10 07:17:29.288-08:00 [WORKER] updateCurrentWorker(): Saving to DataStore
 LOG  2024-01-10 07:17:29.324-08:00 [DATASTORE / ApplicationStack_Synced] DataStore event received:
 LOG  {
  "event": "outboxStatus",
  "data": {
    "isEmpty": false
  }
}
 LOG  2024-01-10 07:17:29.860-08:00 [DATASTORE / ApplicationStack_Synced] DataStore event received:
 LOG  {
  "event": "outboxStatus",
  "data": {
    "isEmpty": true
  }
}
 LOG  2024-01-10 07:17:44.322-08:00 [WORKER] updateCurrentWorker(): Saving to DataStore
 LOG  2024-01-10 07:17:44.345-08:00 [DATASTORE / ApplicationStack_Synced] DataStore event received:
 LOG  {
  "event": "outboxStatus",
  "data": {
    "isEmpty": false
  }
}
WARN  [WARN] 17:44.998 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]
WARN  [WARN] 17:45.485 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]
WARN  [WARN] 17:45.712 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]
WARN  [WARN] 17:46.81 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]
WARN  [WARN] 17:46.326 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]
WARN  [WARN] 17:46.529 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]
WARN  [WARN] 17:46.883 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]
WARN  [WARN] 17:47.244 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]
WARN  [WARN] 17:47.412 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]
WARN  [WARN] 17:47.730 DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]
LOG  2024-01-10 07:17:48.176-08:00 [DATASTORE / ApplicationStack_Synced] DataStore event received:
LOG  {
 "event": "outboxStatus",
 "data": {
   "isEmpty": true
 }
}

I am hopeful this error is quite reproducible on your end, but next step is I'm going to check the AppSync logs in CloudWatch and I'll report back on anything interesting I find there.

@danrivett
Copy link

Looking more into my error directly above, I think it's a separate issue to the one @dmitryusikriseapps reported, but a bug in its own right that I'd like to see fixed.

Regarding the original ReferenceError: Property 'err' doesn't exist error, we're still trying to narrow down a easy recreate, but we can recreate it when we have a function that invokes DataStore.save() on one model and then immediately invokes DataStore.delete() on another model to clear out some stale data.

I'm wondering if this issue is related to the React Native environment, and the Hermes engine in particular, as some initial searching indicates it may be.

Here is the where the error is coming from:

Not being able to reference err seems odd, and makes me think it may be Hermes specific?

We'll try and provide as simple a problem recreate as possible, as we'd really like to see any bug identified fixed, as this is totally blocking our migration to v6.

@danrivett
Copy link

danrivett commented Jan 11, 2024

@chrisbonifacio I've spent quite a bit of time trying to narrow down the problem recreation scenario to help recreate this bug in as simple as manner possible.

What I found is I could reliably recreate both the ReferenceError: Property 'err' doesn't exist error, as well as the DataStore - conflict trycatch [TypeError: WeakSet key must be an Object] error with just a simple change.

Note: I recreated this in a React Native application, and the former error in particular may only be recreatable there, I'm not sure.

I am hopeful the instructions below should allow you or others on the team (perhaps @iartemiev ?) to recreate these too with minimal effort. Both @dmitryusikriseapps and I are happy to help as currently this bug(s) is blocking us from migrating to v6 and we'd really like to do that.

Problem Recreation Scenarios

The code below works on a simple @model type (ContractorJobInfo) and operates in 2 primary different scenarios

  1. When a matching ContractorJobInfo doesn't already exist
    • In this case it first creates a new ContractorJobInfo object in DataStore, and then updates it.
  2. When a matching ContractorJobInfo does already exist
    • In this case it first updates the existing ContractorJobInfo object retrieved from DataStore, then updates it again.

These two scenarios are then repeated but a 5 second sleep is added between the two DataStore mutation operations above.

This gives 4 different scenaros so far.

Finally, I noticed that adding a Pipeline Resolver for the createContractorJobInfo and updateContractorJobInfo which sets a field itself causes different test results, and so the 4 different scenarios are repeated both with these pipeline resolvers active, and without.

This leads to 8 total test scenarios that I've tested and documented my findings.

Problem Recreation Code

const sleep = (ms: number): Promise<void> => {
  return new Promise(resolve => {
    setTimeout(resolve, ms);
  });
};

let jobInfo = (await DataStore.query(DataStoreModels.ContractorJobInfo, info => info.jobId.eq('dummy-id')))[0];

if (jobInfo == null) {
  console.log(`Creating ContractorJobInfo`);

  jobInfo = await DataStore.save(
    new DataStoreModels.ContractorJobInfo({
      contractorId: currentContractor.id,
      cognitoUserId: currentContractor.cognitoUserId,
      jobId: 'dummy-id',
    })
  );
} else {
  console.log(`Updating ContractorJobInfo initially`);

  jobInfo = await DataStore.save(
    DataStoreModels.ContractorJobInfo.copyOf(jobInfo, updated => {
      updated.contractorId = `changed-${dayjs().toISOString()}`;
    })
  );
}

// Some of the test scenarios sleep between DataStore operations, others don't
// This is to see how adding a delay (for outbox syncing) affects things

// await sleep(5000);

console.log(`Updating ContractorJobInfo`);

await DataStore.save(
  DataStoreModels.ContractorJobInfo.copyOf(jobInfo, updated => {
    updated.contractorId = 'overwritten';
  })
);

Pipeline Resolvers

As I mentioned above we have some additional custom PipelineResolvers for createContractorJobInfo and updateContractorJobInfo which add some audit fields keeping track of who last edited a model type as we have some administrators who can modify other people's ContractorJobInfo records and we want the server (rather than the client) to keep track of who is editing it. Each update is then logged separately to a CDC bucket so we can see who edited which version:

Mutation.createContractorJobWorkerInfo.preUpdate.1.req.vtl

This is simplified from what we have, but illustrates the important part.

## We don't allow audit fields to be set manually, we set them
$util.qr($context.args.input.remove("lastEdited"))

#if( $util.authType() == "User Pool Authorization" )
  #set( $currentUserId = $util.defaultIfNull($ctx.identity.claims.get("username"), $util.defaultIfNull($ctx.identity.claims.get("cognito:username"), null)) )
  #set( $now = $util.time.nowISO8601() )
  #set( $currentVersion = $util.defaultIfNull($context.args.input["_version"], 0) )

  $util.qr( $context.args.input.put("createdBy", $currentUserId) )

  ## The 'lastEdited' fields refer to the last time a type was created/updated by a human and not the BE
  ## This includes both a regular user who has access, as well as an internal admin user
  #set( $lastEdited = {
    "cognitoUserId": $currentUserId,
    "at": $now,
    "versionEdited": $currentVersion
  } )

  $util.qr( $context.args.input.put("lastEdited", $lastEdited) )
#end

Mutation.updateContractorJobWorkerInfo.preUpdate.1.req.vtl

Again this is simplified from what we have.

## We don't allow audit fields to be set manually, we set them
$util.qr($context.args.input.remove("lastEdited"))

#if( $util.authType() == "User Pool Authorization" )
  #set( $currentUserId = $util.defaultIfNull($ctx.identity.claims.get("username"), $util.defaultIfNull($ctx.identity.claims.get("cognito:username"), null)) )
  #set( $now = $util.time.nowISO8601() )
  #set( $currentVersion = $util.defaultIfNull($context.args.input["_version"], 0) )

  #set( $lastEdited = {
    "cognitoUserId": $currentUserId,
    "at": $now,
    "versionEdited": $currentVersion
  } )

  $util.qr( $context.args.input.put("lastEdited", $lastEdited) )
#end

I'll add the test results in the following comment as this comment is already very large.

@danrivett
Copy link

danrivett commented Jan 11, 2024

Problem Recreation Results

Based on the code above and 8 different test scenarios I've described, I can consistently recreate both the ReferenceError: Property 'err' doesn't exist error, as well as the DataStore - conflict trycatch [TypeError: WeakSet key must be an Object] error.

Though I'll state immediately that I could only recreate the ReferenceError: Property 'err' doesn't exist error when the custom Pipeline Resolver was active that updated a field itself.

So we'll start when custom Pipeline Resolvers are active:

1. With Custom Pipeline Resolvers Active

Note: The custom pipeline resolver must modify the data for it to recreate the results below.

1.1 With no sleep(5000):

1.1.1 When no existing record exists:

Result: No error is returned, but although the initial create works, I do not see the subsequent update written to DynamoDB.
The contractorId field is not set to overwritten and _version remains 1 from the initial create and isn't incremented to 2.

1.1.2 When an existing record exists:

This can be tested by repeating the same test again after 1.1.1 as there's now a matching ContractorJobInfo record persisted.

Result: No error is returned, but similar to 1.1.1, although the initial update works, I do not see the 2nd update written to DynamoDB.
The contractorId field is set to changed-... and is not set to overwritten, and _version is incremented to 2 not 3.

So far these test results don't show either of the two errors reported above, but they do highlight a new 3rd error - a silent missing update. Just as important if not more so (as it's a silent error).

1.2 With sleep(5000) in between:

1.2.1 When no existing record exists:

Result: The initial create succeeds, but the subsequent update fails with a single attempt and the error: ReferenceError: Property 'err' doesn't exist.

1.2.2 When an existing record exists:

Result: The initial update succeeds, but the subsequent update fails with 10 attempts all failing with: DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]

At this point the test scenarios above all reliably recreate both the originally reported errors, plus the new silent error. This should be all that's needed to recreate all of them, but for completeness I'll include the other test scenario results. Especially as it shows the difference not having a custom Pipeline Resolver that modifies the data itself.

2. With no Custom Pipeline Resolvers Active

2.1 With no sleep(5000):

2.1.1 When no existing record exists:

Result: Same result as 1.1.1.

2.1.2 When an existing record exists:

Result: Same result as 1.1.2.

2.2 With sleep(5000) in between:

2.2.1 When no existing record exists:

Result: The initial create succeeds, but the subsequent update fails with 10 attempts all failing with: DataStore - conflict trycatch [TypeError: WeakSet key must be an Object]

2.2.2 When an existing record exists:

Result: Same result as 1.1.2.

@danrivett
Copy link

danrivett commented Jan 11, 2024

Problem Summary

From my problem recreation scenarios and results above, I observed a few things.

  1. If mutations on the same model record occurred back to back without any gap between, it appears possible for lost updates to occur.

    • See results for test scenarios 1.1.1 and 1.1.2 for examples.
  2. If mutations occur of the same type on the same model record (e.g. two update mutations) with a time gap between them (5 seconds in my case), then the DataStore - conflict trycatch [TypeError: WeakSet key must be an Object] error occurs

    • This is irrespective of whether there are custom pipeline resolvers modifying the data
    • See result for test scenario 1.2.2 as an example
  3. If mutations occur of a different type on the same model record (e.g. a create mutation, followed by an update mutation) with a time gap between them, but there is no custom Pipeline Resolver modifying the data, then the DataStore - conflict trycatch [TypeError: WeakSet key must be an Object] error occurs.

    • See result for test scenario 2.2.1 as an example
  4. Finally, if mutations occur of a different type on the same model record (e.g. a create mutation, followed by an update mutation) with a time gap between them, and there is a custom Pipeline Resolver modifying the data, then the ReferenceError: Property 'err' doesn't exist error occurs.

    • See result for test scenario 1.2.1 as an example
    • I wonder if a different error occurs in a non React Native environment as that error seems odd based on the location of it I referenced in a prevous comment. It looks like a different error occurs which then turns into ReferenceError: Property 'err' doesn't exist when run under Hermes.
  5. Incidentally, we actually noticed that for observation 4 above, it didn't actually need to be the same model record, or even the same model type, we had two different types with an update on one, followed by a delete on another, and we experienced ReferenceError: Property 'err' doesn't exist. And only the model updated (not deleted) had a Custom Pipeline Resolver.

That complicated the problem recreation steps, so for my recreation scenarios above I just used one model type, but it may be worth noting as it seems a Custom Pipeline resolver modifying data on the first mutation is ultimately what causes the 2nd mutation to fail with ReferenceError: Property 'err' doesn't exist. But I haven't confirmed that.

Next Steps

I've tried to provide as much information as I can in order for you to recreate these problems simply and reliably. They seem high priority to me given the errors. Certainly we cannot migrate to v6 until they are fixed.

If you can't recreate them, both @dmitryusikriseapps and I are happy to assist.

You may need to test in a React Native environment to recreate some of these issues.

We use Expo, and we had to build a custom Development Client through Expo to test with because v6 of the Amplify libraries aren't natively linked in Expo Go (see this Expo discussion: expo/expo/discussions/25586).

I'm hopeful experienced AWS DataStore developers such as @chrisbonifacio and @iartemiev can easily recreate this issue and then look for the root cause. That is where our experience really runs dry, but we're happy to assist as we can.

@cwomack cwomack added the DataStore Related to DataStore category label Jan 11, 2024
@chrisbonifacio
Copy link
Member

Hi @danrivett! Thank you for taking the time to deep dive into the issues you're experiencing and sharing your findings. I don't have an update just yet, but wanted to let you know we will be looking into it and what you've shared so far helps a ton!

@cwomack cwomack added investigating This issue is being investigated and removed pending-triage Issue is pending triage labels Jan 17, 2024
@nadetastic nadetastic added the React Native React Native related issue label Mar 4, 2024
@danrivett
Copy link

@chrisbonifacio it seems like your team has been very busy with other initiatives (gen 2 etc) so I've avoided asking for updates up until now to give some space and time whilst your team is working on other things, but this is still blocking my team from upgrading to v6, and so is becoming higher in priority for us.

Is there anything we can do to assist in troubleshooting or investigating this, as we'd really love to see this investigated and any necessary fixes made.

@chrisbonifacio chrisbonifacio removed their assignment May 20, 2024
@danrivett
Copy link

I'd definitely appreciate an update from AWS of some sort.

The radio silence is really disconcerting, as it's a blocker for us to migrate to v6.

We're willing to assist in recreation and root cause analysis if needed, but we can't solve the problem entirely on our own, and hope AWS Support will engage to assist in resolving this.

@danrivett
Copy link

I'm not sure who's leading Amplify DataStore support, but I'm very happy to work directly with them either through this ticket or preferably through an AWS Chime meeting to discuss and work out a plan of attack.

@chrisbonifacio chrisbonifacio added the question General question label Aug 29, 2024
@danrivett
Copy link

danrivett commented Sep 25, 2024

@chrisbonifacio given all the focus on Gen 2, and the fact DataStore seems unlikely to be supported there, I can somewhat understand why this bug report doesn't appear to have been worked on since I raised it in January.

We've been holding off upgrading to Gen 1 v6 libraries because this is a blocker as breaks our application, and like I mentioned in September, I'd be more than happy to work with anyone on AWS' side to move this ticket forward.

We currently use DataStore extensively both in a React Native application as well as a Next.js web application, and any migration off of DataStore as part of a wider Gen 2 migration is likely to be quite a way down the road given all the moving pieces required to do that, and the fact we're not sure how different the proposed DataStore replacement (React-Query + Offline Persistence plugin?) will be to integrate.

So if there is anyone still looking after DataStore that is willing to investigate this bug (including when pipeline resolvers mutate the data too - see writeup above), we'd be thrilled and very happy to assist if helpful.

/cc @undefobj just for his awareness as I know he worked extensively on the initial design and implementation of DataStore so may have some thoughts.

@github-actions github-actions bot added the pending-maintainer-response Issue is pending a response from the Amplify team. label Sep 25, 2024
@chrisbonifacio
Copy link
Member

chrisbonifacio commented Sep 26, 2024

Hi @danrivett & @dmitryusikriseapps,

apologies for the delay on this issue. I ran into some issues early on reproducing the issue and it fell off our radar but we are picking it up back to try and reproduce again with the new information provided.

We will report back once we've reproduced and identified the root cause.

So far, after some trial and error I've got an Expo app ready to start adding the code snippets provided.

From the scenarios listed in your previous comments it seems there are two errors and which one occurs depends on whether or not custom pipeline resolvers are active or not.

I think the only info missing might be the schema of the Worker or ContractorJobWorkerInfo models these errors are occurring on. Please share that if you can or if you have a public repository that reproduces the issue then that would also help expedite reproduction.

In the meantime, thank you for your patience while we try to get this fixed.

@github-actions github-actions bot removed the pending-maintainer-response Issue is pending a response from the Amplify team. label Sep 26, 2024
@chrisbonifacio chrisbonifacio added the pending-community-response Issue is pending a response from the author or community. label Sep 26, 2024
@chrisbonifacio
Copy link
Member

Update: based on the provided information, I've inferred this to be the schema to try and move forward with the reproduction of the issue and make use of the pipeline resolvers. Please correct if wrong!

type ContractorJobInfo @model @auth(rules: [{ allow: private }]) {
  id: ID!
  contractorId: String!
  cognitoUserId: String!
  jobId: String!
  createdBy: String
  lastEdited: LastEdited
}

type LastEdited {
  cognitoUserId: String!
  at: AWSDateTime!
  versionEdited: Int!
}

@chrisbonifacio
Copy link
Member

chrisbonifacio commented Sep 26, 2024

Update: I've got the app "working" so far on v6. When I attempt to create and/or update a record, I am not seeing an error yet but I am observing that the record is not being updated with values from the server such as _version, lastChangedAt, or _deleted.

Here is the app I'm using to attempt to reproduce the issue: https://github.com/chrisbonifacio/amplify-issue-12824

I'm not seeing errors in the console but perhaps they're appearing in the network logs or cloudwatch logs?
Can you confirm where you were able to see the error?

CleanShot 2024-09-26 at 13 48 53@2x

@chrisbonifacio chrisbonifacio self-assigned this Sep 27, 2024
@danrivett
Copy link

Hi @chrisbonifacio thanks so much for your reply and taking time today to work on this. Ironically I'm travelling with work currently so I've only just had time to read through your updates.

Before I posted the update request earlier this week, I asked one of the developers on our team to verify the issue was still present in our app, which they confirmed.

So I'm going to first reverify this myself with any helpful details I find, and then download your repo and try and recreate it there.

I'll post back as soon as I have those results.

Thanks again for taking time on this, I'm sure we'll be able to make some rapid progress between the two of us.

@github-actions github-actions bot added pending-maintainer-response Issue is pending a response from the Amplify team. and removed pending-community-response Issue is pending a response from the author or community. labels Sep 27, 2024
@danrivett
Copy link

And just to confirm, we saw DataStore errors in the React Native log, but I can't remember what I saw in the AppSync CloudWatch logs, so I'll have to update you on that once I recreate it again now.

@danrivett
Copy link

A quick update for the end of the week:

I spent a few hours recreating the issue in various builds of our app over the last 6 months, and I can recreate it using the scenario above in all builds, both Expo 48 and Expo 50 which we're currently on, and both on the aws-amplify: 6.1.0 which we were on when we raised the ticket and aws-amplify: 6.6.2.

We're not on Expo 51 yet as we're waiting for Expo 52 since it should be out pretty soon.

Interestingly although we see the error (technically it's a warning) on the latest build (Expo 50 and aws-amplify: 6.6.2) we don't see the warning logged to the console in Expo 50, just Expo 48, so that may partially explain why you're not seeing anything logged. (Turning on DEBUG level logging also didn't show the warning message - expected, but I still wanted to check).

For reference this is what I see in the simulator:

Simulator Screenshot - iPhone 15 Pro Max - 2024-09-27 at 19 04 59

I need to dig further next week once I'm back from my trip, as I'll more easily be able to try and recreate using your repo too. It was a bit tricky for me to do that today as I also need to deploy a new Amplify BE for it and that's easier for me to do next week.

So more to come, but I just wanted to at least provide an update with what I found today.

@danrivett
Copy link

Just a brief update to say @dmitryusikriseapps and I have been making good progress understanding the errors we're seeing, and during digging and temporarily patching Amplify libraries locally to log more things out and get past things, we've uncovered a couple of bugs on our end that doesn't currently get surfaced by DataStore so we didn't know about.

We also think there are possibly a couple of bugs in the Amplify library too. I'll write up once we've double checked and can summarise things best for review by you and your colleagues Chris.

So expect that summary later this week, and then you can review and see if they are able to be fixed.

@cwomack cwomack removed the investigating This issue is being investigated label Oct 7, 2024
@chrisbonifacio chrisbonifacio removed the pending-maintainer-response Issue is pending a response from the Amplify team. label Oct 8, 2024
@chrisbonifacio
Copy link
Member

Hi @danrivett Were you able to gather the details of the issues?

And was the repo I provided helpful?

@danrivett
Copy link

@chrisbonifacio Sorry for the delay. I've been delaying writing up the results as trying to make it as clear as possible.

The main issue in this ticket ReferenceError: Property 'err' doesn't exist we couldn't recreate in your repo, and digging deeper we resolved it in our app by deleting our yarn.lock file and having it get regenerated. That uncovered the trigger for the error in the first place - which was an unauthorized error being returned by AppSync that was being masked by DataStore.

That unauthorized error was due to a missing delete permission that we hadn't noticed since the item was being deleted locally in DataStore just fine, and then later updated via another process, so it wasn't out of sync with the server for very long.

The other main issue we had was DataStore - conflict trycatch [TypeError: WeakSet key must be an Object] and we tracked that down to a stale input object passed into a DataStore.save(). I don't want to get too specific here yet as I still need to validate a couple of things of what specifically triggers that.

We also found one other bug where DataStore wasn't handling an error response returned by an AppSync Mutation because error.data was undefined. I'll provide a further update with additional details on that as we think the fix for that could be very simple, so will add details.

We're currently running through further regression testing after fixing the issues it uncovered on our end, and then I'll add final details for you to review.

But bottom line is if others run into ReferenceError: Property 'err' doesn't exist it's likely an imcompatible package being used due to a lock file. The exact out of date library/libraries wasn't easy to find, so we haven't yet identified it, but regenerating the lock file got us past that, so we're regression testing to see if that causes other issues.

@github-actions github-actions bot added the pending-maintainer-response Issue is pending a response from the Amplify team. label Oct 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
DataStore Related to DataStore category pending-maintainer-response Issue is pending a response from the Amplify team. question General question React Native React Native related issue
Projects
None yet
Development

No branches or pull requests

5 participants