Amplify Api not working properly when apply filter on it.

[uzairleo]

Description

Hi there, we are using amplify Api backed by dynamodb for our use-case and there is a scenario where we implement search on the user table and fetch records on the basis of user input email, which is based on Amplify Api Query where we actually apply some filter on that Query using the where field of ModelQueries.lis() method

final request = ModelQueries.list(model, where: where, limit: limit);
    final response = await Amplify.API.query(request: request).response;

It was working awesome when the records were small But now it's behaving weirdly as it is not fetching and even not showing an empty response on some of the records that are available in dynamodb and one can pull through dynamodb but the Query fails to fetch and even show at least empty response.

After looking into some of the amplify-js threads on the same topic on GitHub issues I found that using custom Query is something emphasized for our use-case such as search or advanced search implementation. so I replaced ModelQueries.list() based implementation into Custom Graphql Query and that resolved the issue for us, this custom query was able to return every user which matched with user-provided email/literals.

query SearchUsers(\$email: String!, \$id: ID!,\$isCompleted: Boolean, \$nextToken: String) {
         listTotemUSERS(
           filter: {
              email: { contains: \$email }, 
              id: { ne: \$id },
              isCompleted: { eq: \$isCompleted }
              },
            limit: 9999, nextToken: \$nextToken ) {
           items {
             id
             email
             name
             phoneNumber
             isCompleted
           }
            nextToken
         }
       } 

But this fix brings another issue as the fix works awesome in the dev/staging env of Amplify where we have a small number of records, not bulks. But when we tried to migrate the implementation to prod env of amplify for testing purposes where we have a bulk of records then we found that the Custom Graphql Query is also not fully working as its not fetching the record when given the full email But it does fetch record when providing first one to 3ree literals of user email Also keeping in mind this behavior is only limited now for all those user records that were recently not working not even returning an empty response with ModelQueries.list() .
Another thing that causes confusion is the same records that were not pulled by the above methods were also not pulled through the amplify content manager test Graphql console by providing either the same query or a different query of List it can only be fetched through dynamodb and with LIST query when providing email partially. Why is it behaving like that?

Categories

• Analytics
• API (REST)
• API (GraphQL)
• Auth
• Authenticator
• DataStore
• Notifications (Push)
• Storage

Steps to Reproduce

No response

Screenshots

No response

Platforms

• iOS
• Android
• Web
• macOS
• Windows
• Linux

Flutter Version

3.10.6

Amplify Flutter Version

1.3.1

Deployment Method

Amplify CLI
12.4.0

Schema

type User @model @auth(rules: [{allow: public}]) {
  id: ID!
  name: String!
  email: String
  declined: Boolean
}

[uzairleo]

@Jordan-Nelson @jacoblogan

[uzairleo]

@dnys1

[uzairleo]

@khatruong2009 any updates Guys ?

[Equartey]

Hey @uzairleo, sorry you're experiencing this issue.

To summarize the issue, you are querying a list of Users and want to filter the results by passing a full email string. However, you're not receiving any matches for the full string, but you are receiving results when you pass a partial email string. This is working on a small data set, but not a large one. Is all this correct?

If so, you're implementation is almost correct. To have efficient queries on large data sets via AppSync and DynamoDB we recommend you setup global secondary indexes. Can you review this comment. I'll also call out you'll need to adjust the limit parameter so you are not capping your scans at 9999.

Please don't hesitate to reach out if you have more questions.

[uzairleo]

yeap you are right @Equartey Thanks for responding. Can you please share the official setup of up GSI doc for Flutter , Also I do have one Question regarding the implementation of this GSI.

1-If we go with setting up GSI I believe we have to modify Graphql schema if that the case then I have a concern about this fix that is The fix will only work for new users but for an old user it won't work because that's already registered with old schema without GSI implementation in dynamoDb please correct me if i am wrong @Equartey and provide some context on this .

[Equartey]

Hi @uzairleo

Setting up GSI in Flutter has two main steps, CLI/Schema setup and the Flutter implementation.

1. For the required schema updates please reference this doc.

2. For the Flutter implementation, we don't have an explicit guide for filtering use case, but this sorting guide is pretty close. Let me know if you have further questions.

As for updates to the schema, DynamoDB handles backfill of the new index (reference). However, your schema has email marked as an optional property. Querying on GSI won't work on User objects that did not provide an email. This is called a sparse index and is a valuable feature.

In short, existing users should be able to use the new GSI once setup correctly. It's highly encouraged you make back ups and test in your different staging environments to confirm.

[uzairleo]

Got it Also this raised another question Can the GSI setup be done for one of the tables in a schema or is not it required for all the available tables in the schema since we only need for USER record and query-related to the user?
I have checked the shared resources most of them are for JS but I have found the @index implementation Can you please clear it @Equartey the use of directive @index is required only on unique id/field or not it should be with all the fields of our USER in our use-case?

[uzairleo]

@Equartey

[Equartey]

@uzairleo If I understand the question correctly, you want to know if you must add the @index rule to every property on the User Model?

@index should only be added to the property(s) you wish to filter or sort by. There is a limit of 20 GSIs you can have on a single table.

[uzairleo]

Got it , please double confirm @Equartey it wont effect/impact the already available user query or mutation processes that's already been executing in old build installed with users and at the same time we change the schema with GSI implementation
For Example User A have installed the app with old schema without GSI setup for dynamodb filtering purpose and we push changes with GSI then does the old schema still work with dynamodb for all mutations and Query as a fallback or not ?

[Equartey]

@uzairleo This is not a breaking change. GSI enables additional query abilities.

The schema does not live on the user's device.

Once you update and push changes to the schema, the GSI will be present in the DynamoDB table.

DynamoDB handles backfill of the previously created items in the table. So previous entries both persist and function with the new index.

Mutations and queries written before the update will remain unaffected.

[uzairleo]

Got it Thanks @Equartey i would suggest to please update your flutter documentation and add these facts regarding usage of queries for frequent purposes specifically the current document is for JS purpose but yeap its closer so we can implement the workaround but it will help users to directly get the information from official docs instead of back and fourth from the different available threads

[uzairleo]

@Equartey can you please elaborate the usage of GSI for filtering purpose because in recommended docs which you shared the @index annotation is used just only for the purpose of sorting and it enable to use sortbykey to user. But since our use-case is filtering and i am not able to find any close relation of sorting with filtering our usecase inside docs which lead me to setup GSI in our case by only adding @index to the email in Graphql Schema can you please check the above customQuery which i shared above in issue and let us know what will be the second step for us since you mentioned setting up GSI have two main step
1-Graphql schema modification
2-Implementation from flutter side but since i am using custom query what will be these changes in our case as compared to sorting guide provided ?

[uzairleo]

@Equartey

[uzairleo]

UPDATE: @Equartey i setup GSI ( Global seconday index ) but still the issue is there and the same issue is still happening i do test on different environments and everything related to retrieving user email is still the same can you please mention if there is any other fix?
IF not then we will move to static search from this dynamic search and we will use but i am not sure that feature is supported yet or not that is ----> Getting single record from dynamo db by using email just like we are doing for fetching single object by id

[Equartey]

@uzairleo here are two examples of how to perform a query given a specific property (email).

Given this schema:

type USER @model @auth(rules: [{ allow: public }]) {
  id: ID!
  name: String!
  email: String! @index #GSI
}

We can then write a query using the provide model helper functions like so:

const userEmail = ""; // define email

// Query for email via Model helper
final request = ModelQueries.list(
USER.classType,
  where: USER.EMAIL.contains(userEmail),
);

final response = await Amplify.API.query(request: request).response;
final users = response.data?.items ?? [];
if (response.errors.isNotEmpty) {
  safePrint('errors: ${response.errors}');
}

safePrint('Query result: $users');

Alternatively, since we defined a GSI in our schema, a custom query is generated in AppSync. In this example it would be called uSERSByEmail.

Note: The name of this generated function will be determined by GSI definition in the schema.graphql. You can verify the generated query name in the AppSync Console. Navigate to AppSync console in your AWS account > select your api > Queries > inspect the explorer for custom queries.

const userEmail = ""; // define email

// Custom graphql query
final customRequest = GraphQLRequest<PaginatedResult<USER>>(
  document: r'''
    query QueryByEmail($email: String!) {
      uSERSByEmail(email: $email) {
        items {
          email
          id
          name
        }
      }
    }
    ''',
  variables: {'email': userEmail},
  decodePath: 'uSERSByEmail',
  modelType: const PaginatedModelType(USER.classType),
);

final response = await Amplify.API.query(request: customRequest).response;
final users = response.data?.items ?? [];
if (response.errors.isNotEmpty) {
  safePrint('errors: ${response.errors}');
}

safePrint('Query result: $users');

Query Predicate Documentation

GSI Custom Query Documenation

[uzairleo]

@Equartey are you sure because usersByEmail is not defined anywhere i will test that as well
But yeap the macro userById is defined in amplify setup and i do tested thats working but let me test the query by email directly from amplify Graphqltest platform

[Equartey]

@uzairleo the name might vary, but when the field is marked as a GSI - @index() - a generated query should be visible in the AppSync console. You will still need to setup a graphql request by manually writing the document string, but the above example should allow you to keep your types.

Let me know if you have any issues.

[uzairleo]

I tried this query from amplify dashboard but its not fetching email as the resolver is expecting UserByEmail to be defined somewhere but this query is not defined any where i run this below Query

`uSERSByEmail(email: $email) 
{
 items
 {
email 
id
 name
 } }`

Also wanna to mention i do create index on email and set GSI but still the issue is happening and the root cause is userByEmail query is not defined anywhere from backend side while uswrById (id:id)
Is defined and working

And what i get from this is either we have to override a Resolver or create a lamda resolver for this simple purpose if i am not wrong ? Can you please confirm

[Equartey]

When you made updates to your schema.graphql, did you also run $ amplify codegen models & $ amplify push?

Can you please share your current schema.graphql?

[uzairleo]

nope i dont need to do codegen model because i am setting up GSI from dynamoDb dashboard directly from table detail
Plus i am running from amplify dashboard content manager section where you guys provide support for testing Queries

[uzairleo]

I run the above custom Query and this is the response we are continously getting which clearly show that this custom query is not defined anywhere but the one that is running currently such getUserById(id) is defined in your resolver which is abstracted , here is the response

{
  "data": null,
  "errors": [
    {
      "path": null,
      "locations": [
        {
          "line": 62,
          "column": 3,
          "sourceName": null
        }
      ],
      "message": "Validation error of type FieldUndefined: Field 'UserBYEMAIL' in type 'Query' is undefined @ 'UserBYEMAIL'"
    }
  ]
}

Now for this simple use-case such as retrieving single record from DB by non-partitionkey which is fixed through GSI But still not working one have to write a custom LAMDA GRAPHQL resolver for it , i believe @Equartey you have to put enhancement label to this issue to add this functionality as its really basic Query where one will get Item by email non-primarykey by doing GSI setup .

[uzairleo]

If you still think there is quick fix available for this please mention here or not then LAMDA RESOLVER will be the only option for us to go with . because i am not able to found any fix for this and the response of the query make it clearer too that each Query we are executing is somewhere defined like listuser , getuserbyid so it cant be any string and should must be defined before someone type this.

[uzairleo]

@Equartey ??

[Equartey]

@uzairleo Since you setup the GSI manually through DynamoDB, you will need to create the custom query manually. This is why you are unable to find the "UserBYEMAIL" query.

However, when using the Amplify CLI we handle this for you. The custom GSI query gets generated and deployed for you using the the previously mentioned commands. I verified this when I provided the two examples.

Putting aside the custom query, you can use the first example using query predicates to retrieve the data you want. Here is that example:

const userEmail = ""; // define email

// Query for email via Model helper
final request = ModelQueries.list(
USER.classType,
  where: USER.EMAIL.contains(userEmail),
);

final response = await Amplify.API.query(request: request).response;
final users = response.data?.items ?? [];
if (response.errors.isNotEmpty) {
  safePrint('errors: ${response.errors}');
}

safePrint('Query result: $users');

[uzairleo]

its not working @Equartey i believe i do mention this earlier thats why we are using custom Query with this approach the above you mention the record were available in db but the above method not able to fetch for us . but the custom QUery did that but with some issues so we decided to get a single record by passing email only But look like you miss confused that i do created another issue to make it more clear but you close that and now you are giving method that are out of topic @Equartey it still return list of user and something already we did and results are not Good we need similar method now to avoid all these issues , which lead us to use some similar method just like we have for userbyid

  final request = ModelQueries.get(  USER.classType,
        USERModelIdentifier(id: userId),);
      final response = await Amplify.API.query(request: request).response;

This above working for fetching by id but same is not working for email which lead us to use custom Query because there is no support in fetching single record using dynamodb through email because we try this below its failed with same error i put above

final request = ModelQueries.get(  USER.classType,
       USERModelIdentifier(email: email),);
     final response = await Amplify.API.query(request: request).response;

[uzairleo]

Another thing which you are keep ignoring is why the Query not working in amplify dshboard plus one more thing you mentioned that dont set index from dynamodb i believe if either we set from dynamodb or from codebase both updated there this time i created from codebase and push the code through amplify push but still thats not working .

Also you are keep ignoring the LAMDA RESOLVER thing ? does there is something to HIDE ?

[Equartey]

final request = ModelQueries.get(  USER.classType,
      USERModelIdentifier(email: email),);
final response = await Amplify.API.query(request: request).response;

The ModelQueries.get() request as written does not work because .get() only works on the model's identifier property defined in the schema.graphql. In your case id: ID!.

In order to use email as the identifier, aka custom primary key, it has to be defined as the the primary key in the schema file. However, I do not recommend this because such a change would be breaking change to your data set. Which you indicated was not a desired outcome.

---

Also you are keep ignoring the LAMDA RESOLVER thing ? does there is something to HIDE ?

The lambda resolver is not mentioned because there are alternative solutions that are more simple to implement. It can be a solution, but again not recommended.

---

i believe i do mention this earlier thats why we are using custom Query with this approach the above you mention the record were available in db but the above method not able to fetch for us .

From the previous code sample you were passing a limit parameter. This actually sets the amount of items the DynamoDB scans before applying the filter.

Meaning, with out the limit parameter DynamoDB will only be capped by the physical size of the data set (1 MB). Once that limit is reached, you need to paginate the results.

Have you tried removing the limit parameter in the original code snippet when testing?

---

"message": "Validation error of type FieldUndefined: Field 'UserBYEMAIL' in type 'Query' is undefined @ 'UserBYEMAIL'"

This error indicates the @index property was not setup correctly on the model. I'm happy to help troubleshoot this with you.

It would be helpful to see the contents of the schema.graphql when you received this error. Can you please provide it here?

[uzairleo]

@Equartey from the very start we do try alot with ModelQueries.list() but on your this comment i try again multiple times with it but the results are stil same , here are the findings which i mentioned above as well but let me clear here as well. so you will have better understandings
1-Tried with ModelQueries.list() working with very small db records but not working with our production env where we have bulk of records
2- Same happens for Dev env where we have initially small amount of records but now have more so some of email still not working .
3- tried limit without limit , But still the results are same
4- setup GSI index for our email field i can show you too but still thats not working

So this lead us to these below options recently and now again i tried the same , we instead ModelQueries.list() we write a custom Query that do the same using listUser query but that have little different result it work when we provide half literals of email but when provide full email then its not working even i implemented the pagination as well , But the problem is the query response is empty and have no response or nextToken with some user emails when provided as input .

Also i believe the index is not wrong it's right you can see here in the SS as well so the issue is not about index it's saying the query i am trying is undefined .

The more important Question raised why we have to query as a list of item in response if the requirement is to fetch only single record so this does make sense to us and now we need to fetchItemByEmail the same way we do fetchUserByID. to do that we have two options
1-Lamda function (defined)
2- The other one you mention in the above reply can you please define that? How to fetch a single record by non-primary key? instead of a list as the only solution i get is to fetch it in lamda-resolver and then use the lamda-resolver as Graphql Query or either by using it as an endpoint.

As this affects our users a lot. So for now to be on the safer side we just simply have to choose one of them let me know if you still have any suggestions or if you want to troubleshoot more because we almost have troubleshooted and debugged this more than a week with different available possibilities but when there are too much records then Query behaving immature.

[uzairleo]

Also you mention UserByEMAIL can be any string but according to the available docs and data we collected founded that its not like that in the error its clearly mention that this can't be any string otherwise one will get undefined (that STRING) , and this is something we found by running getUSER(id ) with little modification in the name and thats return the same above error in amplify test-graphql console here it is

{
  getUSERBYuid(id: "USERID"){
    name
    id
    email
    pmtRefNo
  }
}

Response

{
  "data": null,
  "errors": [
    {
      "path": null,
      "locations": [
        {
          "line": 62,
          "column": 3,
          "sourceName": null
        }
      ],
      "message": "Validation error of type FieldUndefined: Field 'getUSERBYuid' in type 'Query' is undefined @ 'getUSERBYuid'"
    }
  ]
}

But the Query is working for id when the name is getUSER(id:"USERID") and the query is ```

{
  getUSER(id: "USERID "){
    name
    id
    email
    pmtRefNo
  }
}

So i believe this prove your above concern / point about @index as cause of this response wrong, since index is not required for ID ( primaryKey) but that also not working with different name of query other than getUser(id).

[Equartey]

@uzairleo thank you for the verbose context. This helps me understand your situation better.

I'm going to transfer this issue to a team that can better assist you.

[renebrandel]

Hi @uzairleo - it'd be great if you could share your schema.graphql file. Otherwise it'll be hard to assist.

Is your primary requirement that you want partial information search across all fields? If so, we recommend our OpenSearch integration. Please also review the OpenSearch pricing model.

Here's the documentation on how to setup OpenSearch integration with Amplify: https://docs.amplify.aws/cli/graphql/search-and-result-aggregations/

GSIs allow you to do "exact match" on the partition key, so if your use cases absolutely needs "partial matching", then GSIs in DynamoDB might not be the best fit.

Regarding how to setup GSIs: Just wanted to confirm that you added a @index to the recommended field, will create a new GraphQL query based on that index. Note: it can take a few minutes before the index is accessible. Everytime DynamoDB creates a new index, it takes a few minutes to backfill.

For example, this schema:

type Blog @model @auth(rules: [
  { allow: public, operations: [read] },
  { allow: owner }]) {
  title: String
  content: String @index
}

provides the following queries mutations:

(Separately, I wasn't sure if I fully understood how you've made the DynamoDB GSI changes - ignore below if it doesn't apply to your case)
Regarding the changes made directly within the DynamoDB console: Changed made "out-of-band" from the Amplify experience is not recommended and introduces "drift" between what Amplify expects your infrastructure to be vs. the real state. A bit of information on "drift": https://docs.amplify.aws/cli/project/troubleshooting/#push-failures-from-unmanaged-changes-to-cloud-resources---drift

[uzairleo]

@renebrandel if you focus and zoom out the details of the issue the schmea is already available in the issue details

[AnilMaktala]

Hey 👋 , This issue is being closed due to inactivity. If you are still experiencing the same problem and need further assistance, please feel free to leave a comment. This will enable us to reopen the issue and provide you with the necessary support.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.