Geo search

[rpostulart]

At this moment there is no possibility to search via graphql / dynamodb certain records based on the longitude and latitude (geo) while there is a kind of library within AWS for this:
https://github.com/amazon-archives/dynamodb-geo

Can we make these functionalities also available via Amplify:
Box Queries: Return all of the items that fall within a pair of geo points that define a rectangle as projected onto a sphere.
Radius Queries: Return all of the items that are within a given radius of a geo point.
Basic CRUD Operations: Create, retrieve, update, and delete geospatial data items.
Easy Integration: Adds functionality to the AWS SDK for Java in your server application.
Customizable: Access to raw request and result objects from the AWS SDK for Java.

[temideoye]

@rpostulart, if you're using AppSync client, you can do geo-related queries with Elasticsearch as your data source. Might take a couple more steps than your regular DynamoDb setup but it's not too complicated. There's a tutorial here: https://goo.gl/Nbyrri for setting up your resolvers, and Elasticsearch's geo-query documentation is here: https://goo.gl/VHU4aH

[ghost]

Elastic search is a workaround. We'd like native geo features, like Firebase is about to offer. Amplify is meant for mobile apps, and "get all things in this geo box" or "get all things in a radius around this geo point" is a very popular usage for mobiles apps. Please consider making this a high priority

[undefobj]

DynamoDB does not support geo queries, which is why that repo is in archive now. AppSync supports both DynamoDB as well as Elasticsearch which you can leverage in a single API and stream the data from DynamoDB to Elasticsearch indexes. That's why @searchable was added: https://aws-amplify.github.io/docs/cli/graphql#searchable

I'm not sure what the feature request is here since Amplify leverages the backend AWS services. Could you see if @searchable in the transformer meets your use case?

[temideoye]

@undefobj, the @searchable directive provides an interface for String, Int, Float, and Boolean searches. If I understand it correctly, I think the feature request here is for geo-related searches which is probably two requests in one when broken down:

1. Support geo-types, i.e a new GraphQLObjectType, like AWSGeo?
2 Enable geo-related queries via @searchable, which is a bit more involved than your typical eq, ne, lt, gt e.t.c

[undefobj]

For new GraphQL scalar types that's best logged as a feature request to the AppSync service. I would recommend logging that here: https://forums.aws.amazon.com/forum.jspa?forumID=280&start=0

If there are enhancements needed to the GraphQL transformer it's best logged as a feature request in the Amplify CLI repo (as this repo is for the library): https://github.com/aws-amplify/amplify-cli

@rpostulart does this match with your request?

[ghost]

Why do you need ElasticSearch anyway for simple queries such as 'lt' 'ne' etc? DynamoDB supports these by itself. Is this just a away to force users to use and pay for ES even when not necessary?

Anyway, you can now do a Float lat and Float lon and do box queries by havin lat and lat lt and lon gt and lon lt, but come on... Also it fails at poles or if you need large boxes or.... Anyway you shouldn't have to be a spherical geometry expert to do mobiles apps, that's what libs are for (although basic knowledge is still a must IMHO)

[ghost]

@undefobj
True dynamodb doesn't support geo features natively, so that archived lib simulated geo features by using geohashes and in-memory filtering, similar to what firebase geo libs do. This would be an acceptable solution fornus IMO, or else intergrate @searchable with ES geo queries and types as guys previously said.

[rpostulart]

@undefobj I will check asap and let it know here.

[ghost]

There is this tutorial for geo queries in AppSync. Also, this AppSync tutorial mentions geospatial data/. So it is probably somehow possible, just very convoluted. App-serving frameworks like amplify should be much simpler.

[undefobj]

@doom777 you can use @model to generate the simple queries that you state above, as outlined here: https://aws-amplify.github.io/docs/cli/graphql#generates

However these will just use DynamoDB itself and at scale these filtering techniques may be less performant based on your data access pattern which is why Elasticsearch exists. The @searchable will automatically stream data from DynamoDB to Elasticsearch like in the tutorial that you linked above.

[rpostulart]

I checked and agree with @doom777 . ElasticSearch does not suits my use case. I believe that ElasticSearch is too heavily for this use case. Because I expect it will be a highly used mobile app. I don't want to spin up an instance for each request, I prefer a serverless solutions for example with Lambda.

[ghost]

I may be wrong, but I don't think an ES instance is spun up on every request; I think it stays there permanently, possibly in a cluster.

[rpostulart]

Sorry that's what I mean indeed!

[undefobj]

Right, so to be clear @rpostulart as in the link I posted above @model will include some filtering capability out of the box with DynamoDB. For instance any string types will have operations like less than, between, etc:

input ModelStringFilterInput {
  ne: String
  eq: String
  le: String
  lt: String
  ge: String
  gt: String
  contains: String
  notContains: String
  between: [String]
  beginsWith: String
}

This is probably good for prototyping & small apps, which is why we added this capability based on customer feedback. However, if your app takes off and your table gets very large this becomes less scalable and you'll want to leverage a purpose-built search engine like Elasticsearch which is why @searchable exists. The Elasticsearch instance is not "serverless" though as it will always be up whereas DynamoDB is a per-request model. If you do believe your app will be highly used then you should go this route, however if not then you could just start with the base DynamoDB capabilities and later you could always add in Elasticsearch. It's not a one-way door. AWS does not offer a "serverless full text search" service at this time so outside of this not much else we can offer here.

[rpostulart]

Ok thanks, then I will focus on that.

[ghost]

At the very least AWS could offer great documentation on how to implement this common usw casr

[undefobj]

@doom777 what more documentation could we add here? I linked to the transformer documentation above, is more needed there? Or alternatively is something else specifically needed?

[ghost]

Look, I need to do a "select all entity in this geo box" query. Please show me an AWS tutorial how to do it. This is a common enough case.

[ghost]

An AppSync dev has confirmed that this feature is in the planning pipeline
https://forums.aws.amazon.com/thread.jspa?threadID=296334&tstart=0

[undefobj]

Thanks for the feedback @doom777. As noted at this time you need Elasticsearch with @searchable directive, however we'll look into ways to see if something could be done in the GraphQL transformer in the future for DynamoDB only. I've tagged this as a feature request and added it into the CLI repo.

[ghost]

@undefobj the issue is less in making it dynamodb focused and dropping ES (although running an ES cluster is not very serverless)

In general there is no support for geo in the Amplify/AppSync stack. No geo types, no query functions.

[t0t07]

To add, read https://medium.com/open-graphql/building-a-graphql-api-by-example-restaurant-reviews-part-4-geospatial-search-e0e0d9dc0329 for how to geo search with ES.

[ghost]

@YikSanChan that link is about hacking AppSync to work with ES geo features. Specifically, an object with two Floats is transformed into an ES geo_point. True we can do it, but we are requesting native geo support all the way down to Amplify. Meaning having AppSync create AWSGeoPoint custom type, and graphql transformer/code generator create meaningful search functions when a graphql type has @searchable AND has an AWSGeoPoint variable.

[t0t07]

@doom777 I like your proposal. Maybe a better way to propose this is to submit an RFC?

[n0ne]

Maybe one day DocumentDB will support geo indexes like normal MongoDB?...

[rpostulart]

I have picked up this task again with Amplify and Elasticsearch this time. GEO searching is then still not an out of the box solution, these are just two seperate services and we need to put something in between to do geo searching. It would be still great if AWS can provide something with Amplify to do GEO search on Elasticsearch or Dynamodb. Are there some developments? @doom777 also asked for an update in the AWS forum.

[rpostulart]

Is there maybe an update / progress about supporting GEO search / queries in dynamodb. I have it now up and running in a elasticsearch service (via amplify) which works great, but I also notice that it is very expensive while I am just have a few call per months.

[rpostulart]

@kaustavghosh06 @undefobj @mikeparisstuff Is there maybe an update about geo support in aws amplify on top of dynamodb or more a kind of serverless elasticsearch implementation where you only pay if you query elastic. This, to decrease the heavy costs of the use of @searchable

[rpostulart]

I looked at the new announcement of ultra warm and hoped this fulfill this need, but unfortunately not. Can we expect anything on this soon?

[shawnmmatthews]

What about something like this? https://read.acloud.guru/location-based-search-results-with-dynamodb-and-geohash-267727e5d54f

[rpostulart]

I know, but I don't want to use a package that hardly is used or maintained. It has open issues older than 2 years :(

[rpostulart]

Any updates / progress?

[rpostulart]

Will this be on the roadmap soon or do we not need to hope for it and accept the current solution?

[anaibol]

++1

[oe-bayram]

+1

[kdMoura]

+1

[anaibol]

+1

[yutaoyan]

+1

[ctjlewis]

+1

[anaibol]

+1

[GeorgeBellTMH]

So correct me if I'm wrong, but there are three necessary pieces for this...

1. To be able to store location in DynamoDB in formats that match elastic search's formats (maybe just automatically recognize them as part of @searchable, maybe with an off switch for backwards compatibility)
2. Stream the data to store it in elastic search in the proper format
3. Create a ModelGeoFilterInput and a SearchableGeoFilterInput so you can perform actions

Maybe starting with the simplest use-case would be a good PR:

• Ability to add single location (lat/lng)
• Ability to index/sort by distance from a given point
• Ability to filter by a bounding box

Some more info:
https://www.elastic.co/blog/geo-location-and-search
https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html

[AbdulBasitKhaleeq]

any simple solution for this issue?

[xiaohai0520]

+1

[cjihrig]

Can this be closed in favor of Amplify's geo category - https://docs.amplify.aws/cli/geo/search/ ?

[biller-aivy]

Can this be closed in favor of Amplify's geo category - https://docs.amplify.aws/cli/geo/search/ ?

Geo Search vs geospatial search?

With amplify geo it is Not possible to listPlaces from DynamoDB which Are in the near of me right?

[cjihrig]

With amplify geo it is Not possible to listPlaces from DynamoDB which Are in the near of me right?

Not from DynamoDB, but from Amazon Location Service.

[biller-aivy]

With amplify geo it is Not possible to listPlaces from DynamoDB which Are in the near of me right?

Not from DynamoDB, but from Amazon Location Service.

I guess its not that what here is needed.
it should be a cheaper solution as the @search directive.
So ALS is a forreign api to find places and directions to external data.

[majirosstefan]

any updates here ?

[fomson]

A solution: https://stefan-majiros.com/blog/geo-search-with-aws-amplify-cdk-and-appsync-custom-js-resolvers

(related: aws-amplify/amplify-cli#13183)

Thanks to @mikeparisstuff 👏🏼