Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New places api support #47085

Open
wants to merge 16 commits into
base: main
Choose a base branch
from

Conversation

rojiphil
Copy link
Contributor

@rojiphil rojiphil commented Aug 8, 2024

@rafecolton @mkhutornyi

Details

This PR implements the scope limited to the places migration as used in NewDot.

Dependencies:

  1. Changes in react-native-google-places-autocomplete library which is implemented in this PR
  2. API Key updation for Places API (New) in Google Cloud
  3. BE changes to support API changes for Autocomplete (New) and Place Details(New)

The implementation details for the relevant steps as mentioned here are as follows:

Let us introduce a isNewPlacesAPI property in GooglePlacesAutocomplete component here to introduce support for Places (New) api. By default, we can set this to false here so that the current implementation of Places api is used by default. This way, we do not break existing systems using the old Places API. Further, to make use of the Places API (New), the FE can send isNewPlacesAPI here.

  1. Use HTTP POST instead of HTTP GET for Autocomplete (New) API uses as mentioned here.

For Autocomplete (New) request, we can send a HTTP POST request here and also set the new url for POST i.e. ${url}/v1/places:autocomplete. Further, we can send the request body with the supported parameters here.
In addition, we also need to update the API for Place Details (New) with url ${url}/v1/places/${rowData.place_id} here

  1. Handle the format changes in response structure for Autocomplete (New) API as mentioned here

Format changes include changing the request parameters to languageCode, includedRegionCodes and locationBias here. For the response though, we need to filter the results and handle the format changes. This is done here and here

  1. Use the required parameter fieldmask for Place Details (New) API as mentioned here. This way we can control the billing by including what we need in field mask

Let us introduce a fields property in GooglePlacesAutocomplete component here to specify the desired response fields. By default, we can set this to * here so that all the fields are sent back. In our case, we set fields to make use of Location Only (Place Details) SKU thereby reducing our API cost. The specific value of fields is set here.

  1. Use session tokens to group autocomplete and place details which will help in reduced billing. This is referenced here.

To use sessions, we can create a sessionToken state here and use uuid library to generate v4 uuids for the session. This session token will be used in every Autocomplete (New) API here. And the session token will be reset to a new uuid here after using it for the last time in the Place Details (New) API here

  1. Handle the format changes related to Place object for in response structure of Place Details (New) API as referenced here

Since the response has changed Place Details (New) API, there will not be any status. Hence, we can depend on id here and pass the details to the onPress handler. The FE saveLocationDetails is modified to support the format changes in the Place Details (New) API response.

Fixed Issues

$ #46771
PROPOSAL:

Test

Steps:

  1. Click on the Global FAB —> Submit Expense
  2. Click on the Distance tap and click Start to setup a waypoint
  3. Enter the search text in Address input field
  4. Verify that the autocomplete results are displayed with the waypoint name and waypoint address
  5. Click on one of the results and verify that the selected waypoint name is added as a start waypoint
  6. Click Stop to setup a waypoint
  7. Select an already predefined waypoint and verify that the selected waypoint is added as a stop waypoint
  • Verify that no errors appear in the JS console

Offline tests

Same as the steps in Tests Section.

QA Steps

Same as the steps in Tests Section.

  • Verify that no errors appear in the JS console

PR Author Checklist

  • I linked the correct issue in the ### Fixed Issues section above
  • I wrote clear testing steps that cover the changes made in this PR
    • I added steps for local testing in the Tests section
    • I added steps for the expected offline behavior in the Offline steps section
    • I added steps for Staging and/or Production testing in the QA steps section
    • I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
    • I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
    • I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).
  • I included screenshots or videos for tests on all platforms
  • I ran the tests on all platforms & verified they passed on:
    • Android / native
    • Android / Chrome
    • iOS / native
    • iOS / Safari
    • MacOS / Chrome / Safari
    • MacOS / Desktop
  • I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
  • I followed proper code patterns (see Reviewing the code)
    • I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
    • I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
    • I verified that comments were added to code that is not self explanatory
    • I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
    • I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method
      • If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:
    • I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
    • I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
    • I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
    • I verified the JSDocs style guidelines (in STYLE.md) were followed
  • If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
  • I followed the guidelines as stated in the Review Guidelines
  • I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
  • I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
  • I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
  • I verified that if a function's arguments changed that all usages have also been updated correctly
  • If a new component is created I verified that:
    • A similar component doesn't exist in the codebase
    • All props are defined accurately and each prop has a /** comment above it */
    • The file is named correctly
    • The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
    • The only data being stored in the state is data necessary for rendering and nothing else
    • If we are not using the full Onyx data that we loaded, I've added the proper selector in order to ensure the component only re-renders when the data it is using changes
      • Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
    • All JSX used for rendering exists in the render method
    • The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions
  • If any new file was added I verified that:
    • The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory
  • If a new CSS style is added I verified that:
    • A similar style doesn't already exist
    • The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG))
  • If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
  • If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
  • If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
  • If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
  • If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
  • If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

MacOS: Safari

TODO

iOS: mWeb Safari

TODO

MacOS: Desktop

TODO

iOS: Native

TODO

Android: Native

TODO

Android: mWeb Chrome

TODO

Copy link

melvin-bot bot commented Aug 8, 2024

Hey! I see that you made changes to our Form component. Make sure to update the docs in FORMS.md accordingly. Cheers!

@rojiphil
Copy link
Contributor Author

rojiphil commented Aug 8, 2024

@rafecolton @mkhutornyi Please review the implementation details in OP. Also, please note the following:

  1. Currently, as BE is not yet ready, error code 666 would be received. So, until the BE is ready to process the modified URLs, I have temporarily added Autocomplete (New) response here and Place Details (New) API response here so that we can test. These changes would be removed once the BE changes are ready.
  2. Although in development, we can still test this out by copying GooglePlacesAutocomplete.d.ts and GooglePlacesAutocomplete.js from the PR branch here to the node_modules/react-native-google-places-autocomplete.

Copy link
Member

@rafecolton rafecolton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please merge main and update to use your branch of react-native-google-new-places-autocomplete? Will make it a little easier for me for testing

Edit: You can disregard this, I completed testing.

Copy link
Member

@rafecolton rafecolton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everywhere we use long_name or short_name in both code and comments in src/components/AddressSearch/index.tsx and src/libs/GooglePlacesUtils.ts, it needs to be changed to longName and shortName respectively. With that small change, this works great!

name: details.name,
lat: details.location?.latitude ?? details.geometry.location?.lat ?? 0,
lng: details.location?.longitude ?? details.geometry.location?.lng ?? 0,
name: autocompleteData?.structured_formatting?.main_text ?? details?.name ?? '',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

details.name isn't the same as it was in the old API, the format here is places/<id> so not really good to use here. The next closest thing is details?. shortFormattedAddress

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this context, the details.name is not used from old API. Instead, it indicates a predefined option i.e. saved places in our app. So, I think we need to keep
details.name

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand what you mean, can you show me? The way I understand this code, the value in autocompleteData.structured_formatting.main_text in the new API is most similar to details.name from the old api. However, details.name is no longer a good fallback, as in the new API, it does not contain a short version of the address, but something else entirely

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand what you mean, can you show me?

Please find the test video that demonstrates the use of predefined places. This is the reason why I think we need to keep details.name.

47085-predefined-places.mp4

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm ok. Can you take a look at how this is utilized when changing your address on the profile settings page as well? I wonder if we need different values in the two places.

@@ -174,7 +172,7 @@ function AddressSearch(

const values = {
street: `${streetNumber} ${streetName}`.trim(),
name: details.name ?? '',
name: autocompleteData?.structured_formatting?.main_text ?? details?.name ?? '',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment here as above

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above

@rojiphil
Copy link
Contributor Author

Sorry for the delay here. I will look into this today and revert.

@rojiphil
Copy link
Contributor Author

@rojiphil @mkhutornyi Are you able to test on Expensify Staging? If not, are you able to get your own API key so you can test e2e locally?

@rafecolton I am not able to test as I still get 666 response on accessing the new places API through the Staging server. Here is a video demonstrating this. What am I missing here?

47085-api-issue.mp4

@rafecolton
Copy link
Member

Ah, looks like you tested a couple hours before the code was on staging. Sorry for the confusion, I was asking in general. It's on staging now, so please try again and LMK.

@rojiphil
Copy link
Contributor Author

Ah, looks like you tested a couple hours before the code was on staging. Sorry for the confusion, I was asking in general. It's on staging now, so please try again and LMK.

Yeah @rafecolton. The BE changes are reflected in staging as I do not get 666 now. But I ran into the following problem.

While I can fetch the autocomplete place predictions, I cannot fetch the place details. Strangely, it returns 200 but has no response contents. I am running out of ideas on why it does not return the response. I however can see that the API returns a valid response in the Google Maps Platform. I have also attached a test video to demonstrate all this.

Can you please help confirm in BE if the place details proxy request from FE returns a valid response from the actual Place Details API request made with the google platform?

47085-placedetails-issue.mp4

@rafecolton
Copy link
Member

That makes sense. See my prior review.

@rojiphil
Copy link
Contributor Author

That makes sense. See my prior review.

@rafecolton Sorry if I am missing something in your review here but can you please point out which one?
As I understand, passing languageCode and regionCode as mentioned here is optional. Also, I assume that key would get added in BE as we would not want to expose the key in FE. Further, the comment here deals with the response but we are not there yet. The place details API request itself is sending back empty response which baffles me.

@rafecolton
Copy link
Member

rafecolton commented Sep 22, 2024

If you're seeing the API return data and the UI is not working, that means we're parsing it wrong in the front end. You can print it with console.log and compare it to the code to see what's wrong.

That's what I did when I posted #47085 (review), which explains what to do to fix the error you're seeing.

Edit: I misread your response, I see now you said you're not getting back any data at all from the API. Can you please start by making the change I requested in my review? I tested it locally and it works with that change, so I know that change is required.

It looks like the request format may have changed from when I tested based on your updates to the library function so once you make the changes I requested here, I'll test again against the BE proxy locally to see what's up.

Copy link
Contributor Author

@rojiphil rojiphil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it needs to be changed to longName and shortName respectively

Instead of longName and shortName the response now consists of longText and shortText. Looks like the response format has changed as can be seen here. I got this response when requesting via API explorer on Google Maps platform. With this data, the FE works well.

To test this, please use Statue of Liberty as search text and select Statue of Liberty National Monument to request the place details api. For testing purposes, I am constructing a response when BE sends back empty response as referenced here and it works well. Here is a test video for this.

47085-api-issue2.mp4

name: details.name,
lat: details.location?.latitude ?? details.geometry.location?.lat ?? 0,
lng: details.location?.longitude ?? details.geometry.location?.lng ?? 0,
name: autocompleteData?.structured_formatting?.main_text ?? details?.name ?? '',
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this context, the details.name is not used from old API. Instead, it indicates a predefined option i.e. saved places in our app. So, I think we need to keep
details.name

@@ -174,7 +172,7 @@ function AddressSearch(

const values = {
street: `${streetNumber} ${streetName}`.trim(),
name: details.name ?? '',
name: autocompleteData?.structured_formatting?.main_text ?? details?.name ?? '',
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above

@rojiphil
Copy link
Contributor Author

so once you make the changes I requested here, I'll test again against the BE proxy locally to see what's up.

@rafecolton I have addressed the review comments. Hopefully we will unravel the mystery of empty response when we test the BE proxy locally.

I have added the key here to address the review comment here. But looks like we need not worry about this as BE will add the key on receiving the proxy request from FE.

I have added languageCode and regionCode here to address the review comment here

@rojiphil
Copy link
Contributor Author

@rafecolton While we address the issue of placeDetails, a different issue related to AutoComplete API request is also noticed. Let me also mention that here so that when you look into this issue, both can be covered.

This issue related to AutoComplete will occur when we have a valid locationBias to send. This will happen only after we select any location as a waypoint. Once added as a waypoint in our app, further autocomplete requests will contain locationBias. However, this will fail due to Invalid JSON payload error as demonstrated below. It looks like stringifying the JSON is causing an issue here.

As our current focus is to address the PlaceDetails issue, I have suppressed the error due locationBias by excluding from the parameter here. To trigger this issue though, we can include the locationBias again.

47085-api-issue3.mp4

@rojiphil
Copy link
Contributor Author

Hopefully we will unravel the mystery of empty response when we test the BE proxy locally.

@rafecolton I just tested placedetails API now and, woohoo, it works great. So, we can ignore the comment here. I have also removed the test code in the library as it is no longer required. The following test video demonstrates the working of placedetails API via proxy URL.

47085-place-details-works-1.mp4

@rojiphil
Copy link
Contributor Author

@rafecolton So the only pending issue is the comment here concerning autocomplete(new) api request via proxy. Wondering if a change in content-type is needed for the autocomplete(new) api.

@rafecolton
Copy link
Member

Instead of longName and shortName the response now consists of longText and shortText. Looks like the response format has changed as can be seen here. I got this response when requesting via API explorer on Google Maps platform. With this data, the FE works well.

Yes you are correct, it's longText not longName and shortText not shortName. You still haven't made this change to your code, so it's still not working. Any particular reason you still have not made that change yet? Here is the specific change required to make this work:

diff --git a/src/components/AddressSearch/index.tsx b/src/components/AddressSearch/index.tsx
index 712c364b646..f18c6c63272 100644
--- a/src/components/AddressSearch/index.tsx
+++ b/src/components/AddressSearch/index.tsx
@@ -135,27 +135,27 @@ function AddressSearch(
             country: countryPrimary,
         } = GooglePlacesUtils.getAddressComponents(addressComponents, {
             // eslint-disable-next-line @typescript-eslint/naming-convention
-            street_number: 'long_name',
-            route: 'long_name',
-            subpremise: 'long_name',
-            locality: 'long_name',
-            sublocality: 'long_name',
+            street_number: 'longText',
+            route: 'longText',
+            subpremise: 'longText',
+            locality: 'longText',
+            sublocality: 'longText',
             // eslint-disable-next-line @typescript-eslint/naming-convention
-            postal_town: 'long_name',
+            postal_town: 'longText',
             // eslint-disable-next-line @typescript-eslint/naming-convention
-            postal_code: 'long_name',
+            postal_code: 'longText',
             // eslint-disable-next-line @typescript-eslint/naming-convention
-            administrative_area_level_1: 'short_name',
+            administrative_area_level_1: 'shortText',
             // eslint-disable-next-line @typescript-eslint/naming-convention
-            administrative_area_level_2: 'long_name',
-            country: 'short_name',
+            administrative_area_level_2: 'longText',
+            country: 'shortText',
         });
 
-        // The state's iso code (short_name) is needed for the StatePicker component but we also
-        // need the state's full name (long_name) when we render the state in a TextInput.
+        // The state's iso code (shortText) is needed for the StatePicker component but we also
+        // need the state's full name (longText) when we render the state in a TextInput.
         const {administrative_area_level_1: longStateName} = GooglePlacesUtils.getAddressComponents(addressComponents, {
             // eslint-disable-next-line @typescript-eslint/naming-convention
-            administrative_area_level_1: 'long_name',
+            administrative_area_level_1: 'longText',
         });
 
         // Make sure that the order of keys remains such that the country is always set above the state.
diff --git a/src/libs/GooglePlacesUtils.ts b/src/libs/GooglePlacesUtils.ts
index 312a0dc61c1..147d6933435 100644
--- a/src/libs/GooglePlacesUtils.ts
+++ b/src/libs/GooglePlacesUtils.ts
@@ -1,8 +1,8 @@
 type AddressComponent = {
     // eslint-disable-next-line @typescript-eslint/naming-convention
-    long_name: string;
+    longText: string;
     // eslint-disable-next-line @typescript-eslint/naming-convention
-    short_name: string;
+    shortText: string;
     types: string[];
 };
 type FieldsToExtract = Record<string, Exclude<keyof AddressComponent, 'types'>>;
@@ -11,8 +11,8 @@ type FieldsToExtract = Record<string, Exclude<keyof AddressComponent, 'types'>>;
  * Finds an address component by type, and returns the value associated to key. Each address component object
  * inside the addressComponents array has the following structure:
  * [{
- *   long_name: "New York",
- *   short_name: "New York",
+ *   longText: "New York",
+ *   shortText: "New York",
  *   types: [ "locality", "political" ]
  * }]
  */

@rafecolton
Copy link
Member

Looking at the waypoint issue now

@rafecolton
Copy link
Member

Was able to reproduce the location bias issue. You are right, it looks like the problem is Google isn't handling URI-encoded data that's multiple layers deep, even though it's encoded correctly. Experimenting a little to see if it's better to send as JSON from the front-end or decode in PHP. Probably the former so this works for people not using a proxy

Copy link
Member

@rafecolton rafecolton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR for back-end changes is up for review. I tested and confirmed it works with the diff from my review here and the diff posted in this comment. It fixes the locationBias issue you described. Please make both of those changes, and I will let you know when it's ready to test on staging.

Please also reply to my comment here 🙏

This is almost ready! Thank you for your persistence and diligence in checking edge cases.

@rojiphil
Copy link
Contributor Author

rojiphil commented Oct 2, 2024

It fixes the locationBias issue you described. Please make both of those changes, and I will let you know when it's ready to test on staging.

Yeah! Looks like this is pretty close to getting done. Thanks. I will pick this up tomorrow my day for making the necessary changes.

@rafecolton
Copy link
Member

The BE fix has been deployed to staging so you should be good to test.

Do you think we can get the PRs ready for review today?

@rojiphil
Copy link
Contributor Author

rojiphil commented Oct 3, 2024

You still haven't made this change to your code, so it's still not working. Any particular reason you still have not made that change yet? Here is the specific change required to make this work:

Yeah. That got missed out. The required changes are made and thanks for pointing this out.

@rojiphil
Copy link
Contributor Author

rojiphil commented Oct 3, 2024

Please also reply to my comment here 🙏

I have replied to the comment here

@rojiphil
Copy link
Contributor Author

rojiphil commented Oct 3, 2024

The BE fix has been deployed to staging so you should be good to test.

Do you think we can get the PRs ready for review today?

@rafecolton The BE fix works great with staging along with the fixes in FE.
I have bumped up the version in library and have placed the library in ready for review state. Once released, I will test this out again with the updated library and make this PR ready for review.

Meanwhile, here is a test video that demonstrates the working of new places API.

47085-web-safari-000.mp4

@rafecolton
Copy link
Member

Library was published, so placing this in Ready for review status

@rafecolton rafecolton marked this pull request as ready for review October 4, 2024 19:21
@rafecolton rafecolton requested a review from a team as a code owner October 4, 2024 19:21
@melvin-bot melvin-bot bot requested review from mkhutornyi and removed request for a team October 4, 2024 19:21
Copy link

melvin-bot bot commented Oct 4, 2024

@mkhutornyi Please copy/paste the Reviewer Checklist from here into a new comment on this PR and complete it. If you have the K2 extension, you can simply click: [this button]

Copy link
Member

@rafecolton rafecolton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested this out locally and it worked great! @mkhutornyi @rojiphil please complete your cross-platform testing and checklists, and assuming no unexpected errors, I will merge.

Great work!

@rojiphil
Copy link
Contributor Author

rojiphil commented Oct 4, 2024

please complete your cross-platform testing and checklists, and assuming no unexpected errors, I will merge.

@rafecolton On native iOS session token generation is failing due to unsupported crypto.getRandomValues. Please hold merge until I finish investigation on this.

Screenshot 2024-10-05 at 1 03 26 AM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants