[ETL-505] Add Garmin transforms to S3 to JSON job #65
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
thomasyu888
reviewed
Jul 25, 2023
| and isinstance(json_obj["CustomFields"][field_name], str) | ||
| ): | ||
| if len(json_obj["CustomFields"][field_name]) > 0: | ||
| # This JSON string was written in a couple different ways |
There was a problem hiding this comment.
Wow, this is nasty. Is this worth letting care evolution know about this?
There was a problem hiding this comment.
I haven't seen the [{\\\"id\\\": ... format in the production data (we would log a warning if we ran into a JSONDecodeError), so whatever issues they had with the pilot data seem to have been resolved.
There was a problem hiding this comment.
We just have to make sure we are monitoring the logs for these warnings.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In addition to the Garmin transforms, I refactored the S3 to JSON job to be more functional. What used to just be
write_file_to_json_datasethas been split up into three different functions:
transform_json- which takes care of any inserts/updates to the JSON datatransform_object_to_array_of_objects- which is a helper function fortransform_jsonget_output_filename- which takes care of filename / S3 object name formatting.As a result of refactoring in the job, the tests have been refactored as well. Some tests were deleted and replaced with a more specific test.