Prepare active_users_aggregates for a backfill with shredder mitigation. #6349
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…on. Rename columns [first_seen_date, os_version, segment] to cascade upstream changes during the backfill for stable numbers.
…ith_shredder_mitigation
…n is backfilled. Remove the view generation from the new version which conflicts with previous version.
This comment has been minimized.
This comment has been minimized.
…ith_shredder_mitigation
This comment has been minimized.
This comment has been minimized.
…ith_shredder_mitigation
This comment has been minimized.
This comment has been minimized.
bochocki
reviewed
Oct 18, 2024
| scheduling: | ||
| dag_name: bqetl_analytics_aggregations | ||
| task_name: {{ app_name }}_active_users_aggregates | ||
| task_name: {{ app_name }}_active_users_aggregates_v3 |
There was a problem hiding this comment.
In the CODEOWNERS file, all of the tables are referenced as active_users_aggregates_v4. Should the task name match this? Should the directory also be updated from sql_generators/active_users_aggregates_v3/templates/metadata.yaml to sql_generators/active_users_aggregates_v4/templates/metadata.yaml?
There was a problem hiding this comment.
You;re right that the task must be named _v4, this happens in the folder for version 4.
In v3 folder the task remains as v3 and pending to be removed from the DAG once the v4 is in production.
Comment on lines
+97
to
+99
| focus_android_view_template = env.get_template("focus_android_view.sql") | ||
| mobile_view_template = env.get_template("mobile_view.sql") | ||
| view_template = env.get_template("view.sql") |
There was a problem hiding this comment.
I'm intentionally avoiding generating the view for v4 until the table is validated and in production. There should be a PR to generate it then.
| TABLE_NAME = os.path.basename(os.path.normpath(THIS_PATH)) | ||
| BASE_NAME = "_".join(TABLE_NAME.split("_")[:-1]) | ||
| DATASET_FOR_UNIONED_VIEWS = "telemetry" | ||
| CHECKS_TEMPLATE_CHANNELS = { |
There was a problem hiding this comment.
This doesn't contain Fenix; is that expected?
There was a problem hiding this comment.
Yes it is expected. Fenix checks are in a separate file.
Comment on lines
+118
to
+125
| elif browser.name == "focus_android": | ||
| query_sql = reformat( | ||
| focus_android_query_template.render( | ||
| project_id=target_project, | ||
| app_name=browser.name, | ||
| ) | ||
| ) | ||
| schema_template = mobile_schema_template |
There was a problem hiding this comment.
Why does focus_android have a query template that's distinct from the mobile query template?
There was a problem hiding this comment.
It was the only browser that required keeping legacy and Glean data, probably something to reevaluate soon.
| type: STRING | ||
| mode: NULLABLE | ||
| - name: attributed |
There was a problem hiding this comment.
attribution_medium, attribution_source, and attributed all seem to be removed from the schema. Is that intentional?
There was a problem hiding this comment.
Not intentional, added back, thanks!
| @@ -0,0 +1,93 @@ | |||
| fields: | |||
There was a problem hiding this comment.
The desktop v4 schema contains attribution columns while the mobile schema doesn't; is that intentional?
Integration report for "Ensure attribution columns are present in mobile's schema."
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR prepares active_users_aggregates for a backfill with shredder mitigation, which includes renaming columns to cascade upstream changes.
Related Tickets & Documents
Reviewer, please follow this checklist
┆Issue is synchronized with this Jira Task