Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Assume deployed materialized views are unchanged during view deploys #4679

Closed
wants to merge 1 commit into from

Conversation

scholtzan
Copy link
Collaborator

@scholtzan scholtzan commented Dec 11, 2023

Related to #4576

Checklist for reviewer:

  • Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title).
  • If the PR comes from a fork, trigger integration CI tests by running the Push to upstream workflow and provide the <username>:<branch> of the fork as parameter. The parameter will also show up
    in the logs of the manual-trigger-required-for-fork CI task together with more detailed instructions.
  • If adding a new field to a query, ensure that the schema and dependent downstream schemas have been updated.
  • When adding a new derived dataset, ensure that data is not available already (fully or partially) and recommend extending an existing dataset in favor of creating new ones. Data can be available in the bigquery-etl repository, looker-hub or in looker-spoke-default.

For modifications to schemas in restricted namespaces (see CODEOWNERS):

┆Issue is synchronized with this Jira Task

@scholtzan scholtzan changed the title Assume defployed materialized views are unchanged during view deploys Assume deployed materialized views are unchanged during view deploys Dec 11, 2023
Comment on lines +258 to +261
if table.view_query is None:
# Materialized views do not have view_query.
# Always assume they are unchanged since they can't be replaced anyway
return False
Copy link
Contributor

@fbertsch fbertsch Dec 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This kind of special casing makes the codebase hard to reason about. Is there a way for us to prevent bqetl view publish from indexing materialized views to begin with?

Copy link
Contributor

@fbertsch fbertsch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a test for this behavior?

@dataops-ci-bot
Copy link

Integration report for "Assume defployed materialized views are unchanged during view deploys"

sql.diff

Click to expand!
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/releases_latest/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/releases_latest/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/releases_latest/schema.yaml	2023-12-11 17:40:27.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/releases_latest/schema.yaml	2023-12-11 17:32:57.000000000 +0000
@@ -2,18 +2,21 @@
 - name: date
   type: DATE
   mode: NULLABLE
+  description: null
 - name: product
   type: STRING
   mode: NULLABLE
 - name: category
   type: STRING
   mode: NULLABLE
+  description: null
 - name: channel
   type: STRING
   mode: NULLABLE
 - name: build_number
   type: INTEGER
   mode: NULLABLE
+  description: null
 - name: release_date
   type: DATE
   mode: NULLABLE

Link to full diff

@fbertsch
Copy link
Contributor

Instead, we're going to rollback #4576

@fbertsch fbertsch closed this Dec 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants