-
Notifications
You must be signed in to change notification settings - Fork 179
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Microbatch strategy #1179
Merged
Merged
Microbatch strategy #1179
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-snowflake contributing guide. |
4 tasks
5 tasks
MichelleArk
commented
Sep 14, 2024
QMalcolm
force-pushed
the
microbatch-strategy
branch
from
September 17, 2024 16:17
ffde66f
to
ab720f6
Compare
mikealfare
approved these changes
Sep 18, 2024
QMalcolm
added a commit
to dbt-labs/dbt-redshift
that referenced
this pull request
Oct 2, 2024
This work is basically in entirety a duplicate of the work done by MichelleArk in dbt-labs/dbt-snowflake#1179. I don't really expect this to work first try, but it might. I expect to need to do some edits, but who knows, maybe I'll get lucky.
4 tasks
QMalcolm
added a commit
to dbt-labs/dbt-redshift
that referenced
this pull request
Nov 7, 2024
* Add microbatch strategy This work is basically in entirety a duplicate of the work done by MichelleArk in dbt-labs/dbt-snowflake#1179. I don't really expect this to work first try, but it might. I expect to need to do some edits, but who knows, maybe I'll get lucky. * Add changie doc * Add comment to microbatch macro to explain why we are re-implementing delete+insert * Add `begin` to microbatch config in test_incremental_microbatch.py * Cleanup predicates in microbatch materialization * Fix predicate/incremental predicate logic in microbatch macro * Remove unnecessary `if` in microbatch macro The `if` is unnecessary because predicates are guaranteed to exist, but the `if` was guarding against when there are no predicates. * Get batch start and end time in the same way * Remove unnecessary `target` specifications for columns of predicates in microbatch materialization The `target.` portion of `target.<column_name>` is unnecessary for the predicates in the microbatch materialization macro because the delete statement already ensures the "targeting` of `target` in the delete statement via the clause `delete from {{ target }}`. Said another way, there is no use of the word `using` in the delete clause, thus it is unambiguous what is being deleted from. --------- Co-authored-by: Michelle Ark <[email protected]> Co-authored-by: Mike Alfare <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves #1182
docs dbt-labs/docs.getdbt.com/# N/A
Problem
dbt-snowflake needs a microbatch implementation that:
compiled_code
will be filtered down by event_timeunique_key
configuration on the model.Solution
unique_key
, which is actually not strictly necessary!compiled_code
Testing:
insert_two_rows_sql
fixture to use snowflake-specific syntaxmicrobatch_model_sql
fixture to demonstrate the lack of requirement of aunique_key
configChecklist