Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add datatype casting and temp deps #12

Merged
merged 10 commits into from
Jan 6, 2025
Merged

Conversation

fivetran-reneeli
Copy link
Contributor

@fivetran-reneeli fivetran-reneeli commented Nov 28, 2024

PR Overview

This PR will address the following Issue/Feature:
#10

This PR will result in the following new package version:

v0.3.0 - source release is breaking

Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:

PR #12 folds in upstream updates from the qualtrics_source package:

Under the Hood

  • Explicitly casts all boolean fields as {{ dbt.type_boolean() }}.
  • (Affects Redshift only) Creates new qualtrics_union_data macro to accommdate Redshift's treatment of empty tables.
    • For each staging model, if the source table is not found in any of your schemas, the package will create a empty table with 0 rows for non-Redshift warehouses and a table with 1 all-null row for Redshift destinations.
    • This is necessary as Redshift will ignore explicit data casts when a table is completely empty and materialize every column as a varchar. This throws errors in dowstream transformations in the zendesk package. The 1 row will ensure that Redshift will respect the package's datatype casts.

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

  • dbt run –full-refresh && dbt test
  • dbt run (if incremental models are present) && dbt test

Before marking this PR as "ready for review" the following have been applied:

  • The appropriate issue has been linked, tagged, and properly assigned
  • All necessary documentation and version upgrades have been applied
  • docs were regenerated (unless this PR does not include any code or yml updates)
  • BuildKite integration tests are passing
  • Detailed validation steps have been provided below

Detailed Validation

Please share any and all of your validation steps:

Validation tests added for each end model and they're all passing
image

If you had to summarize this PR in an emoji, which would it be?

💃

@fivetran-jamie fivetran-jamie self-assigned this Dec 18, 2024
Copy link
Collaborator

@fivetran-joemarkiewicz fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fivetran-jamie this PR looks good to go with just a few small requests for updates under the hood and also to adjust one of the validation tests to avoid false failures. This will be good to go once those updates are applied and the source is approved.

integration_tests/dbt_project.yml Outdated Show resolved Hide resolved
packages.yml Outdated
Comment on lines 2 to 10
# version: [">=0.2.0", "<0.3.0"]

- git: https://github.com/fivetran/dbt_qualtrics_source.git
revision: redshift_datatype_casts
warn-unpinned: false

# - local: ../Qualtrics/dbt_qualtrics_source
# - local: ../../dbt_qualtrics_source
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update before merging

Copy link
Contributor Author

@fivetran-reneeli fivetran-reneeli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just small suggestions!

@fivetran-jamie fivetran-jamie merged commit 00e3228 into main Jan 6, 2025
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants