dbt-spark v1.9.0b1

dbt-spark 1.9.0-b1 - October 01, 2024

Features

• Add tests for cross-database cast macro (#1028)
• Cross-database date macro (#1031)
• Allow configuring snapshot column names (#1096)
• Support custom ODBC connection parameters via connection_string_suffix config (#1092)
• Add Microbatch Strategy to dbt-spark (#1109)

Fixes

• Fix incremental python models error where Databricks could not find the temp table transaction logs (#1033)

Under the Hood

• Lazy load agate to improve performance (#1049)

Dependencies

• Update freezegun requirement from ~=1.3 to ~=1.4 (#966)
• Bump actions/download-artifact from 3 to 4 (#1010)
• Bump actions/upload-artifact from 3 to 4 (#1011)
• Bump dbt-labs/actions from 1.1.0 to 1.1.1 (#1012)
• Update wheel requirement from ~=0.42 to ~=0.43 (#1035)
• Update pytest-xdist requirement from ~=3.5 to ~=3.6 (#1073)
• Loosen pin on pre-commit from 3.7.0 to 3.7 (#1074)

Contributors

• @jpoley (#1092)
• @nilan3 (#1092)

dbt-spark v1.8.0

dbt-spark 1.8.0 - May 09, 2024

Features

• Implement spark__safe_cast and add functional tests for unit testing (#987)
• : Add new workflow for internal patch releases (#38)

Fixes

• Support new agate Integer type and empty seed test (#935)
• Fix hardcoded file format for python models (#803)

Under the Hood

• Add GitHub action for integration testing and use dagger-io to run tests. Remove CircleCI workflow. (#719)
• Add tests for --empty flag (#949)
• Remove unused invalid_insert_overwrite_delta_msg message (#962)
• Update import paths and list_relations to support decoupling adapters/core (#972)
• Update dependabot config to cover GHA and Docker (#1006)

Dependencies

• Update pre-commit-hooks requirement from ~=4.4 to ~=4.5 (#903)
• Bump mypy from 1.5.1 to 1.6.0 (#904)
• Update pyodbc requirement from ~=4.0.39 to ~=5.0.0 (#905)
• Update pre-commit requirement from ~=3.4 to ~=3.5 (#914)
• Update pyodbc requirement from ~=5.0.0 to ~=5.0.1 (#925)
• Bump mypy from 1.6.0 to 1.6.1 (#926)
• Update black requirement from ~=23.9 to ~=23.10 (#927)
• Update black requirement from ~=23.10 to ~=23.11 (#942)
• Bump mypy from 1.6.1 to 1.7.0 (#946)
• Update pytest-xdist requirement from ~=3.3 to ~=3.4 (#947)
• Update pytest-xdist requirement from ~=3.4 to ~=3.5 (#951)
• Update wheel requirement from ~=0.41 to ~=0.42 (#952)
• Bump mypy from 1.7.0 to 1.7.1 (#953)
• Update freezegun requirement from ~=1.2 to ~=1.3 (#956)
• Update black requirement from ~=23.11 to ~=23.12 (#959)
• Add dbt-core as a dependency to preserve backwards compatibility for installation (#1002)
• Bump actions/setup-python from 4 to 5 (#1013)
• Bump actions/checkout from 3 to 4 (#1014)

Security

• Pin black>=24.3 in dev-requirements.txt (#1000)

Contributors

• @JCZuurmond, (#719)
• @ben-schreiber (#803)

dbt-spark v1.8.0rc1

dbt-spark 1.8.0-rc1 - May 06, 2024

Under the Hood

• Update dependabot config to cover GHA and Docker (#1006)

Dependencies

• Bump actions/setup-python from 4 to 5 (#1013)
• Bump actions/checkout from 3 to 4 (#1014)

dbt-spark v1.8.0b1

dbt-spark 1.8.0-b1 - March 01, 2024

Features

• Implement spark__safe_cast and add functional tests for unit testing (#987)

Fixes

• Support new agate Integer type and empty seed test (#935)
• Fix hardcoded file format for python models (#803)

Under the Hood

• Add GitHub action for integration testing and use dagger-io to run tests. Remove CircleCI workflow. (#719)
• Add tests for --empty flag (#949)
• Remove unused invalid_insert_overwrite_delta_msg message (#962)
• Update import paths and list_relations to support decoupling adapters/core (#972)

Dependencies

• Update pre-commit-hooks requirement from ~=4.4 to ~=4.5 (#903)
• Bump mypy from 1.5.1 to 1.6.0 (#904)
• Update pyodbc requirement from ~=4.0.39 to ~=5.0.0 (#905)
• Update pre-commit requirement from ~=3.4 to ~=3.5 (#914)
• Update pyodbc requirement from ~=5.0.0 to ~=5.0.1 (#925)
• Bump mypy from 1.6.0 to 1.6.1 (#926)
• Update black requirement from ~=23.9 to ~=23.10 (#927)
• Update black requirement from ~=23.10 to ~=23.11 (#942)
• Bump mypy from 1.6.1 to 1.7.0 (#946)
• Update pytest-xdist requirement from ~=3.3 to ~=3.4 (#947)
• Update pytest-xdist requirement from ~=3.4 to ~=3.5 (#951)
• Update wheel requirement from ~=0.41 to ~=0.42 (#952)
• Bump mypy from 1.7.0 to 1.7.1 (#953)
• Update freezegun requirement from ~=1.2 to ~=1.3 (#956)
• Update black requirement from ~=23.11 to ~=23.12 (#959)

Contributors

• @JCZuurmond, (#719)
• @ben-schreiber (#803)

dbt-spark v1.6.2

dbt-spark 1.6.2 - December 13, 2023

Dependencies

• Replace sasl with pure-sasl for PyHive (#818)

dbt-spark v1.5.3

dbt-spark 1.5.3 - December 13, 2023

Features

• Persist Column level comments when creating views (#372)

Dependencies

• Replace sasl with pure-sasl for PyHive (#818)

Contributors

• @jurasan (#372)

dbt-spark v1.6.1

dbt-spark 1.6.1 - November 09, 2023

Features

• Persist Column level comments when creating views (#372)

Fixes

• Support new agate Integer type and empty seed test (#935)

Contributors

• @jurasan (#372)

dbt-spark v1.7.1

dbt-spark 1.7.1 - November 07, 2023

Fixes

• Support new agate Integer type and empty seed test (#935)

dbt-spark v1.7.0

dbt-spark 1.7.0 - November 02, 2023

Features

• Support server side parameters in thrift connection (#387)
• Support server_side_parameters for Spark session connection method (#690)
• Add server_side_parameters to HTTP connection method (#824)
• Enforce server side parameters keys and values to be strings (#826)
• Add SessionConnectionWrapper (#829)
• Persist Column level comments when creating views (#372)
• Support storing test failures as views (#6914)
• Create temporary views with 'or replace' (#350)

Fixes

• Wrap AnalysisException into DbtRuntimeError (#782)
• include tblproperties macro in adapters.sql create table (#865)

Under the Hood

• Update stale workflow to use centralized version (#842)
• Remove dependency on hologram (#881)

Dependencies

• Update tox requirement from ~=3.0 to ~=4.5 (#741)
• Update pyodbc requirement from ~=4.0.30 to ~=4.0.39 (#742)
• Update pre-commit requirement from ~=2.21 to ~=3.3 (#748)
• Update types-requests requirement from ~=2.28 to ~=2.29 (#749)
• Bump mypy from 1.2.0 to 1.3.0 (#768)
• Replace sasl with pure-sasl for PyHive (#818)
• Update flake8 requirement from ~=6.0 to ~=6.1 (#849)
• Update types-requests requirement from ~=2.29 to ~=2.31 (#850)
• Update pytest-xdist requirement from ~=3.2 to ~=3.3 (#851)
• Update pytest requirement from ~=7.3 to ~=7.4 (#852)
• Update pip-tools requirement from ~=6.13 to ~=7.2 (#856)
• Update black requirement from ~=23.3 to ~=23.7 (#857)
• Update wheel requirement from ~=0.40 to ~=0.41 (#858)
• Update mock requirement from ~=5.0 to ~=5.1 (#859)
• Bump mypy from 1.3.0 to 1.4.1 (#860)
• Update tox requirement from ~=4.5 to ~=4.6 (#861)
• Update pip-tools requirement from ~=7.2 to ~=7.3 (#863)
• Update tox requirement from ~=4.6 to ~=4.7 (#867)
• Update tox requirement from ~=4.7 to ~=4.8 (#871)
• Bump mypy from 1.4.1 to 1.5.0 (#872)
• Update tox requirement from ~=4.8 to ~=4.9 (#874)
• Bump mypy from 1.5.0 to 1.5.1 (#875)
• Update tox requirement from ~=4.9 to ~=4.10 (#879)
• Update pre-commit requirement from ~=3.3 to ~=3.4 (#884)
• Update black requirement from ~=23.7 to ~=23.9 (#886)
• Update tox requirement from ~=4.10 to ~=4.11 (#887)

Security

• Add docker image to the repo (#876)

Contributors

• @Fokko (#829, #782, #876)
• @Fokko,JCZuurmond (#824, #826)
• @alarocca-apixio (#690)
• @annazizian (#350)
• @etheleon (#865)
• @hanna-liashchuk (#387)
• @jurasan (#372)

dbt-spark v1.7.0rc1

dbt-spark 1.7.0-rc1 - October 12, 2023

Features

• Support storing test failures as views (#6914)
• Create temporary views with 'or replace' (#350)

Contributors

• @annazizian (#350)