Migrating to pyproject.toml and Hatch #853

benc-db · 2024-11-22T19:38:25Z

Description

Begin using Hatch and pyproject.toml to manage environments/builds/testing. This matches the direction that dbt Labs has been moving. I have added instructions on how to get set up for development. Reviewer can you please follow the instructions and see if you hit any snags?

I will add PR comments to explain files

Checklist

I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

benc-db · 2024-11-23T00:07:26Z

.github/workflows/integration.yml

      - name: Get http path from environment
        run: python .github/workflows/build_cluster_http_path.py
        shell: sh
-      - name: Install tox
+
+      - name: Install Hatch


Like Tox, installing Hatch is sufficient to install everything else

benc-db · 2024-11-23T00:08:40Z

.github/workflows/main.yml

-  build:
-    name: build packages
+      - name: Run Unit Tests
+        run: hatch run -v +py=${{ matrix.python-version }} test:unit


in the test environment we have a matrix for the supported python versions. This is how we request a particular entry in the matrix.

benc-db · 2024-11-23T00:09:27Z

.github/workflows/main.yml

-        run: |
-          dbt --version
+      - name: Verify distributions
+        run: hatch run verify:check-all


This does the same checks as before, but I'm no longer separating build and verify as separate tasks, just separate steps.

benc-db · 2024-11-23T00:10:01Z

dbt/adapters/databricks/__version__.py

@@ -1 +1 @@
-version: str = "1.9.0b1"
+version = "1.9.0b1"


Had to drop the type info here to satisfy Hatch

benc-db · 2024-11-23T00:10:45Z

dbt/adapters/databricks/impl.py

@@ -109,7 +109,7 @@
 @dataclass
 class DatabricksConfig(AdapterConfig):
    file_format: str = "delta"
-    table_format: TableFormat = TableFormat.DEFAULT
+    table_format: str = TableFormat.DEFAULT


on 3.12, mypy starts complaining that these enums actually inherit str, so their values are str.

benc-db · 2024-11-23T00:11:02Z

dbt/adapters/databricks/impl.py

@@ -764,7 +764,7 @@ class RelationAPIBase(ABC, Generic[DatabricksRelationConfig]):
    For the most part, these are just namespaces to group related methods together.
    """

-    relation_type: ClassVar[DatabricksRelationType]
+    relation_type: ClassVar[str]


Same str, enum issue as above.

benc-db · 2024-11-23T00:12:06Z

dbt/adapters/databricks/python_models/python_submissions.py

@@ -224,7 +223,9 @@ def compile(self, path: str) -> PythonJobDetails:
        if access_control_list:
            job_spec["access_control_list"] = access_control_list

-        return PythonJobDetails(self.run_name, job_spec, additional_job_config)
+        return PythonJobDetails(


For some reason, mypy was complaining that this init was giving too many args when it was a dataclass. I have no idea what causes that, but switching to a pydantic model fixes the complaint.

benc-db · 2024-11-23T00:13:22Z

pyproject.toml

+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+]
+dependencies = [


These dependencies are the ones required to build the package. The default env dependencies specified below are what we want installed when we're working in IDE/using the CLI.

are the dependencies exactly same as previous version? If not, shall we bump up the version

benc-db · 2024-11-23T00:13:54Z

tests/functional/adapter/materialized_view_tests/fixtures.py

@@ -13,9 +13,9 @@ def query_relation_type(project, relation: BaseRelation) -> Optional[str]:
        fetch="one",
    )[0]
    if table_type == "STREAMING_TABLE":
-        return DatabricksRelationType.StreamingTable.value
+        return DatabricksRelationType.StreamingTable


same str, enum issue as above

benc-db · 2024-11-23T00:14:29Z

tests/unit/api_client/api_test_base.py

@@ -1,8 +1,8 @@
 from typing import Any, Callable
+from unittest.mock import Mock


during this process I realized we were needlessly installing mock, as it has been a part of Python since 3.3.

docs/dbt-databricks-dev.md

Co-authored-by: Alex Guo <[email protected]>

Signed-off-by: huangxingyi <[email protected]> Co-authored-by: Ben Cassell <[email protected]>

jackyhu-db · 2024-11-26T05:26:19Z

CHANGELOG.md

@@ -34,6 +34,7 @@
 - Fix behavior flag use in init of DatabricksAdapter (thanks @VersusFacit!) ([836](https://github.com/databricks/dbt-databricks/pull/836))
 - Restrict pydantic to V1 per dbt Labs' request ([843](https://github.com/databricks/dbt-databricks/pull/843))
 - Switching to Ruff for formatting and linting ([847](https://github.com/databricks/dbt-databricks/pull/847))
+- Switching to Hatch and pyproject.toml for project config ([853](https://github.com/databricks/dbt-databricks/pull/853))


do you have to bump the minor version of this change/migration?

I would but this version hasn't been released yet.

Everything but the docs

3c36cae

benc-db requested review from andrefurlan-db and rcypher-databricks as code owners November 22, 2024 19:38

attempt to fix flows

b090883

benc-db had a problem deploying to azure-prod November 22, 2024 19:50 — with GitHub Actions Error

try to fix

318cd29

benc-db had a problem deploying to azure-prod November 22, 2024 20:00 — with GitHub Actions Error

stop using cov

ac91225

benc-db had a problem deploying to azure-prod November 22, 2024 20:11 — with GitHub Actions Failure

another batch of fix attempts

e5bc096

benc-db had a problem deploying to azure-prod November 22, 2024 22:53 — with GitHub Actions Error

benc-db had a problem deploying to azure-prod November 22, 2024 22:53 — with GitHub Actions Failure

benc-db had a problem deploying to azure-prod November 22, 2024 22:53 — with GitHub Actions Error

benc-db added 2 commits November 22, 2024 15:04

fix unit tests

bcdd0e0

fix build and verify

a3741b0

benc-db had a problem deploying to azure-prod November 22, 2024 23:24 — with GitHub Actions Error

benc-db had a problem deploying to azure-prod November 22, 2024 23:24 — with GitHub Actions Failure

docs

b0611f8

benc-db had a problem deploying to azure-prod November 23, 2024 00:03 — with GitHub Actions Error

benc-db requested review from jackyhu-db and alexguo-db November 23, 2024 00:05

benc-db commented Nov 23, 2024

View reviewed changes

benc-db added 2 commits November 25, 2024 09:54

changelog

e6a963b

try moving the flag

4b9059e

benc-db had a problem deploying to azure-prod November 25, 2024 17:57 — with GitHub Actions Failure

benc-db requested a review from eric-wang-1990 November 25, 2024 23:45

alexguo-db reviewed Nov 26, 2024

View reviewed changes

docs/dbt-databricks-dev.md Outdated Show resolved Hide resolved

Update docs/dbt-databricks-dev.md

3d25b43

Co-authored-by: Alex Guo <[email protected]>

alexguo-db previously approved these changes Nov 26, 2024

View reviewed changes

huangxingyi-git and others added 3 commits November 25, 2024 16:45

Set unique temp table suffix to allow parallel merge execution (#854)

bc10666

Signed-off-by: huangxingyi <[email protected]> Co-authored-by: Ben Cassell <[email protected]>

Try setting queue enabled for Python notebook jobs (#856)

356c7b2

update doc and gitignore

d01cc9e

benc-db dismissed alexguo-db’s stale review via d01cc9e November 26, 2024 00:47

benc-db temporarily deployed to azure-prod November 26, 2024 00:47 — with GitHub Actions Inactive

Merge branch 'main' into pyproject_toml

d63c53c

jackyhu-db reviewed Nov 26, 2024

View reviewed changes

jackyhu-db approved these changes Nov 26, 2024

View reviewed changes

benc-db merged commit b95a04a into main Nov 26, 2024
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrating to pyproject.toml and Hatch #853

Migrating to pyproject.toml and Hatch #853

benc-db commented Nov 22, 2024 •

edited

Loading

benc-db Nov 23, 2024

benc-db Nov 23, 2024

benc-db Nov 23, 2024

benc-db Nov 23, 2024

benc-db Nov 23, 2024

benc-db Nov 23, 2024

benc-db Nov 23, 2024

benc-db Nov 23, 2024

jackyhu-db Nov 26, 2024

benc-db Nov 23, 2024

benc-db Nov 23, 2024

jackyhu-db Nov 26, 2024

benc-db Nov 26, 2024 •

edited

Loading

		@@ -1,8 +1,8 @@
		from typing import Any, Callable
		from unittest.mock import Mock

Migrating to pyproject.toml and Hatch #853

Migrating to pyproject.toml and Hatch #853

Conversation

benc-db commented Nov 22, 2024 • edited Loading

Description

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benc-db Nov 26, 2024 • edited Loading

Choose a reason for hiding this comment

benc-db commented Nov 22, 2024 •

edited

Loading

benc-db Nov 26, 2024 •

edited

Loading