Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrating to pyproject.toml and Hatch #853

Merged
merged 16 commits into from
Nov 26, 2024
Merged

Migrating to pyproject.toml and Hatch #853

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
3c36cae
Everything but the docs
benc-db Nov 22, 2024
b090883
attempt to fix flows
benc-db Nov 22, 2024
318cd29
try to fix
benc-db Nov 22, 2024
ac91225
stop using cov
benc-db Nov 22, 2024
e5bc096
another batch of fix attempts
benc-db Nov 22, 2024
bcdd0e0
fix unit tests
benc-db Nov 22, 2024
a3741b0
fix build and verify
benc-db Nov 22, 2024
b0611f8
docs
benc-db Nov 23, 2024
90637c5
verbose logging to see what versions are installed
benc-db Nov 23, 2024
e6a963b
changelog
benc-db Nov 25, 2024
4b9059e
try moving the flag
benc-db Nov 25, 2024
3d25b43
Update docs/dbt-databricks-dev.md
benc-db Nov 26, 2024
bc10666
Set unique temp table suffix to allow parallel merge execution (#854)
huangxingyi-git Nov 25, 2024
356c7b2
Try setting queue enabled for Python notebook jobs (#856)
benc-db Nov 25, 2024
d01cc9e
update doc and gitignore
benc-db Nov 26, 2024
d63c53c
Merge branch 'main' into pyproject_toml
benc-db Nov 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 36 additions & 24 deletions .github/workflows/integration.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ concurrency:
cancel-in-progress: true

jobs:
run-tox-tests-uc-cluster:
run-uc-cluster-e2e-tests:
runs-on: ubuntu-latest
environment: azure-prod
env:
Expand All @@ -26,22 +26,26 @@ jobs:
TEST_PECO_UC_CLUSTER_ID: ${{ secrets.TEST_PECO_UC_CLUSTER_ID }}
steps:
- name: Check out repository
uses: actions/checkout@v3
uses: actions/checkout@v4

- name: Set up python
id: setup-python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: "3.10"
python-version: "3.9"

- name: Get http path from environment
run: python .github/workflows/build_cluster_http_path.py
shell: sh
- name: Install tox

- name: Install Hatch
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like Tox, installing Hatch is sufficient to install everything else

id: install-dependencies
run: pip install tox
- name: Run integration-uc-databricks-cluster
run: [email protected] DBT_DATABRICKS_LOCATION_ROOT=$DBT_DATABRICKS_LOCATION_ROOT DBT_DATABRICKS_HOST_NAME=$DBT_DATABRICKS_HOST_NAME DBT_DATABRICKS_UC_CLUSTER_HTTP_PATH=$DBT_DATABRICKS_UC_CLUSTER_HTTP_PATH DBT_DATABRICKS_CLIENT_ID=$DBT_DATABRICKS_CLIENT_ID DBT_DATABRICKS_CLIENT_SECRET=$DBT_DATABRICKS_CLIENT_SECRET tox -e integration-databricks-uc-cluster
uses: pypa/hatch@install

run-tox-tests-uc-sql:
- name: Run UC Cluster Functional Tests
run: [email protected] DBT_DATABRICKS_LOCATION_ROOT=$DBT_DATABRICKS_LOCATION_ROOT DBT_DATABRICKS_HOST_NAME=$DBT_DATABRICKS_HOST_NAME DBT_DATABRICKS_UC_CLUSTER_HTTP_PATH=$DBT_DATABRICKS_UC_CLUSTER_HTTP_PATH DBT_DATABRICKS_CLIENT_ID=$DBT_DATABRICKS_CLIENT_ID DBT_DATABRICKS_CLIENT_SECRET=$DBT_DATABRICKS_CLIENT_SECRET hatch -v run uc-cluster-e2e

run-sqlwarehouse-e2e-tests:
runs-on: ubuntu-latest
environment: azure-prod
env:
Expand All @@ -54,22 +58,26 @@ jobs:
TEST_PECO_UC_CLUSTER_ID: ${{ secrets.TEST_PECO_UC_CLUSTER_ID }}
steps:
- name: Check out repository
uses: actions/checkout@v3
uses: actions/checkout@v4

- name: Set up python
id: setup-python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: "3.10"
python-version: "3.9"

- name: Get http path from environment
run: python .github/workflows/build_cluster_http_path.py
shell: sh
- name: Install tox

- name: Install Hatch
id: install-dependencies
run: pip install tox
- name: Run integration-databricks-uc-sql-endpoint
run: [email protected] DBT_DATABRICKS_LOCATION_ROOT=$DBT_DATABRICKS_LOCATION_ROOT DBT_DATABRICKS_HOST_NAME=$DBT_DATABRICKS_HOST_NAME DBT_DATABRICKS_UC_CLUSTER_HTTP_PATH=$DBT_DATABRICKS_UC_CLUSTER_HTTP_PATH DBT_DATABRICKS_CLIENT_ID=$DBT_DATABRICKS_CLIENT_ID DBT_DATABRICKS_CLIENT_SECRET=$DBT_DATABRICKS_CLIENT_SECRET tox -e integration-databricks-uc-sql-endpoint
uses: pypa/hatch@install

- name: Run Sql Endpoint Functional Tests
run: [email protected] DBT_DATABRICKS_LOCATION_ROOT=$DBT_DATABRICKS_LOCATION_ROOT DBT_DATABRICKS_HOST_NAME=$DBT_DATABRICKS_HOST_NAME DBT_DATABRICKS_UC_CLUSTER_HTTP_PATH=$DBT_DATABRICKS_UC_CLUSTER_HTTP_PATH DBT_DATABRICKS_CLIENT_ID=$DBT_DATABRICKS_CLIENT_ID DBT_DATABRICKS_CLIENT_SECRET=$DBT_DATABRICKS_CLIENT_SECRET hatch -v run sqlw-e2e

run-tox-tests-non-uc:
run-cluster-e2e-tests:
runs-on: ubuntu-latest
environment: azure-prod
env:
Expand All @@ -79,17 +87,21 @@ jobs:
DBT_DATABRICKS_LOCATION_ROOT: ${{ secrets.TEST_PECO_EXTERNAL_LOCATION }}test
steps:
- name: Check out repository
uses: actions/checkout@v3
uses: actions/checkout@v4

- name: Set up python
id: setup-python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: "3.10"
python-version: "3.9"

- name: Get http path from environment
run: python .github/workflows/build_cluster_http_path.py
shell: sh
- name: Install tox

- name: Install Hatch
id: install-dependencies
run: pip install tox
- name: Run integration-databricks-cluster
run: [email protected] DBT_DATABRICKS_LOCATION_ROOT=$DBT_DATABRICKS_LOCATION_ROOT DBT_DATABRICKS_HOST_NAME=$DBT_DATABRICKS_HOST_NAME DBT_DATABRICKS_HTTP_PATH=$DBT_DATABRICKS_CLUSTER_HTTP_PATH DBT_DATABRICKS_CLIENT_ID=$DBT_DATABRICKS_CLIENT_ID DBT_DATABRICKS_CLIENT_SECRET=$DBT_DATABRICKS_CLIENT_SECRET tox -e integration-databricks-cluster
uses: pypa/hatch@install

- name: Run Cluster Functional Tests
run: [email protected] DBT_DATABRICKS_LOCATION_ROOT=$DBT_DATABRICKS_LOCATION_ROOT DBT_DATABRICKS_HOST_NAME=$DBT_DATABRICKS_HOST_NAME DBT_DATABRICKS_HTTP_PATH=$DBT_DATABRICKS_CLUSTER_HTTP_PATH DBT_DATABRICKS_CLIENT_ID=$DBT_DATABRICKS_CLIENT_ID DBT_DATABRICKS_CLIENT_SECRET=$DBT_DATABRICKS_CLIENT_SECRET hatch -v run cluster-e2e
150 changes: 23 additions & 127 deletions .github/workflows/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,40 +42,28 @@ defaults:

jobs:
code-quality:
name: ${{ matrix.toxenv }}
name: Code Quality

runs-on: ubuntu-latest
timeout-minutes: 10

strategy:
fail-fast: false
matrix:
toxenv: [linter]

env:
TOXENV: ${{ matrix.toxenv }}
PYTEST_ADDOPTS: "-v --color=yes"

steps:
- name: Check out the repository
uses: actions/checkout@v3
with:
persist-credentials: false
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v4.3.0
uses: actions/setup-python@v5
with:
python-version: "3.9"

- name: Install python dependencies
run: |
python -m pip install --user --upgrade pip
python -m pip --version
python -m pip install tox
tox --version
- name: Install Hatch
uses: pypa/hatch@install

- name: Run tox
run: tox
- name: Run Code Quality
run: hatch -v run code-quality

unit:
name: unit test / python ${{ matrix.python-version }}
Expand All @@ -88,131 +76,39 @@ jobs:
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12"]

env:
TOXENV: "unit"
PYTEST_ADDOPTS: "-v --color=yes --csv unit_results.csv"

steps:
- name: Check out the repository
uses: actions/checkout@v3
with:
persist-credentials: false
uses: actions/checkout@v4

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4.3.0
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

- name: Install python dependencies
run: |
python -m pip install --user --upgrade pip
python -m pip --version
python -m pip install tox
tox --version
- name: Run tox
run: tox

- name: Get current date
if: always()
id: date
run: echo "::set-output name=date::$(date +'%Y-%m-%dT%H_%M_%S')" #no colons allowed for artifacts

- uses: actions/upload-artifact@v4
if: always()
with:
name: unit_results_${{ matrix.python-version }}-${{ steps.date.outputs.date }}.csv
path: unit_results.csv
- name: Install Hatch
uses: pypa/hatch@install

build:
name: build packages
- name: Run Unit Tests
run: hatch run -v +py=${{ matrix.python-version }} test:unit
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in the test environment we have a matrix for the supported python versions. This is how we request a particular entry in the matrix.


build:
name: Build and Verify Packages
runs-on: ubuntu-latest

outputs:
is_alpha: ${{ steps.check-is-alpha.outputs.is_alpha }}

steps:
- name: Check out the repository
uses: actions/checkout@v3
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v4.3.0
uses: actions/setup-python@v5
with:
python-version: "3.9"

- name: Install python dependencies
run: |
python -m pip install --user --upgrade pip
python -m pip install --upgrade setuptools wheel twine check-wheel-contents
python -m pip --version
- name: Build distributions
run: ./scripts/build-dist.sh

- name: Show distributions
run: ls -lh dist/

- name: Check distribution descriptions
run: |
twine check dist/*
- name: Check wheel contents
run: |
check-wheel-contents dist/*.whl --ignore W007,W008

- name: Check if this is an alpha version
id: check-is-alpha
run: |
export is_alpha=0
if [[ "$(ls -lh dist/)" == *"a1"* ]]; then export is_alpha=1; fi
echo "::set-output name=is_alpha::$is_alpha"

- uses: actions/upload-artifact@v3
with:
name: dist
path: dist/

test-build:
name: verify packages / python ${{ matrix.python-version }} / ${{ matrix.os }}

if: needs.build.outputs.is_alpha == 0
- name: Install Hatch
uses: pypa/hatch@install

needs: build

runs-on: ${{ matrix.os }}

strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: ["3.9", "3.10", "3.11", "3.12"]

steps:
- name: Set up Python ${{ matrix.python-version }}
uses: actions/[email protected]
with:
python-version: ${{ matrix.python-version }}
- name: Build distributions
run: hatch -v build

- name: Install python dependencies
run: |
python -m pip install --user --upgrade pip
python -m pip install --upgrade wheel
python -m pip --version
- uses: actions/download-artifact@v3
with:
name: dist
path: dist/

- name: Show distributions
run: ls -lh dist/

- name: Install wheel distributions
run: |
find ./dist/*.whl -maxdepth 1 -type f | xargs python -m pip install --force-reinstall --find-links=dist/
- name: Check wheel distributions
run: |
dbt --version
- name: Install source distributions
run: |
find ./dist/*.gz -maxdepth 1 -type f | xargs python -m pip install --force-reinstall --find-links=dist/
- name: Check source distributions
run: |
dbt --version
- name: Verify distributions
run: hatch run verify:check-all
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does the same checks as before, but I'm no longer separating build and verify as separate tasks, just separate steps.

15 changes: 15 additions & 0 deletions .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
# Ruff version.
rev: v0.8.0
hooks:
# Run the linter.
- id: ruff
# Run the formatter.
- id: ruff-format

- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.13.0
hooks:
- id: mypy
additional_dependencies: [types-requests]
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
- Fix behavior flag use in init of DatabricksAdapter (thanks @VersusFacit!) ([836](https://github.com/databricks/dbt-databricks/pull/836))
- Restrict pydantic to V1 per dbt Labs' request ([843](https://github.com/databricks/dbt-databricks/pull/843))
- Switching to Ruff for formatting and linting ([847](https://github.com/databricks/dbt-databricks/pull/847))
- Switching to Hatch and pyproject.toml for project config ([853](https://github.com/databricks/dbt-databricks/pull/853))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you have to bump the minor version of this change/migration?

Copy link
Collaborator Author

@benc-db benc-db Nov 26, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would but this version hasn't been released yet.


## dbt-databricks 1.8.7 (October 10, 2024)

Expand Down
1 change: 0 additions & 1 deletion MANIFEST.in
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion dbt/adapters/databricks/__version__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
version: str = "1.9.0b1"
version = "1.9.0b1"
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had to drop the type info here to satisfy Hatch

4 changes: 2 additions & 2 deletions dbt/adapters/databricks/impl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@
@dataclass
class DatabricksConfig(AdapterConfig):
file_format: str = "delta"
table_format: TableFormat = TableFormat.DEFAULT
table_format: str = TableFormat.DEFAULT
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on 3.12, mypy starts complaining that these enums actually inherit str, so their values are str.

location_root: Optional[str] = None
include_full_name_in_path: bool = False
partition_by: Optional[Union[list[str], str]] = None
Expand Down Expand Up @@ -764,7 +764,7 @@ class RelationAPIBase(ABC, Generic[DatabricksRelationConfig]):
For the most part, these are just namespaces to group related methods together.
"""

relation_type: ClassVar[DatabricksRelationType]
relation_type: ClassVar[str]
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same str, enum issue as above.


@classmethod
@abstractmethod
Expand Down
8 changes: 4 additions & 4 deletions dbt/adapters/databricks/logging.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
import logging
import os
from logging import Handler, LogRecord, getLogger
from typing import Union

from dbt.adapters.events.logging import AdapterLogger

logger = AdapterLogger("Databricks")


class DbtCoreHandler(logging.Handler):
class DbtCoreHandler(Handler):
def __init__(self, level: Union[str, int], dbt_logger: AdapterLogger):
super().__init__(level=level)
self.logger = dbt_logger

def emit(self, record: logging.LogRecord) -> None:
def emit(self, record: LogRecord) -> None:
# record.levelname will be debug, info, warning, error, or critical
# these map 1-to-1 with methods of the AdapterLogger
log_func = getattr(self.logger, record.levelname.lower())
Expand All @@ -21,7 +21,7 @@ def emit(self, record: logging.LogRecord) -> None:

dbt_adapter_logger = AdapterLogger("databricks-sql-connector")

pysql_logger = logging.getLogger("databricks.sql")
pysql_logger = getLogger("databricks.sql")
pysql_logger_level = os.environ.get("DBT_DATABRICKS_CONNECTOR_LOG_LEVEL", "WARN").upper()
pysql_logger.setLevel(pysql_logger_level)

Expand Down
Loading