-
Notifications
You must be signed in to change notification settings - Fork 51
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge remote-tracking branch 'origin/main'
- Loading branch information
Showing
110 changed files
with
20,868 additions
and
25,408 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| ## 1.1.6 - 2025-01-04 | ||
|
|
||
| ### Added | ||
|
|
||
| * Seeds now require a `+dbt-osmosis: <path>` in `dbt_project.yml`, or dbt-osmosis raises an exception. | ||
| * Added a pluggy-based plugin system for custom fuzzy matching of column names (prefix, case transforms, etc.). | ||
| * Configuration can now be set at multiple levels (folder-level, node-level, column-level), merging with global defaults. | ||
| * Added `--synthesize` to `refactor`/`document` commands for auto-generating missing docs via OpenAI. | ||
| * Added `--skip-add-source-columns` to skip adding columns to source YAMLs specifically (helpful for wide or rapidly changing source schemas). | ||
| * A Python API for **Transform Operations** and **Transform Pipelines** that make them nice to use in custom automations built on top of dbt-osmosis. | ||
| * Enhanced API for chaining manifest and node transformers, enabling more modular and flexible operations. | ||
| * Added logging improvements and auto-apply parameters for better usability and traceability. | ||
| * Introduced support for environment variables in discovery functions, improving dynamic configuration options. | ||
| * Resolved sorting issues in YAML column ordering, allowing directory/node-level configurations for alphabetical sorting. | ||
|
|
||
| ### Changed | ||
|
|
||
| * Migrated `vars.dbt-osmosis` structure to nest sources under `sources:` key (e.g., `vars: dbt-osmosis: sources:`). | ||
| * Renamed flags: | ||
| * `--char-length` → `--string-length` | ||
| * `--numeric-precision` → `--numeric-precision-and-scale` | ||
| * `--catalog-file` → `--catalog-path` | ||
| * Organized and refactored prompts for file moves unless `--auto-apply` is passed. | ||
| * Child docs no longer overwrite existing downstream descriptions unless `--force-inherit-descriptions` is used. The old `osmosis_keep_description` is deprecated. | ||
| * Child meta merges with upstream meta instead of overwriting it, giving precedence to the node being merged into for conflicting keys. | ||
| * Refactored the entire package to follow a more hybrid functional programming (FP) and Go-style approach, emphasizing simplicity and functions operating on pure data structures. | ||
| * Improved LLM context handling with additional test coverage and enhanced input structure. | ||
| * Topological sorting now ensures consistent operation order during `_iter` traversal. | ||
|
|
||
| ### Removed | ||
|
|
||
| * The `diff` code and command are removed. | ||
| * The dbt bottle server was removed. | ||
| * The vendored dbt core interface was removed, folding into the core library, reducing unnecessary complexity and dependencies. | ||
| * Deprecated less-maintained SQLite adapter in favor of DuckDB, which provides significantly better performance and support. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| ## 1.1.7 - 2025-01-05 | ||
| ### Added | ||
| * better default str dumping for multiline strings in dbt yamls | ||
| * working dbt proxy speaking mysql protocol, part of an experiment but could be useful as is | ||
| ### Changed | ||
| * force dbt 1.9 compat via uv dep override, we need to keep streamlit pinned to where it is to to a breaking change which affects our material UI components |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| ## 1.1.8 - 2025-01-08 | ||
| ### Changed | ||
| * more string dumping improvements including canonicalization by default and dynamic determination of flow style based on line breaks or len vs config width |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| ## 1.1.9 - 2025-01-17 | ||
| ### Added | ||
| * best effort pass through of unrendered extra inheritables | ||
| ### Fixed | ||
| * Only inject node description if yaml description is empty | ||
| * Correct path matching for positional args |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,2 @@ | ||
| pip==24.0 | ||
| poetry==1.3.2 | ||
| virtualenv==20.21.0 | ||
| uv==0.5.13 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -25,10 +25,10 @@ jobs: | |
| pip install --constraint=.github/workflows/constraints.txt pip | ||
| pip --version | ||
| - name: Install Poetry | ||
| - name: Install uv | ||
| run: | | ||
| pip install --constraint=.github/workflows/constraints.txt poetry | ||
| poetry --version | ||
| pip install --constraint=.github/workflows/constraints.txt uv | ||
| uv --version | ||
| - name: Check if there is a parent commit | ||
| id: check-parent-commit | ||
|
|
@@ -42,33 +42,31 @@ jobs: | |
| uses: salsify/[email protected] | ||
| with: | ||
| version-command: | | ||
| bash -o pipefail -c "poetry version | awk '{ print \$2 }'" | ||
| sed -nE 's/^version = "([^"]+)"/\1/p' pyproject.toml | ||
| - name: Bump version for developmental release | ||
| if: "! steps.check-version.outputs.tag" | ||
| run: | | ||
| poetry version patch && | ||
| version="$(poetry version | awk '{ print $2 }')" && | ||
| poetry version "${version:?}.dev.$(date +%s)" | ||
| sed -i -E "s/^version\s*=\s*\"[^\"]+\"/version = \"1.0.0.dev.$(date +%Y%m%d)\"/" pyproject.toml | ||
| - name: Build package | ||
| run: | | ||
| poetry build --ansi | ||
| uvx --from=hatchling==1.26.3 hatchling build | ||
| - name: Publish package on PyPI | ||
| if: steps.check-version.outputs.tag | ||
| uses: pypa/gh-action-pypi-publish@v1.8.3 | ||
| uses: pypa/gh-action-pypi-publish@v1.12.3 | ||
| with: | ||
| user: __token__ | ||
| password: ${{ secrets.PYPI_TOKEN }} | ||
|
|
||
| - name: Publish package on TestPyPI | ||
| if: "! steps.check-version.outputs.tag" | ||
| uses: pypa/gh-action-pypi-publish@v1.8.3 | ||
| with: | ||
| user: __token__ | ||
| password: ${{ secrets.TEST_PYPI_TOKEN }} | ||
| repository_url: https://test.pypi.org/legacy/ | ||
| # - name: Publish package on TestPyPI | ||
| # if: "! steps.check-version.outputs.tag" | ||
| # uses: pypa/gh-action-pypi-publish@v1.12.3 | ||
| # with: | ||
| # user: __token__ | ||
| # password: ${{ secrets.TEST_PYPI_TOKEN }} | ||
| # repository_url: https://test.pypi.org/legacy/ | ||
|
|
||
| - name: Publish the release notes | ||
| uses: release-drafter/[email protected] | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -131,3 +131,6 @@ dmypy.json | |
|
|
||
| # Nix | ||
| .devenv | ||
|
|
||
| # Makefile touch target | ||
| .uv-installed-* | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,36 +1,31 @@ | ||
| repos: | ||
| - repo: https://github.com/pre-commit/pre-commit-hooks | ||
| rev: v4.6.0 | ||
| rev: v5.0.0 | ||
| hooks: | ||
| - id: end-of-file-fixer | ||
| - id: trailing-whitespace | ||
| - id: check-ast | ||
| - id: check-json | ||
| - id: check-yaml | ||
| - id: check-toml | ||
| - id: end-of-file-fixer | ||
| - id: trailing-whitespace | ||
| - id: detect-private-key | ||
| - id: debug-statements | ||
| - repo: https://github.com/rhysd/actionlint | ||
| rev: v1.7.1 | ||
| hooks: | ||
| - id: actionlint-docker | ||
| - repo: https://github.com/shellcheck-py/shellcheck-py | ||
| rev: v0.10.0.1 | ||
| - repo: https://github.com/commitizen-tools/commitizen | ||
| rev: v4.1.0 | ||
| hooks: | ||
| - id: shellcheck | ||
| - repo: https://github.com/psf/black | ||
| rev: 24.4.2 | ||
| - id: commitizen | ||
| - repo: https://github.com/astral-sh/ruff-pre-commit | ||
| rev: v0.8.0 | ||
| hooks: | ||
| - id: black | ||
| args: ["--config", "pyproject.toml"] | ||
| - repo: https://github.com/pycqa/isort | ||
| rev: 5.13.2 | ||
| - id: ruff-format | ||
| args: ["--preview"] | ||
| - id: ruff | ||
| args: [--fix] | ||
| - repo: https://github.com/gitleaks/gitleaks | ||
| rev: v8.22.1 | ||
| hooks: | ||
| - id: isort | ||
| args: ["-sp", "pyproject.toml"] | ||
| - repo: https://github.com/hadolint/hadolint | ||
| rev: v2.13.0-beta | ||
| - id: gitleaks | ||
| - repo: https://github.com/rhysd/actionlint | ||
| rev: v1.7.5 | ||
| hooks: | ||
| - id: hadolint-docker | ||
| # - repo: https://github.com/pycqa/flake8 | ||
| # rev: 4.0.1 | ||
| # hooks: | ||
| # - id: flake8 | ||
| - id: actionlint |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| # Author: Alex B | ||
| # Description: Makefile for setting up the development environment. Includes convenient aliases for common tasks. | ||
|
|
||
| PY_VERSION := $(shell cat .python-version) | ||
|
|
||
| .PHONY: uv venv pre-commit dev all test lint format scan scan-new-baseline \ | ||
| scan-without-baseline clean | ||
|
|
||
| check: format lint test | ||
|
|
||
| .uv-installed-$(PY_VERSION): .python-version | ||
| @if [ ! -f .python-version ]; then echo 'Please create a .python-version file with the desired Python version'; exit 1; fi | ||
| @if command -v uv > /dev/null; then echo 'Verified uv is installed'; else echo 'Please install uv by running `curl -LsSf https://astral.sh/uv/install.sh | sh` or visit https://docs.astral.sh/uv/ for more information'; exit 1; fi | ||
| @uv tool update-shell | ||
| @uv python install | ||
| @rm -f .uv-installed-* | ||
| @touch .uv-installed-$(PY_VERSION) | ||
|
|
||
|
|
||
| uv: .uv-installed-$(PY_VERSION) | ||
|
|
||
| .venv: .uv-installed-$(PY_VERSION) | ||
| @uv venv .venv | ||
|
|
||
| venv: .venv | ||
|
|
||
| .git/hooks/pre-commit: .uv-installed-$(PY_VERSION) | ||
| @uv tool install pre-commit | ||
| @uv tool run pre-commit install | ||
|
|
||
| pre-commit: .git/hooks/pre-commit | ||
|
|
||
| dev: .venv .git/hooks/pre-commit | ||
| @uv sync --extra=dev --extra=duckdb | ||
|
|
||
| clean: | ||
| @rm -rf .venv target demo_duckdb/target demo_sqlite/target | ||
|
|
||
| lint: .uv-installed-$(PY_VERSION) | ||
| @uvx ruff check | ||
|
|
||
| format: .uv-installed-$(PY_VERSION) | ||
| @uvx ruff check --fix --select I | ||
| @uvx ruff format --preview | ||
|
|
||
| test: .uv-installed-$(PY_VERSION) | ||
| @uv run pytest tests/ | ||
|
|
||
| scan: .uv-installed-$(PY_VERSION) | ||
| @uvx bandit -r src -b tests/bandit_baseline.json | ||
|
|
||
| scan-new-baseline: .uv-installed-$(PY_VERSION) | ||
| @uvx bandit -r src -f json -o tests/bandit_baseline.json | ||
|
|
||
| scan-without-baseline: .uv-installed-$(PY_VERSION) | ||
| @uvx bandit -r src | ||
|
|
||
| requirements.txt: .uv-installed-$(PY_VERSION) | ||
| @uv export -o requirements.txt --no-hashes --frozen |
Oops, something went wrong.