Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dev Container environment #882

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Dev Container environment #882

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions .devcontainer/devcontainer.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/python
{
"name": "dbt-databricks",
// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
"image": "mcr.microsoft.com/devcontainers/python:1-3.12-bullseye",
ghjklw marked this conversation as resolved.
Show resolved Hide resolved
"features": {
"ghcr.io/devcontainers-extra/features/hatch:2": {}
},
"postStartCommand": "./.devcontainer/initialize.sh",
"customizations": {
"vscode": {
"extensions": [
"ms-python.python",
"charliermarsh.ruff",
"ms-python.mypy-type-checker"
],
"settings": {
"editor.formatOnSave": true,
"editor.formatOnType": true,
"editor.rulers": [
100
],
"files.exclude": {
".hatch": true,
".mypy_cache": true,
".pytest_cache": true,
"**/__pycache__": true
},
"python.analysis.diagnosticMode": "workspace",
ghjklw marked this conversation as resolved.
Show resolved Hide resolved
"python.analysis.enablePytestSupport": true,
"python.defaultInterpreterPath": ".hatch/bin/python",
"python.envFile": "test.env",
"python.testing.pytestEnabled": true,
"python.languageServer": "Pylance",
"ruff.fixAll": true,
"ruff.format.args": [
"--config=pyproject.toml"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't think this is necessary; I would think it would default to local pyproject.toml.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're probably right, I will test and remove if unnecessary.

],
"ruff.lint.args": [
"--config=pyproject.toml"
],
"[python]": {
"editor.defaultFormatter": "charliermarsh.ruff",
"editor.codeActionsOnSave": {
"source.organizeImports": "always",
"source.fixAll": "always"
}
},
}
}
},
"containerEnv": {
"WORKSPACE_FOLDER": "${containerWorkspaceFolder}"
}
}
14 changes: 14 additions & 0 deletions .devcontainer/initialize.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!/usr/bin/env bash
set -x

# Create the Hatch environment
hatch env create

# Install pre-commit
(hatch run setup-precommit && pre-commit gc) &

# Set up test.env file if it does not already exist
if test -f ${WORKSPACE_FOLDER}/test.env; then
cp ${WORKSPACE_FOLDER}/test.env.example ${WORKSPACE_FOLDER}/test.env
fi

1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,4 @@ logs/
*.sublime*
.python-version
.hatch
databricks_demo
22 changes: 9 additions & 13 deletions CONTRIBUTING.MD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,7 @@ We follow [PEP 8](https://www.python.org/dev/peps/pep-0008/) with one exception:

### Linting

This project uses [Black](https://pypi.org/project/black/), [flake8](https://flake8.pycqa.org/en/latest/), and [mypy](https://www.mypy-lang.org/) for linting and static type checks. Run all three with the `linter` command and commit before opening your pull request.

```
tox -e linter
```
This project uses [Ruff](https://docs.astral.sh/ruff/) and [mypy](https://www.mypy-lang.org/) for linting and static type checks.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!


To simplify reviews you can commit any format changes in a separate commit.

Expand Down Expand Up @@ -118,21 +114,21 @@ If you set your `user.name` and `user.email` git configs, you can sign your comm
Unit tests do not require a Databricks account. Please confirm that your pull request passes our unit test suite before opening a pull request.

```bash
tox -e unit
hatch run unit
```

## Functional Tests

Functional tests require a Databricks account with access to a workspace containing three specific compute resources as detailed below.
The `tox` commands to run each set of these tests appear below:
The `hatch` commands to run each set of these tests appear below:

| Compute Type | Unity Catalog | Command |
| ------------------- | ------------- | ----------------------------------------------- |
| SQL Warehouse | Yes | `tox -e integration-databricks-uc-sql-endpoint` |
| All Purpose Cluster | Yes | `tox -e integration-databricks-uc-cluster` |
| All Purpose | No | `tox -e integration-databricks-cluster` |
| Compute Type | Unity Catalog | Command |
| ------------------- | ------------- | -------------------------- |
| SQL Warehouse | Yes | `hatch run sqlw-e2e` |
| All Purpose Cluster | Yes | `hatch run uc-cluster-e2e` |
| All Purpose | No | `hatch run cluster-e2e` |

These tests are configured with environment variables that `tox` reads from a file called [test.env](/test.env.example) which you can copy from the example:
These tests are configured with environment variables that `hatch` reads from a file called [test.env](/test.env.example) which you can copy from the example:

```sh
cp test.env.example test.env
Expand Down
Loading