-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[embedded-elt] add an example for the Sling Decorator (#19999)
## Summary & Motivation This adds a basic example for the Sling integration. It includes a few helper make targets to start a seeded Postgres instance using Docker, as well as a sample Sling replication.yaml and asset decorator.
- Loading branch information
1 parent
e83a5ef
commit 0b8421d
Showing
9 changed files
with
139 additions
and
30 deletions.
There are no files selected for viewing
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
@PHONY: build run clean sync stop check_version | ||
|
||
check_version: | ||
@required_version="1.1.7"; \ | ||
current_version=$$(sling --version | awk '{print $$2}'); \ | ||
if [ "$$(printf '%s\n' "$$required_version" "$$current_version" | sort -V | head -n1)" = "$$required_version" ]; then \ | ||
exit 0; \ | ||
else \ | ||
echo "sling version is less than $$required_version", run pip install --upgrade sling; \ | ||
exit 1; \ | ||
fi | ||
|
||
clean: | ||
docker ps -q -f name=sling-postgres | xargs -r docker stop && docker ps -aq -f name=sling-postgres | xargs -r docker rm | ||
|
||
stop: | ||
docker ps -q -f name=sling-postgres | xargs -r docker stop | ||
|
||
build: | ||
docker build -t sling-postgres . | ||
|
||
run: build stop | ||
docker run --name sling-postgres --detach -p 54321:5432 -ti --rm sling-postgres | ||
|
||
sync: check_version | ||
MY_POSTGRES=postgres://postgres:postgres@localhost:54321/finance?sslmode=disable \ | ||
MY_DUCKDB=duckdb:///var/tmp/duckdb.db \ | ||
sling run -r sling_replication.yaml | ||
|
||
verify: | ||
@echo " POSTGRES" | ||
@echo "--------------------------------" | ||
@echo "SELECT * FROM public.\"all_Users\";" | docker exec -i sling-postgres psql -U postgres -d finance | ||
@echo " DUCKDB" | ||
@echo "--------------------------------" | ||
@echo "SELECT * FROM public.all_users;" | duckdb /var/tmp/duckdb.db |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,72 @@ | ||
# (Experimental) Sling with Asset Decorators | ||
|
||
This is an example of how to use the new Sling `@sling_assets` decorator | ||
to sync data from Postgres to DuckDB. | ||
|
||
In order to run this example, we will setup a Docker instance of Postgres | ||
which some sample data to sync into DuckDB. The following steps will help you | ||
validate that you are able to run the example locally outside of Dagster using | ||
the included replication.yaml | ||
|
||
## Initial Setup | ||
|
||
First, we need to start the Postgres instance. The Makefile offers aliases | ||
to simplify building and running a Postgres instance: | ||
|
||
```bash | ||
make build # builds the docker image | ||
make run # runs the docker image | ||
``` | ||
|
||
Next, validate that everything works locally: | ||
|
||
```bash | ||
pip install --upgrade sling | ||
make sync # runs sling | ||
make verify | ||
``` | ||
|
||
You should see output similar to this | ||
|
||
``` | ||
POSTGRES | ||
-------------------------------- | ||
all_user_id | name | ||
-------------+---------------- | ||
1 | Alice Johnson | ||
2 | Bob Williams | ||
3 | Charlie Miller | ||
(3 rows) | ||
DUCKDB | ||
-------------------------------- | ||
┌─────────────┬────────────────┬──────────────────┐ | ||
│ all_user_id │ name │ _sling_loaded_at │ | ||
│ int32 │ varchar │ int32 │ | ||
├─────────────┼────────────────┼──────────────────┤ | ||
│ 1 │ Alice Johnson │ 1708665267 │ | ||
│ 2 │ Bob Williams │ 1708665267 │ | ||
│ 3 │ Charlie Miller │ 1708665267 │ | ||
└─────────────┴────────────────┴──────────────────┘ | ||
``` | ||
|
||
## Getting started | ||
|
||
To download this example, run: | ||
|
||
```shell | ||
dagster project from-example --name my-dagster-project --example sling_decorator | ||
``` | ||
|
||
To install this example and its dependencies, run: | ||
|
||
```shell | ||
cd my-dagster-project | ||
pip install -e . | ||
``` | ||
|
||
Now run Dagster to load the sample Sling pipeline: | ||
|
||
```shell | ||
dagster dev | ||
``` |
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
[build-system] | ||
requires = ["setuptools", "wheel"] | ||
build-backend = "setuptools.build_meta" | ||
|
||
[project] | ||
name = "sling_decorator_example" | ||
version = "0.1.0" | ||
dependencies = [ | ||
"dagster-webserver","sling>=1.1.7" | ||
] | ||
[tool.setuptools] | ||
packages = ["sling_decorator"] | ||
|
||
[tool.dagster] | ||
module_name = "sling_decorator" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
12 changes: 0 additions & 12 deletions
12
...-elt/dagster_embedded_elt_tests/replication_configs/pg_to_duckdb_with_dag_config/Makefile
This file was deleted.
Oops, something went wrong.
12 changes: 0 additions & 12 deletions
12
...r_embedded_elt_tests/replication_configs/pg_to_duckdb_with_dag_config/README.md
This file was deleted.
Oops, something went wrong.