Skip to content

Commit

Permalink
[docs] Move Sling docs out of embedded-elt docs (#27073)
Browse files Browse the repository at this point in the history
## Summary & Motivation

As title. Examples in `/examples` are also updated with their guides.
  • Loading branch information
maximearmstrong authored Jan 14, 2025
1 parent 3175a31 commit ab7b45e
Show file tree
Hide file tree
Showing 34 changed files with 113 additions and 137 deletions.
14 changes: 7 additions & 7 deletions docs/content/_apidocs.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -438,13 +438,6 @@ Dagster also provides a growing set of optional add-on libraries to integrate wi
</td>
<td>Provides support for storing PySpark DataFrames in DuckDB.</td>
</tr>
<tr>
<td>
<a href="/_apidocs/libraries/dagster-embedded-elt">Embedded ELT</a> (
<code>dagster-embedded-elt</code>)
</td>
<td>Provides support for running embedded ELT within Dagster</td>
</tr>
<tr>
<td>
<a href="/_apidocs/libraries/dagster-fivetran">Fivetran</a> (
Expand Down Expand Up @@ -665,6 +658,13 @@ Dagster also provides a growing set of optional add-on libraries to integrate wi
</td>
<td>Provides a simple integration with Slack.</td>
</tr>
<tr>
<td>
<a href="/_apidocs/libraries/dagster-sling">dlt</a> (
<code>dagster-sling</code>)
</td>
<td>Provides support for running Sling within Dagster</td>
</tr>
<tr>
<td>
<a href="/_apidocs/libraries/dagster-snowflake">Snowflake</a> (
Expand Down
10 changes: 5 additions & 5 deletions docs/content/_navigation.json
Original file line number Diff line number Diff line change
Expand Up @@ -1122,7 +1122,7 @@
},
{
"title": "Sling & Dagster",
"path": "/integrations/embedded-elt/sling"
"path": "/integrations/sling"
}
]
},
Expand Down Expand Up @@ -1594,10 +1594,6 @@
"title": "DuckDB & Polars (dagster-duckdb-polars)",
"path": "/_apidocs/libraries/dagster-duckdb-polars"
},
{
"title": "Embedded ELT (dagster-embedded-elt)",
"path": "/_apidocs/libraries/dagster-embedded-elt"
},
{
"title": "Fivetran (dagster-fivetran)",
"path": "/_apidocs/libraries/dagster-fivetran"
Expand Down Expand Up @@ -1698,6 +1694,10 @@
"title": "Slack (dagster-slack)",
"path": "/_apidocs/libraries/dagster-slack"
},
{
"title": "Sling (dagster-sling)",
"path": "/_apidocs/libraries/dagster-sling"
},
{
"title": "Snowflake (dagster-snowflake)",
"path": "/_apidocs/libraries/dagster-snowflake"
Expand Down
Binary file modified docs/content/api/modules.json.gz
Binary file not shown.
Binary file modified docs/content/api/searchindex.json.gz
Binary file not shown.
Binary file modified docs/content/api/sections.json.gz
Binary file not shown.
9 changes: 5 additions & 4 deletions docs/content/integrations.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ Explore guides for integrations with external services.
title="Pandera"
href="/integrations/pandera"
></ArticleListItem>
<ArticleListItem title="Sling" href="/integrations/sling"></ArticleListItem>
<ArticleListItem title="Spark" href="/integrations/spark"></ArticleListItem>
<ArticleListItem
title="Snowflake"
Expand Down Expand Up @@ -143,10 +144,6 @@ Explore libraries that are maintained by the Dagster core team.
title="dlt"
href="/_apidocs/libraries/dagster-dlt"
></ArticleListItem>
<ArticleListItem
title="Embedded ELT"
href="/_apidocs/libraries/dagster-embedded-elt"
></ArticleListItem>
<ArticleListItem
title="Fivetran"
href="/_apidocs/libraries/dagster-fivetran"
Expand Down Expand Up @@ -223,6 +220,10 @@ Explore libraries that are maintained by the Dagster core team.
title="Slack"
href="/_apidocs/libraries/dagster-slack"
></ArticleListItem>
<ArticleListItem
title="Sling"
href="/_apidocs/libraries/dagster-sling"
></ArticleListItem>
<ArticleListItem
title="Snowflake"
href="/_apidocs/libraries/dagster-snowflake"
Expand Down
2 changes: 1 addition & 1 deletion docs/content/integrations/dlt.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -398,6 +398,6 @@ Want to see real-world examples of dlt in production? Check out how we use it in
></ArticleListItem>
<ArticleListItem
title="Sling & Dagster"
href="/integrations/embedded-elt/sling"
href="/integrations/sling"
></ArticleListItem>
</ArticleList>
2 changes: 1 addition & 1 deletion docs/content/integrations/embedded-elt.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ We plan to add additional embedded ELT tool integrations in the future.

Sling provides an easy-to-use YAML configuration layer for loading data from files, replicating data between databases, exporting custom SQL queries to cloud storage, and much more. The Dagster integration allows you to derive Dagster assets from a replication configuration file.

[Click here to get started](/integrations/embedded-elt/sling) with the Sling integration.
[Click here to get started](/integrations/sling) with the Sling integration.

---

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@ The Dagster integration allows you to derive Dagster assets from a replication c

1. Define a Sling [`replication.yaml`](https://docs.slingdata.io/sling-cli/run/configuration/replication) file that specifies the source and target connections, as well as which streams to sync from.

2. Create a <PyObject module="dagster_embedded_elt.sling" object="SlingResource" /> and pass a list of <PyObject module="dagster_embedded_elt.sling" object="SlingConnectionResource" /> for each connection to the `connection` parameter, ensuring the resource uses the same name given to the connection in the Sling configuration.
2. Create a <PyObject module="dagster_sling" object="SlingResource" /> and pass a list of <PyObject module="dagster_sling" object="SlingConnectionResource" /> for each connection to the `connection` parameter, ensuring the resource uses the same name given to the connection in the Sling configuration.

3. Use the <PyObject module="dagster_embedded_elt.sling" object="sling_assets" decorator /> decorator to define an asset that runs the Sling replication job and yields from the <PyObject module="dagster_embedded_elt.sling" object="SlingResource" method="replicate" /> method to run the sync.
3. Use the <PyObject module="dagster_sling" object="sling_assets" decorator /> decorator to define an asset that runs the Sling replication job and yields from the <PyObject module="dagster_sling" object="SlingResource" method="replicate" /> method to run the sync.

We'll walk you through each of these steps in this guide.

Expand All @@ -27,11 +27,11 @@ We'll walk you through each of these steps in this guide.

To follow the steps in this guide:

- **Familiarize yourself with [Sling's replication configuration](https://docs.slingdata.io/sling-cli/run/configuration/replication)**, if you've never worked with Sling before. The replication configuration is a YAML file that specifies the source and target connections, as well as which streams to sync from. The `dagtser-embedded-elt` integration uses this configuration to build assets for both sources and destinations.
- **Familiarize yourself with [Sling's replication configuration](https://docs.slingdata.io/sling-cli/run/configuration/replication)**, if you've never worked with Sling before. The replication configuration is a YAML file that specifies the source and target connections, as well as which streams to sync from. The `dagtser-sling` integration uses this configuration to build assets for both sources and destinations.
- **To install the following libraries**:

```bash
pip install dagster dagster-embedded-elt
pip install dagster dagster-sling
```

Refer to the [Dagster installation](/getting-started/install) guide for more info.
Expand Down Expand Up @@ -73,7 +73,7 @@ streams:
This example creates a replication configuration using Python:
```python file=/integrations/embedded_elt/replication_config.py
```python file=/integrations/sling/replication_config.py
replication_config = {
"source": "MY_POSTGRES",
"target": "MY_DUCKDB",
Expand All @@ -93,10 +93,10 @@ replication_config = {

## Step 2: Create a Sling resource

Next, you'll create a <PyObject module="dagster_embedded_elt.sling" object="SlingResource" /> object that contains references to the connections specified in the replication configuration:
Next, you'll create a <PyObject module="dagster_sling" object="SlingResource" /> object that contains references to the connections specified in the replication configuration:

```python file=/integrations/embedded_elt/sling_connection_resources.py
from dagster_embedded_elt.sling import SlingConnectionResource, SlingResource
```python file=/integrations/sling/sling_connection_resources.py
from dagster_sling import SlingConnectionResource, SlingResource

from dagster import EnvVar

Expand Down Expand Up @@ -126,22 +126,22 @@ sling_resource = SlingResource(
)
```

A <PyObject module="dagster_embedded_elt.sling" object="SlingResource" /> takes a `connections` parameter, where each <PyObject module="dagster_embedded_elt.sling" object="SlingConnectionResource" /> represents a connection to a source or target database. You may provide as many connections to the `SlingResource` as needed.
A <PyObject module="dagster_sling" object="SlingResource" /> takes a `connections` parameter, where each <PyObject module="dagster_sling" object="SlingConnectionResource" /> represents a connection to a source or target database. You may provide as many connections to the `SlingResource` as needed.

The `name` parameter in the <PyObject module="dagster_embedded_elt.sling" object="SlingConnectionResource" /> should match the `source` and `target` keys in the replication configuration.
The `name` parameter in the <PyObject module="dagster_sling" object="SlingConnectionResource" /> should match the `source` and `target` keys in the replication configuration.

You can pass a connection string or arbitrary keyword arguments to the <PyObject module="dagster_embedded_elt.sling" object="SlingConnectionResource" /> to specify the connection details. Refer to [Sling's connections reference](https://docs.slingdata.io/connections/database-connections) for the specific connection types and parameters.
You can pass a connection string or arbitrary keyword arguments to the <PyObject module="dagster_sling" object="SlingConnectionResource" /> to specify the connection details. Refer to [Sling's connections reference](https://docs.slingdata.io/connections/database-connections) for the specific connection types and parameters.

---

## Step 3: Define the Sling assets

Next, define a Sling asset using the <PyObject module="dagster_embedded_elt.sling" object="sling_assets" decorator /> decorator. Dagster will read the replication configuration to produce assets.
Next, define a Sling asset using the <PyObject module="dagster_sling" object="sling_assets" decorator /> decorator. Dagster will read the replication configuration to produce assets.

Each stream will render two assets, one for the source stream and one for the target destination. You can override how assets are named by passing in a custom <PyObject module="dagster_embedded_elt.sling" object="DagsterSlingTranslator" /> object.
Each stream will render two assets, one for the source stream and one for the target destination. You can override how assets are named by passing in a custom <PyObject module="dagster_sling" object="DagsterSlingTranslator" /> object.

```python file=/integrations/embedded_elt/sling_dagster_translator.py lines=1-16
from dagster_embedded_elt.sling import SlingResource, sling_assets
```python file=/integrations/sling/sling_dagster_translator.py lines=1-16
from dagster_sling import SlingResource, sling_assets

from dagster import Definitions, file_relative_path

Expand All @@ -165,7 +165,7 @@ defs = Definitions(

The last step is to include the Sling assets and resource in a <PyObject object="Definitions" /> object. This enables Dagster tools to load everything we've defined:

```python file=/integrations/embedded_elt/sling_dagster_translator.py lines=19-26
```python file=/integrations/sling/sling_dagster_translator.py lines=19-26
],
resources={
"sling": sling_resource,
Expand All @@ -183,12 +183,8 @@ That's it! You should now be able to view your assets in the [Dagster UI](/conce

To set up a Sling sync between two databases, such as Postgres and Snowflake, you could do something like the following:

```python file=/integrations/embedded_elt/postgres_snowflake.py
from dagster_embedded_elt.sling import (
SlingConnectionResource,
SlingResource,
sling_assets,
)
```python file=/integrations/sling/postgres_snowflake.py
from dagster_sling import SlingConnectionResource, SlingResource, sling_assets

from dagster import EnvVar

Expand Down Expand Up @@ -239,12 +235,8 @@ def my_assets(context, sling: SlingResource):

To set up a Sling sync between a file in an object store and a database, such as from Amazon S3 to Snowflake, you could do something like the following:

```python file=/integrations/embedded_elt/s3_snowflake.py startafter=start_storage_config endbefore=end_storage_config
from dagster_embedded_elt.sling import (
SlingConnectionResource,
SlingResource,
sling_assets,
)
```python file=/integrations/sling/s3_snowflake.py startafter=start_storage_config endbefore=end_storage_config
from dagster_sling import SlingConnectionResource, SlingResource, sling_assets

from dagster import EnvVar

Expand Down Expand Up @@ -290,12 +282,12 @@ def my_assets(context, sling: SlingResource):

## APIs in this guide

| Name | Description |
| --------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| <PyObject module="dagster_embedded_elt.sling" object="sling_assets" decorator /> | The core Sling asset factory for building syncs |
| <PyObject module="dagster_embedded_elt.sling" object="SlingResource" /> | The Sling resource used for handing credentials to databases and object stores |
| <PyObject module="dagster_embedded_elt.sling" object="DagsterSlingTranslator" /> | A translator for specifying how to map between Sling and Dagster types |
| <PyObject module="dagster_embedded_elt.sling" object="SlingConnectionResource" /> | A Sling connection resource for specifying database and storage connection credentials |
| Name | Description |
| -------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| <PyObject module="dagster_sling" object="sling_assets" decorator /> | The core Sling asset factory for building syncs |
| <PyObject module="dagster_sling" object="SlingResource" /> | The Sling resource used for handing credentials to databases and object stores |
| <PyObject module="dagster_sling" object="DagsterSlingTranslator" /> | A translator for specifying how to map between Sling and Dagster types |
| <PyObject module="dagster_sling" object="SlingConnectionResource" /> | A Sling connection resource for specifying database and storage connection credentials |

---

Expand Down
6 changes: 3 additions & 3 deletions docs/docs-beta/docs/integrations/libraries/sling.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ title: Dagster & Sling
sidebar_label: Sling
excerpt: Extract and load data from popular data sources to destinations with Sling through Dagster.
date: 2024-08-30
apireflink: https://docs.dagster.io/_apidocs/libraries/dagster-embedded-elt
docslink: https://docs.dagster.io/integrations/embedded-elt/sling
apireflink: https://docs.dagster.io/_apidocs/libraries/dagster-sling
docslink: https://docs.dagster.io/integrations/sling
partnerlink: https://slingdata.io/
categories:
- ETL
Expand All @@ -23,7 +23,7 @@ This integration allows you to use [Sling](https://slingdata.io/) to extract and
### Installation

```bash
pip install dagster-embedded-elt
pip install dagster-sling
```

### Example
Expand Down
Binary file modified docs/next/public/objects.inv
Binary file not shown.
10 changes: 10 additions & 0 deletions docs/next/util/redirectUrls.json
Original file line number Diff line number Diff line change
Expand Up @@ -763,5 +763,15 @@
"source": "/integrations/embedded-elt/dlt",
"destination": "/integrations/dlt",
"statusCode": 302
},
{
"source": "/integrations/embedded-elt/sling",
"destination": "/integrations/sling",
"statusCode": 302
},
{
"source": "/_apidocs/libraries/dagster-embedded-elt",
"destination": "/integrations/embedded-elt",
"statusCode": 302
}
]
2 changes: 1 addition & 1 deletion docs/sphinx/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,6 @@
sections/api/apidocs/libraries/dagster-duckdb-pandas
sections/api/apidocs/libraries/dagster-duckdb-pyspark
sections/api/apidocs/libraries/dagster-duckdb-polars
sections/api/apidocs/libraries/dagster-embedded-elt
sections/api/apidocs/libraries/dagster-fivetran
sections/api/apidocs/libraries/dagster-docker
sections/api/apidocs/libraries/dagster-gcp
Expand All @@ -75,6 +74,7 @@
sections/api/apidocs/libraries/dagster-shell
sections/api/apidocs/libraries/dagster-sigma
sections/api/apidocs/libraries/dagster-slack
sections/api/apidocs/libraries/dagster-sling
sections/api/apidocs/libraries/dagster-snowflake
sections/api/apidocs/libraries/dagster-snowflake-pandas
sections/api/apidocs/libraries/dagster-snowflake-pyspark
Expand Down

This file was deleted.

24 changes: 24 additions & 0 deletions docs/sphinx/sections/api/apidocs/libraries/dagster-sling.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
dlt (dagster-dlt)
---------------------------

This library provides a Dagster integration with `Sling <https://slingdata.io>`_.

For more information on getting started, see the `Sling & Dagster <https://docs.dagster.io/integrations/sling>`_ documentation.


.. currentmodule:: dagster_sling

Assets
======

.. autodecorator:: sling_assets

.. autoclass:: DagsterSlingTranslator

Resources
=========

.. autoclass:: SlingResource
:members: replicate

.. autoclass:: SlingConnectionResource
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
from dagster_embedded_elt.sling import (
SlingConnectionResource,
SlingResource,
sling_assets,
)
from dagster_sling import SlingConnectionResource, SlingResource, sling_assets

import dagster as dg

Expand Down
1 change: 1 addition & 0 deletions examples/docs_beta_snippets/tox.ini
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ deps =
-e ../../python_modules/libraries/dagster-duckdb
-e ../../python_modules/libraries/dagster-duckdb-pandas
-e ../../python_modules/libraries/dagster-embedded-elt
-e ../../python_modules/libraries/dagster-sling
-e ../../python_modules/libraries/dagster-aws
-e ../../python_modules/libraries/dagster-snowflake-pandas
-e ../../python_modules/libraries/dagster-gcp
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,7 @@
# pyright: reportCallIssue=none
# pyright: reportOptionalMemberAccess=none

from dagster_embedded_elt.sling import (
SlingConnectionResource,
SlingResource,
sling_assets,
)
from dagster_sling import SlingConnectionResource, SlingResource, sling_assets

from dagster import EnvVar

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,7 @@
# pyright: reportOptionalMemberAccess=none

# start_storage_config
from dagster_embedded_elt.sling import (
SlingConnectionResource,
SlingResource,
sling_assets,
)
from dagster_sling import SlingConnectionResource, SlingResource, sling_assets

from dagster import EnvVar

Expand Down
Loading

2 comments on commit ab7b45e

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-docs ready!

✅ Preview
https://dagster-docs-9d2zuouw5-elementl.vercel.app
https://master.dagster.dagster-docs.io

Built with commit ab7b45e.
This pull request is being automatically deployed with vercel-action

@github-actions
Copy link

@github-actions github-actions bot commented on ab7b45e Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-docs-beta ready!

✅ Preview
https://dagster-docs-beta-3sfwq7b68-elementl.vercel.app

Built with commit ab7b45e.
This pull request is being automatically deployed with vercel-action

Please sign in to comment.