Skip to content

Commit

Permalink
restructure to suggested layout
Browse files Browse the repository at this point in the history
  • Loading branch information
cmpadden committed Aug 8, 2024
1 parent 0f9c772 commit 1111ece
Showing 1 changed file with 20 additions and 58 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -12,67 +12,21 @@ Existing integrations that implement this approach include:
- [Embedded ELT: dlt](https://docs.dagster.io/integrations/embedded-elt/dlt)
- [Embedded ELT: Sling](https://docs.dagster.io/integrations/embedded-elt/sling)

## Background
In this guide, you'll develop a new multi-asset integration that takes a YAML file as input and produces a multi-asset. This will allow your end users to customize the translation of the definition spec and understand how it maps to Dagster concepts.

### What even is a decorator?

Before we dive into writing an integration that provides a multi-asset decorator, let's take a step back and review what a decorator is, and why you may consider using it.

A Python decorator is a function that wraps another function by adding an annotation above the function prefixed with an `@` symbol. It allows you to modify the behavior of an existing function without changing the source code of that function directly.

For example, say we have a function that performs some kind of computation.

```python
def add(a: int, b: int) -> int:
return a + b
```

Say we want to double the output of a function. We can create a decorator that modifies the output of the function that it wraps, and return the new modified value.

<Note>
A function that takes another function as input, or returns a function as a
result, is traditionally called a{" "}
<a href="https://en.wikipedia.org/wiki/Higher-order_function">
higher-order function
</a>
.
</Note>

To define a decorator, you write a function that takes a function as an input argument. Within this function, you define a wrapper function that encapsulates the function you're modifying, and then this wrapper function is returned by your decorator.

```python
def double(func):
def _wrapper(*args, **kwargs):
result = func(*args, **kwargs)
if isinstance(result, (int, float)):
return result * 2
else:
return result
return _wrapper
```

We've defined a function called `double` that implements a `_wrapper` function. This function accepts variable `*args` and `**kwargs`, passing them to the inner function that it wraps. It calls the function it wraps, and modifies the output if the result is a numeric type.

```python
@double
def add(a: int, b: int) -> int:
return a + b

# >>> add(2, 3)
# 10
```
---

Decorators are extremely powerful in that you can perform operations before and after the function they wrap, and you can also manipulate the value returned by the function itself.
## Prerequisites

### Why decorators in Dagster
To follow the steps in this guide, **you'll need familiarity with Dagster and Python decorators**. Reviewing Python functionality is out of scope for this guide, but you can refer to [Real Python's Primer on Python Decorators](https://realpython.com/primer-on-python-decorators/) to learn the fundamentals.

In the context of Dagster, decorators are helpful in that we are often wrapping some form of processing. For example, when writing an asset, you define your processing code, and by annotating that function with the <PyObject object="asset" decorator /> decorator. Then, the internal Dagster code can register the asset, assign metadata, pass in context data, or perform any other variety of operations that are required to integrate your asset code with the Dagster platform.
### Why decorators?

## Walkthrough
In the context of Dagster, decorators are helpful in that we are often wrapping some form of processing. For example, when writing an asset, you define your processing code, and by annotating that function with the @asset decorator. Then, the internal Dagster code can register the asset, assign metadata, pass in context data, or perform any other variety of operations that are required to integrate your asset code with the Dagster platform.

Now that we have a general idea of what a decorator is, and why it's a useful to in the context of developing Dagster code, let's walk through the development of a new multi-asset integration. This integration will take a YAML file, and produce a multi-asset, allowing the end-user to customize the translation of definition spec to how it maps to Dagster concepts.
---

### Input
## Step 1: Input

This hypothetical tool is configured using a YAML definition file where someone can define source and destination databases, along with the tables that they would like to replicate.

Expand Down Expand Up @@ -112,7 +66,9 @@ def replicate(replication_configuration_yaml: Path) -> Iterator[Mapping[str, Any
yield {"table": table.get("name"), "status": "success"}
```
### Implementation
---
## Step 2: Implementation
First, let's define a `Project` object that takes in the path of our configuration YAML file. This will allow us to encapsulate the logic that gets metadata and table information from our project configuration.

Expand Down Expand Up @@ -182,7 +138,9 @@ You may have noticed, however, that there are a couple of limitations to this ap

For the first limitation, we can resolve them by refactoring the code in the body of our asset function into a `Resource`.

### Moving our replication logic into a Resource
---

## Step 3: Moving our replication logic into a Resource

To accomplish this, we will extend the `ConfigurableResource` object to create our own custom resource. Then, we will define a `run` method that will perform the replication operation.

Expand Down Expand Up @@ -217,7 +175,9 @@ def my_assets(replication_resource: ReplicationProject):
replication_resource.run(replication_project)
```

### Using translators
---

## Step 4: Using translators

Previously we mentioned that there was no way for an end-user to customize the asset definitions, for example, changing the key of the asset. The recommended way to accomplish this is through the use of a translator class.

Expand Down Expand Up @@ -300,7 +260,9 @@ class ReplicationResource(ConfigurableResource):
)
```

### Conclusion
---

## Conclusion

In this guide we walked through how to define a custom multi-asset decorator, a resource for encapsulating tool logic, and a translator for defining the logic to translate a specification to Dagster concepts.

Expand Down

0 comments on commit 1111ece

Please sign in to comment.