[docs][guide] writing a multi-asset backed integration

[cmpadden]

• Introduces a guide that walks through the process of building a multi-asset decorator with translator

NOTE: Once content has been reviewed, I can refactor the snippets to docs_snippets for proper linting.

[cmpadden]

• [docs][guide] writing a multi-asset backed integration #22910 👈
• [docs][guide] approaches to writing integrations #22903
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @cmpadden and the rest of your teammates on Graphite

[github-actions]

Deploy preview for dagster-docs ready!

Preview available at https://dagster-docs-82q5f1ylh-elementl.vercel.app
https://colton-guide-integration-multi-asset.dagster.dagster-docs.io

Direct link to changed pages:

• https://dagster-docs-82q5f1ylh-elementl.vercel.app
  https://colton-guide-integration-multi-asset.dagster.dagster-docs.io/guides/integrations/approaches-to-writing-integrations
• https://dagster-docs-82q5f1ylh-elementl.vercel.app
  https://colton-guide-integration-multi-asset.dagster.dagster-docs.io/guides/integrations/writing-a-multi-asset-decorator-integration

[danielgafni]

I think this content looks really great!

I have 2 thoughts:

1. Might make sense to add a sentence (or a short code snippet) explicitly explaining that asset dependencies will still work (and why)
2. We might want to link this blogpost (what a coincidence!) which explains how to go further into more advanced asset customization (like providing common data transformations on library side). The tutorial suggests creating a resource to handle common logic, and this is an alternative approach. It might be slightly more convenient because the user doesn't need to call the resource. I don't know how often is something like this needed, just a suggestion.
3. Should the tutorial also provide an example of writing an IOManager to handle common IO instead doing it in a Resource? Or perhaps just link to IOManager docs?

[cmpadden]

I think this content looks really great!

I have 2 thoughts:

1. Might make sense to add a sentence (or a short code snippet) explicitly explaining that asset dependencies will still work (and why)

2. We might want to link [this](https://dagster.io/blog/unlocking-flexible-pipelines-customizing-asset-decorator) blogpost (what a coincidence!) which explains how to go further into more advanced asset customization (like providing common data transformations on library side). The tutorial suggests creating a resource to handle common logic, and this is an alternative approach. It might be slightly more convenient because the user doesn't need to call the resource. I don't know how often is something like this needed, just a suggestion.

3. Should the tutorial also provide an example of writing an `IOManager` to handle common IO instead doing it in a `Resource`? Or perhaps just link to `IOManager` docs?

Thanks for the feedback, @danielgafni.

I've included a link to your blog post - what a coincidence indeed! And made it more explicit in that dependencies can be defined.

I'm going to hold off on I/O managers for now as we don't do that for the Sling or dlt integration.

Edit: I've also included links in the side menu.

[erinkcochran87]

Back to you!

[cmpadden]

Thanks again for your feedback, @erinkcochran87 , I believe I have addressed it all.

[yuhan]

left comments around IA/navigation. overall looking good - we may be able to simplify the metadata threading, but i think we can land this and iterate.

[yuhan]

feel free to push back, and we can address it in the jobs-to-be-done world

i wonder if we could add a "when to model your integration in this way" section, where we can explain the whys (e.g. composability), map pattern to category like "ETL", "BI" and also list example implementations like you did on the "Approaches to writing integrations"

[cmpadden]

I do like that idea - however, let's see if we can merge this first, and then make improvements.

Much of this may be rewritten or scrapped as a part of the docs overhaul. If we can merge this, then we can consider restructuring in the docathon if that works for you.

[cmpadden]

Merge activity

• Aug 23, 2:46 PM EDT: @cmpadden started a stack merge that includes this pull request via Graphite.