Skip to content

Commit

Permalink
[docs] Add more explanation for FilePathMapping for code references (#…
Browse files Browse the repository at this point in the history
…23385)

## Summary

Adds .rst entry and link to apidoc for file path mapping objects, since
this is probably the most confusing part of mapping to source control.

## Test Plan

Vercel preview.

---------

Co-authored-by: Erin Cochran <[email protected]>
  • Loading branch information
benpankow and erinkcochran87 authored Aug 6, 2024
1 parent bea0af6 commit 07b5902
Show file tree
Hide file tree
Showing 8 changed files with 66 additions and 5 deletions.
Binary file modified docs/content/api/modules.json.gz
Binary file not shown.
Binary file modified docs/content/api/searchindex.json.gz
Binary file not shown.
Binary file modified docs/content/api/sections.json.gz
Binary file not shown.
4 changes: 2 additions & 2 deletions docs/content/guides/dagster/code-references.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ Dagster's dbt integration can automatically attach references to the SQL files b

In some cases, you may want to manually attach code references to your asset definitions. Some assets may have a more complex source structure, such as an asset whose definition is spread across multiple Python source files or an asset which is partially defined with a `.sql` model file.

To manually attach code references to an asset definition, use `CodeReferencesMetadataValue`. You can then choose to augment these manual references with `with_source_code_references`:
To manually attach code references to an asset definition, use <PyObject module="dagster" object="CodeReferencesMetadataValue"/>. You can then choose to augment these manual references with <PyObject module="dagster" object="with_source_code_references"/>:

```python file=/guides/dagster/code_references/manual_references.py
import os
Expand Down Expand Up @@ -165,7 +165,7 @@ defs = Definitions(

### In any Dagster environment

The <PyObject module="dagster" object="link_code_references_to_git"/> utility allows you to convert local file code references to source control links. You'll need to provide the base URL of your git repository, the branch or commit hash, and a path to the repository root locally.
The <PyObject module="dagster" object="link_code_references_to_git"/> utility allows you to convert local file code references to source control links. You'll need to provide the base URL of your git repository, the branch or commit hash, and a <PyObject module="dagster" object="FilePathMapping"/> which tells Dagster how to convert local file paths to paths in the repository. The simplest way to do so is with an <PyObject module="dagster" object="AnchorBasedFilePathMapping"/>, which uses a local file path and the corresponding path in the repository to infer the mapping for other files.

```python file=/guides/dagster/code_references/link_to_source_control.py
from pathlib import Path
Expand Down
Binary file modified docs/next/public/objects.inv
Binary file not shown.
6 changes: 6 additions & 0 deletions docs/sphinx/sections/api/apidocs/metadata.rst
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@ All metadata types inherit from `MetadataValue`. The following types are defined

.. autoclass:: UrlMetadataValue

.. autoclass:: CodeReferencesMetadataValue

Tables
^^^^^^

Expand Down Expand Up @@ -76,3 +78,7 @@ For more information, refer to the `Linking to asset definition code with code r
.. autofunction:: with_source_code_references

.. autofunction:: link_code_references_to_git

.. autoclass:: FilePathMapping

.. autoclass:: AnchorBasedFilePathMapping
38 changes: 37 additions & 1 deletion examples/quickstart_etl/quickstart_etl/definitions.py
Original file line number Diff line number Diff line change
@@ -1,16 +1,52 @@
from pathlib import Path

from dagster import (
Definitions,
ScheduleDefinition,
define_asset_job,
graph_asset,
link_code_references_to_git,
load_assets_from_package_module,
op,
with_source_code_references,
)
from dagster._core.definitions.metadata.source_code import AnchorBasedFilePathMapping

from . import assets

daily_refresh_schedule = ScheduleDefinition(
job=define_asset_job(name="all_assets_job"), cron_schedule="0 0 * * *"
)


@op
def foo_op():
return 5


@graph_asset
def my_asset():
return foo_op()


my_assets = with_source_code_references(
[
my_asset,
*load_assets_from_package_module(assets),
]
)

my_assets = link_code_references_to_git(
assets_defs=my_assets,
git_url="https://github.com/dagster-io/dagster/",
git_branch="master",
file_path_mapping=AnchorBasedFilePathMapping(
local_file_anchor=Path(__file__).parent,
file_anchor_path_in_repository="examples/quickstart_etl/quickstart_etl/",
),
)

defs = Definitions(
assets=load_assets_from_package_module(assets), schedules=[daily_refresh_schedule]
assets=my_assets,
schedules=[daily_refresh_schedule],
)
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
from typing import TYPE_CHECKING, Any, Callable, List, Optional, Sequence, Union

import dagster._check as check
from dagster._annotations import experimental
from dagster._annotations import experimental, public
from dagster._model import DagsterModel
from dagster._serdes import whitelist_for_serdes

Expand Down Expand Up @@ -160,8 +160,17 @@ class FilePathMapping(ABC):
mapping function can be provided to handle these cases.
"""

@public
@abstractmethod
def convert_to_source_control_path(self, local_path: Path) -> str: ...
def convert_to_source_control_path(self, local_path: Path) -> str:
"""Maps a local file path to the corresponding path in a source control repository.
Args:
local_path (Path): The local file path to map.
Returns:
str: The corresponding path in the hosted source control repository, relative to the repository root.
"""


@experimental
Expand Down Expand Up @@ -192,7 +201,17 @@ class AnchorBasedFilePathMapping(FilePathMapping):
local_file_anchor: Path
file_anchor_path_in_repository: str

@public
def convert_to_source_control_path(self, local_path: Path) -> str:
"""Maps a local file path to the corresponding path in a source control repository
based on the anchor file and its corresponding path in the repository.
Args:
local_path (Path): The local file path to map.
Returns:
str: The corresponding path in the hosted source control repository, relative to the repository root.
"""
path_from_anchor_to_target = os.path.relpath(
local_path,
self.local_file_anchor,
Expand Down

1 comment on commit 07b5902

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-docs ready!

✅ Preview
https://dagster-docs-hgq9lftvp-elementl.vercel.app
https://master.dagster.dagster-docs.io

Built with commit 07b5902.
This pull request is being automatically deployed with vercel-action

Please sign in to comment.