Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

clarify use cases for packages #4410

Merged
merged 14 commits into from
Nov 29, 2023
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 2 additions & 8 deletions website/docs/docs/build/packages.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,15 +25,9 @@ dbt _packages_ are in fact standalone dbt projects, with models and macros that
* It's important to note that defining and installing dbt packages is different from [defining and installing Python packages](/docs/build/python-models#using-pypi-packages)


:::info `dependencies.yml` has replaced `packages.yml`
Starting from dbt v1.6, `dependencies.yml` has replaced `packages.yml`. This file can now contain both types of dependencies: "package" and "project" dependencies.
- "Package" dependencies lets you add source code from someone else's dbt project into your own, like a library.
- "Project" dependencies provide a different way to build on top of someone else's work in dbt. Refer to [Project dependencies](/docs/collaborate/govern/project-dependencies) for more info.
-
You can rename `packages.yml` to `dependencies.yml`, _unless_ you need to use Jinja within your packages specification. This could be necessary, for example, if you want to add an environment variable with a git token in a private git package specification.

:::
import UseCaseInfo from '/snippets/_packages_or_dependencies.md';

<UseCaseInfo/>

## How do I add a package to my project?
1. Add a file named <VersionBlock firstVersion="1.6"> `dependencies.yml` or </VersionBlock> `packages.yml` to your dbt project. This should be at the same level as your `dbt_project.yml` file.
Expand Down
22 changes: 5 additions & 17 deletions website/docs/docs/collaborate/govern/project-dependencies.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,12 @@ This year, dbt Labs is introducing an expanded notion of `dependencies` across m
- **Packages** &mdash; Familiar and pre-existing type of dependency. You take this dependency by installing the package's full source code (like a software library).
- **Projects** &mdash; A _new_ way to take a dependency on another project. Using a metadata service that runs behind the scenes, dbt Cloud resolves references on-the-fly to public models defined in other projects. You don't need to parse or run those upstream models yourself. Instead, you treat your dependency on those models as an API that returns a dataset. The maintainer of the public model is responsible for guaranteeing its quality and stability.

import UseCaseInfo from '/snippets/_packages_or_dependencies.md';

<UseCaseInfo/>

Refer to the [FAQs](#faqs) for more info.

Starting in dbt v1.6 or higher, `packages.yml` has been renamed to `dependencies.yml`. However, if you need use Jinja within your packages config, such as an environment variable for your private package, you need to keep using `packages.yml` for your packages for now. Refer to the [FAQs](#faqs) for more info.

## Prerequisites

Expand All @@ -33,22 +37,6 @@ In order to add project dependencies and resolve cross-project `ref`, you must:
- Have a successful run of the upstream ("producer") project
- Have a multi-tenant or single-tenant [dbt Cloud Enterprise](https://www.getdbt.com/pricing) account (Azure ST is not supported but coming soon)

<!-- commenting out in case we can repurpose content
### About dependencies.yml

There are some important differences between using a `dependencies.yml` compared to a `packages.yml` file:

- `dependencies.yml`
- Primarily designed for dbt Mesh and cross-project reference workflow.
- Supports both Projects and non-private dbt packages (private packages aren't supported yet).
- Helps maintain your project's organization by allowing you to specify hub packages like `dbt_utils`, reducing the need for multiple YAML files.
- Does not support conditional configuration using Jinja-in-yaml

- `packages.yml`
- Does not contribute to the dbt Mesh workflow.
- Serves as a list of dbt Packages (such as dbt projects) that you want to download into your root or parent dbt project.
- Can only include packages, including private packages (doesn't support Projects)
-->
## Example

As an example, let's say you work on the Marketing team at the Jaffle Shop. The name of your team's project is `jaffle_marketing`:
Expand Down
32 changes: 32 additions & 0 deletions website/snippets/_packages_or_dependencies.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@

## Use cases

Starting from dbt v1.6, `dependencies.yml` has replaced `packages.yml`. The `dependencies.yml` file can now contain both types of dependencies: "package" and "project" dependencies.
- "Package" dependencies lets you add source code from someone else's dbt project into your own, like a library.
- "Project" dependencies provide a different way to build on top of someone else's work in dbt. Refer to [Project dependencies](/docs/collaborate/govern/project-dependencies) for more info.
- You can rename your existing `packages.yml` to `dependencies.yml` unless you need to use Jinja within your packages specification. This is necessary if you want to add an environment variable with a Git token in a private Git package specification.

There are some important differences between using a `dependencies.yml` compared to a `packages.yml` file:
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

<Tabs>
<TabItem value="dependencies" label="When to use dependencies.yml">

`dependencies.yml` is designed for the [dbt Mesh](/guides/best-practices/how-we-mesh/mesh-1-intro) and cross-project reference workflow. Consider using it in the following scenarios:
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

- Use `dependencies.yml` when you need to set up cross-project references between different dbt projects, especially in a dbt Mesh setup.
- Use `dependencies.yml` when you want to include both projects and non-private dbt packages in your project's dependencies.
- Private packages are not supported in `dependencies.yml` because it intentionally doesn't support Jinja rendering or conditional configuration. This is to maintain static and predictable configuration, and ensures compatibility with other services, like dbt Cloud.
- Use `dependencies.yml` for organization and maintainability. It can help maintain your project's organization by allowing you to specify hub packages like `dbt_utils`. This reduces the need for multiple YAML files to manage dependencies.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

</TabItem>

<TabItem value="packages" label="When to use packages.yml">

Packages allows you to add source code from someone else's dbt project into your own, like a library. Consider using it in the following scenarios:

- Use `packages.yml` when you want to download dbt packages, such as dbt projects, into your root or parent dbt project. Something to note is that it doesn't contribute to the dbt Mesh workflow.
- Use `packages.yml` to include packages, including private packages, in your project's dependencies. If you have private packages that you need to reference, `packages.yml` is the way to go.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
- `packages.yml` supports Jinja rendering for historical reasons, allowing dynamic configurations. This can be useful if you need to insert values, like a Git token from an environment variable, into your package specifications.

</TabItem>
</Tabs>