Skip to content

Commit

Permalink
Merge branch 'current' into semantic-label
Browse files Browse the repository at this point in the history
  • Loading branch information
matthewshaver authored Oct 11, 2023
2 parents 7ab0602 + 3a9d15e commit 6f30e11
Show file tree
Hide file tree
Showing 20 changed files with 139 additions and 62 deletions.
23 changes: 23 additions & 0 deletions website/docs/docs/build/semantic-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,29 @@ semantic_models:
type: categorical
```
<VersionBlock firstVersion="1.7">
Semantic models support configs in either the schema file or at the project level.
Semantic model config in `models/semantic.yml`:
```yml
semantic_models:
- name: orders
config:
enabled: true | false
group: some_group
```

Semantic model config in `dbt_project.yml`:
```yml
semantic_models:
my_project_name:
+enabled: true | false
+group: some_group
```

</VersionBlock>

### Name

Define the name of the semantic model. You must define a unique name for the semantic model. The semantic graph will use this name to identify the model, and you can update it at any time. Avoid using double underscores (__) in the name as they're not supported.
Expand Down
12 changes: 11 additions & 1 deletion website/docs/docs/cloud/dbt-cloud-ide/lint-format.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,11 @@ With the dbt Cloud IDE, you can seamlessly use [SQLFluff](https://sqlfluff.com/)
- Works with Jinja and SQL,
- Comes with built-in [linting rules](https://docs.sqlfluff.com/en/stable/rules.html). You can also [customize](#customize-linting) your own linting rules.
- Empowers you to [enable linting](#enable-linting) with options like **Lint** (displays linting errors and recommends actions) or **Fix** (auto-fixes errors in the IDE).
- Displays a **Code Quality** tab to view code errors, and provides code quality visibility and management.
- Displays a **Code Quality** tab to view code errors, and provides code quality visibility and management.

:::info Ephemeral models not supported
Linting doesn't support ephemeral models in dbt v1.5 and lower. Refer to the [FAQs](#faqs) for more info.
:::

### Enable linting

Expand Down Expand Up @@ -223,6 +227,12 @@ Currently, running SQLFluff commands from the terminal isn't supported.
Make sure you're on a development branch. Formatting or Linting isn't available on "main" or "read-only" branches.
</details>

<details>
<summary>Why is there inconsistent SQLFluff behavior when running outside the dbt Cloud IDE (such as a GitHub Action)?</summary>
&mdash; Double-check your SQLFluff version matches the one in dbt Cloud IDE (found in the <b>Code Quality</b> tab after a lint operation). <br /><br />
&mdash; If your lint operation passes despite clear rule violations, confirm you're not linting models with ephemeral models. Linting doesn't support ephemeral models in dbt v1.5 and lower.
</details>

## Related docs

- [User interface](/docs/cloud/dbt-cloud-ide/ide-user-interface)
Expand Down
8 changes: 8 additions & 0 deletions website/docs/docs/collaborate/govern/project-dependencies.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,3 +100,11 @@ There are a few cases where installing another internal project as a package can
- Coordinated changes &mdash; In development, if you wanted to test the effects of a change to a public model in an upstream project (`jaffle_finance.monthly_revenue`) on a downstream model (`jaffle_marketing.roi_by_channel`) _before_ introducing changes to a staging or production environment, you can install the `jaffle_finance` package as a package within `jaffle_marketing`. The installation can point to a specific git branch, however, if you find yourself frequently needing to perform end-to-end testing across both projects, we recommend you re-examine if this represents a stable interface boundary.

These are the exceptions, rather than the rule. Installing another team's project as a package adds complexity, latency, and risk of unnecessary costs. By defining clear interface boundaries across teams, by serving one team's public models as "APIs" to another, and by enabling practitioners to develop with a more narrowly-defined scope, we can enable more people to contribute, with more confidence, while requiring less context upfront.

## FAQs

<details>
<summary>Can I define private packages in the <code>dependencies.yml</code> file?</summary>

If you're using private packages with the [git token method](/docs/build/packages#git-token-method), you must define them in the `packages.yml` file instead of the `dependencies.yml` file. This is because conditional rendering (like Jinja-in-yaml) is not supported.
</details>
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ The following profile fields are optional to set up. They let you configure your
| Profile field | Example | Description |
| ----------------------------- | -------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| `threads` | `8` | How many threads dbt should use (default is `1`) |
| `roles` | `system: analyst` | Catalog roles can be set under the optional `roles` parameter using following format: `catalog: role`. |
| `roles` | `system: analyst` | Catalog roles can be set under the optional `roles` parameter using the following format: `catalog: role`. |
| `session_properties` | `query_max_run_time: 4h` | Sets Trino session properties used in the connection. Execute `SHOW SESSION` to see available options |
| `prepared_statements_enabled` | `true` or `false` | Enable usage of Trino prepared statements (used in `dbt seed` commands) (default: `true`) |
| `retries` | `10` | Configure how many times all database operation is retried when connection issues arise (default: `3`) |
Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/dbt-cloud-apis/user-tokens.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ permissions of the user the that they were created for.
You can find your User API token in the Profile page under the `API Access`
label.

<Lightbox src="/img/api-access-profile.png" title="Finding your API token in your dbt Cloud Profile" />
<Lightbox src="/img/api-access-profile.jpg" title="Finding your API token in your dbt Cloud Profile" />

## FAQs

Expand Down
7 changes: 7 additions & 0 deletions website/docs/docs/use-dbt-semantic-layer/gsheets.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,3 +51,10 @@ To use the filter functionality, choose the dimension you want to filter by and
- If it's a categorical dimension, type in the dimension value you want to filter by (no quotes needed) and press enter.
- Continue adding additional filters as needed with AND and OR. If it's a time dimension, choose the operator and select from the calendar.



**Limited Use Policy Disclosure**

The dbt Semantic Layer for Sheet's use and transfer to any other app of information received from Google APIs will adhere to [Google API Services User Data Policy](https://developers.google.com/terms/api-services-user-data-policy), including the Limited Use requirements.


11 changes: 0 additions & 11 deletions website/docs/faqs/Models/reference-models-in-another-project.md

This file was deleted.

12 changes: 12 additions & 0 deletions website/docs/reference/commands/cmd-docs.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,18 @@ The command is responsible for generating your project's documentation website b
dbt docs generate
```

<VersionBlock firstVersion="1.7">

Use the `--select` argument to limit the nodes included within `catalog.json`. When this flag is provided, step (3) will be restricted to the selected nodes. All other nodes will be excluded. Step (2) is unaffected.

**Example**:
```shell
dbt docs generate --select +orders
```

</VersionBlock>


Use the `--no-compile` argument to skip re-compilation. When this flag is provided, `dbt docs generate` will skip step (2) described above.

**Example**:
Expand Down
11 changes: 11 additions & 0 deletions website/docs/reference/commands/init.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,21 @@ Then, it will:
- Create a new folder with your project name and sample files, enough to get you started with dbt
- Create a connection profile on your local machine. The default location is `~/.dbt/profiles.yml`. Read more in [configuring your profile](/docs/core/connect-data-platform/connection-profiles).

<VersionBlock firstVersion="1.7">

When using `dbt init` to initialize your project, include the `--profile` flag to specify an existing `profiles.yml` as the `profile:` key to use instead of creating a new one. For example, `dbt init --profile`.



If the profile does not exist in `profiles.yml` or the command is run inside an existing project, the command raises an error.

</VersionBlock>

## Existing project

If you've just cloned or downloaded an existing dbt project, `dbt init` can still help you set up your connection profile so that you can start working quickly. It will prompt you for connection information, as above, and add a profile (using the `profile` name from the project) to your local `profiles.yml`, or create the file if it doesn't already exist.


## profile_template.yml

`dbt init` knows how to prompt for connection information by looking for a file named `profile_template.yml`. It will look for this file in two places:
Expand Down
84 changes: 45 additions & 39 deletions website/docs/reference/node-selection/test-selection-examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,14 @@ Run generic tests only:


```bash
$ dbt test --select test_type:generic
dbt test --select test_type:generic
```

Run singular tests only:


```bash
$ dbt test --select test_type:singular
dbt test --select test_type:singular
```

In both cases, `test_type` checks a property of the test itself. These are forms of "direct" test selection.
Expand Down Expand Up @@ -87,8 +87,8 @@ By default, a test will run when ANY parent is selected; we call this "eager" in
In this mode, any test that depends on unbuilt resources will raise an error.

```shell
$ dbt test --select orders
$ dbt build --select orders
dbt test --select orders
dbt build --select orders
```

</TabItem>
Expand All @@ -102,8 +102,8 @@ It will only include tests whose references are each within the selected nodes.
Put another way, it will prevent tests from running if one or more of its parents is unselected.

```shell
$ dbt test --select orders --indirect-selection=cautious
$ dbt build --select orders --indirect-selection=cautious
dbt test --select orders --indirect-selection=cautious
dbt build --select orders --indirect-selection=cautious
```

</TabItem>
Expand All @@ -122,8 +122,8 @@ By default, a test will run when ANY parent is selected; we call this "eager" in
In this mode, any test that depends on unbuilt resources will raise an error.

```shell
$ dbt test --select orders
$ dbt build --select orders
dbt test --select orders
dbt build --select orders
```

</TabItem>
Expand All @@ -137,8 +137,8 @@ It will only include tests whose references are each within the selected nodes.
Put another way, it will prevent tests from running if one or more of its parents is unselected.

```shell
$ dbt test --select orders --indirect-selection=cautious
$ dbt build --select orders --indirect-selection=cautious
dbt test --select orders --indirect-selection=cautious
dbt build --select orders --indirect-selection=cautious
```

</TabItem>
Expand All @@ -152,8 +152,8 @@ It will only include tests whose references are each within the selected nodes (
This is useful in the same scenarios as "cautious", but also includes when a test depends on a model **and** a direct ancestor of that model (like confirming an aggregation has the same totals as its input).

```shell
$ dbt test --select orders --indirect-selection=buildable
$ dbt build --select orders --indirect-selection=buildable
dbt test --select orders --indirect-selection=buildable
dbt build --select orders --indirect-selection=buildable
```

</TabItem>
Expand All @@ -172,8 +172,8 @@ By default, a test will run when ANY parent is selected; we call this "eager" in
In this mode, any test that depends on unbuilt resources will raise an error.

```shell
$ dbt test --select orders
$ dbt build --select orders
dbt test --select orders
dbt build --select orders
```

</TabItem>
Expand All @@ -187,8 +187,8 @@ It will only include tests whose references are each within the selected nodes.
Put another way, it will prevent tests from running if one or more of its parents is unselected.

```shell
$ dbt test --select orders --indirect-selection=cautious
$ dbt build --select orders --indirect-selection=cautious
dbt test --select orders --indirect-selection=cautious
dbt build --select orders --indirect-selection=cautious
```

</TabItem>
Expand All @@ -202,8 +202,8 @@ It will only include tests whose references are each within the selected nodes (
This is useful in the same scenarios as "cautious", but also includes when a test depends on a model **and** a direct ancestor of that model (like confirming an aggregation has the same totals as its input).

```shell
$ dbt test --select orders --indirect-selection=buildable
$ dbt build --select orders --indirect-selection=buildable
dbt test --select orders --indirect-selection=buildable
dbt build --select orders --indirect-selection=buildable
```

</TabItem>
Expand All @@ -213,8 +213,8 @@ $ dbt build --select orders --indirect-selection=buildable
This mode will only include tests whose references are each within the selected nodes and will ignore all tests from attached nodes.

```shell
$ dbt test --select orders --indirect-selection=empty
$ dbt build --select orders --indirect-selection=empty
dbt test --select orders --indirect-selection=empty
dbt build --select orders --indirect-selection=empty
```

</TabItem>
Expand All @@ -234,22 +234,25 @@ The following examples should feel somewhat familiar if you're used to executing

```bash
# Run tests on a model (indirect selection)
$ dbt test --select customers
dbt test --select customers

# Run tests on two or more specific models (indirect selection)
dbt test --select customers orders

# Run tests on all models in the models/staging/jaffle_shop directory (indirect selection)
$ dbt test --select staging.jaffle_shop
dbt test --select staging.jaffle_shop

# Run tests downstream of a model (note this will select those tests directly!)
$ dbt test --select stg_customers+
dbt test --select stg_customers+

# Run tests upstream of a model (indirect selection)
$ dbt test --select +stg_customers
dbt test --select +stg_customers

# Run tests on all models with a particular tag (direct + indirect)
$ dbt test --select tag:my_model_tag
dbt test --select tag:my_model_tag

# Run tests on all models with a particular materialization (indirect selection)
$ dbt test --select config.materialized:table
dbt test --select config.materialized:table

```

Expand All @@ -258,16 +261,19 @@ The following examples should feel somewhat familiar if you're used to executing

```bash
# tests on all sources
$ dbt test --select source:*
dbt test --select source:*

# tests on one source
$ dbt test --select source:jaffle_shop
dbt test --select source:jaffle_shop

# tests on two or more specific sources
dbt test --select source:jaffle_shop source:raffle_bakery

# tests on one source table
$ dbt test --select source:jaffle_shop.customers
dbt test --select source:jaffle_shop.customers

# tests on everything _except_ sources
$ dbt test --exclude source:*
dbt test --exclude source:*
```

### More complex selection
Expand All @@ -276,10 +282,10 @@ Through the combination of direct and indirect selection, there are many ways to


```bash
$ dbt test --select assert_total_payment_amount_is_positive # directly select the test by name
$ dbt test --select payments,test_type:singular # indirect selection, v1.2
$ dbt test --select payments,test_type:data # indirect selection, v0.18.0
$ dbt test --select payments --data # indirect selection, earlier versions
dbt test --select assert_total_payment_amount_is_positive # directly select the test by name
dbt test --select payments,test_type:singular # indirect selection, v1.2
dbt test --select payments,test_type:data # indirect selection, v0.18.0
dbt test --select payments --data # indirect selection, earlier versions
```


Expand All @@ -288,13 +294,13 @@ Through the combination of direct and indirect selection, there are many ways to

```bash
# Run tests on all models with a particular materialization
$ dbt test --select config.materialized:table
dbt test --select config.materialized:table

# Run tests on all seeds, which use the 'seed' materialization
$ dbt test --select config.materialized:seed
dbt test --select config.materialized:seed

# Run tests on all snapshots, which use the 'snapshot' materialization
$ dbt test --select config.materialized:snapshot
dbt test --select config.materialized:snapshot
```

Note that this functionality may change in future versions of dbt.
Expand Down Expand Up @@ -322,7 +328,7 @@ models:


```bash
$ dbt test --select tag:my_column_tag
dbt test --select tag:my_column_tag
```

Currently, tests "inherit" tags applied to columns, sources, and source tables. They do _not_ inherit tags applied to models, seeds, or snapshots. In all likelihood, those tests would still be selected indirectly, because the tag selects its parent. This is a subtle distinction, and it may change in future versions of dbt.
Expand Down Expand Up @@ -350,5 +356,5 @@ models:


```bash
$ dbt test --select tag:my_test_tag
dbt test --select tag:my_test_tag
```
19 changes: 10 additions & 9 deletions website/snippets/quickstarts/schedule-a-job.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,15 +24,16 @@ Jobs are a set of dbt commands that you want to run on a schedule. For example,

As the `jaffle_shop` business gains more customers, and those customers create more orders, you will see more records added to your source data. Because you materialized the `customers` model as a table, you'll need to periodically rebuild your table to ensure that the data stays up-to-date. This update will happen when you run a job.

1. After creating your deployment environment, you should be directed to the page for new environment. If not, select **Deploy** in the upper left, then click **Jobs**.
2. Click **Create one** and provide a name, for example "Production run", and link to the Environment you just created.
3. Scroll down to "Execution Settings" and select **Generate docs on run**.
4. Under "Commands," add this command as part of your job if you don't see them:
* `dbt build`
5. For this exercise, do _not_ set a schedule for your project to run &mdash; while your organization's project should run regularly, there's no need to run this example project on a schedule. Scheduling a job is sometimes referred to as _deploying a project_.
6. Select **Save**, then click **Run now** to run your job.
7. Click the run and watch its progress under "Run history."
8. Once the run is complete, click **View Documentation** to see the docs for your project.
1. After creating your deployment environment, you should be directed to the page for a new environment. If not, select **Deploy** in the upper left, then click **Jobs**.
2. Click **Create one** and provide a name, for example, "Production run", and link to the Environment you just created.
3. Scroll down to the **Execution Settings** section.
4. Under **Commands**, add this command as part of your job if you don't see it:
* `dbt build`
5. Select the **Generate docs on run** checkbox to automatically [generate updated project docs](/docs/collaborate/build-and-view-your-docs) each time your job runs.
6. For this exercise, do _not_ set a schedule for your project to run &mdash; while your organization's project should run regularly, there's no need to run this example project on a schedule. Scheduling a job is sometimes referred to as _deploying a project_.
7. Select **Save**, then click **Run now** to run your job.
8. Click the run and watch its progress under "Run history."
9. Once the run is complete, click **View Documentation** to see the docs for your project.

:::tip
Congratulations 🎉! You've just deployed your first dbt project!
Expand Down
Binary file modified website/static/img/Filtering.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified website/static/img/Paginate.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added website/static/img/api-access-profile.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed website/static/img/api-access-profile.png
Binary file not shown.
Binary file modified website/static/img/dbt-cloud-project-setup-flow-next.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified website/static/img/delete_projects_from_dbt_cloud_20221023.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified website/static/img/node_color_example.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed website/static/img/sample_email_data.png
Binary file not shown.
Loading

0 comments on commit 6f30e11

Please sign in to comment.