diff --git a/website/docs/reference/about-resources.md b/website/docs/reference/about-resources.md new file mode 100644 index 00000000000..8b7a020ccfa --- /dev/null +++ b/website/docs/reference/about-resources.md @@ -0,0 +1,64 @@ +--- +title: "About resource types" +id: "about-resources" +description: "Learn about the different resource types in dbt, their supported properties and configurations, and how to use them." +sidebar_label: "About resource types" +--- + +dbt supports different types of resource types. Each one has a set of supported [properties and configurations](/reference/configs-and-properties) within a dbt project, which are key to how they function and integrate within your data project. + +## Resource types + +| Resource type | Executable | Refable | Versioned | Description | +|---------------|------------|---------|-----------| ----------- | +| `analysis` | ✅ | ❌ | ❌ | Similar to models, but usually used for exploratory data analysis and not directly integrated into the dbt DAG (Directed Acyclic Graph). | +| `doc` | ✅ | ❌ | ❌ | Used for documenting the project, including descriptions of models, columns, and more. | +| `exposure` | ❌ | ❌ | ❌ | Document downstream use of dbt models in dashboards, reports, or analysis. | +| `fixture` | ❌ | ❌ | ❌ | +| `group` | ❌ | ❌ | ❌ | Used for grouping and organizing other dbt resources, potentially for managing permissions or categorization. | +| `macro` | ✅ | ❌ | ❌ | Reusable code snippets in Jinja, allowing for custom logic and SQL reuse. | +| `metric` | ❌ | ❌ | ❌ | Define business metrics using a standardized syntax, which can be used across models and analyses. | +| `model` | ✅ | ✅ | ✅ | Central to dbt projects, models are SQL files that define transformations and datasets. These are typically the primary analytical outputs. | +| `operation` | ✅ | ❌ | ❌ | Custom operations that can be run from the dbt command line, often for administrative or maintenance tasks. | +| `rpc` | ✅ | ❌ | ❌ | Deprecated in dbt v1.6. Run a`rpc` to execute dbt commands over a network. Deprecated in dbt v1.6. | +| `saved_query` | ❌ | ❌ | ❌ | Represent saved SQL queries, potentially for reuse or tracking purposes. | +| `seed` | ✅ | ✅ | ❌ | CSV files that are loaded into the database as static tables, useful for small reference data. | +| `semantic_model` | ❌ | ❌ | ❌ | A higher-level abstraction of data models, focusing on the semantic layer or business logic representation. | +| `snapshot` | ✅ | ✅ | ❌ | Capture data at a specific point in time to track changes, useful for historical reporting and auditing. | +| `source` | ❌ | ❌ | ❌ | Represents raw data sources, helping to define and document the initial state of data that dbt interacts with. | +| `sql_operation` | ✅ | ❌ | ❌ | Custom SQL operations that can be executed within dbt workflows. | +| `test` | ✅ | ❌ | ❌ | Define tests to ensure data quality and integrity, such as uniqueness, not-null constraints, and custom data validation. | +| `unit_test` | ❌ | ❌ | ❌ | Define unit tests to test your projects during development, before pushing to production. | + +## Supported properties and configs table + +Resources, such as models, snapshots, seeds, tests, and so on, are the backbone of your data project. Each resource can have defined properties and configurations, which are key to how they function and integrate within your project and pipeline. + +- Properties — Descriptive attributes that provide context and validation rules for your data resources. + - There are special properties, like `descriptions` or `tests`, that are solely defined in `YAML` files alongside resources. They cannot be set in config blocks or the `dbt_project.yml` file since they offer static, context-specific details rather than hierarchical configurations. +- Configurations — Instruct dbt on how to build and maintain them efficiently in your data warehouse environment. + +Refer to [About configs and properties](/reference/configs-and-properties) for more details. + +Properties or configurations support different resource types and are applied in different ways. The following table describes each resource type, its supported properties and configurations, and how they are applied. + +| Resource type | Supported properties | Supported configurations | Config inheritance | +| --------------| -------------------- | ----------------------- | ------------------ | +| Exposures | description, columns, tests, docs, macros | tags, materialized, and so on. | `properties.yml` → `.yml file` → `dbt_project.yml` | +| Group | | +| Macro | | +| Models | columns, tests, docs, macros | tags, materialized, persist_docs, tags, and so on. | `properties.yml` → `.yml file` → `dbt_project.yml` | +| Saved Query | | +| Semantic Model | | +| Seeds | tests, macros | tags, persist_docs, and so on. | In-file `config()` block → `.yml file` → `dbt_project.yml` | +| Snapshots | tests, macros | tags, materialized, and so on.| In-file `config()` block → `.yml file` → `dbt_project.yml` | +| Sources | description, columns, tests, docs, macros | tags, persist_docs, and so on. | `properties.yml` → `.yml file` → `dbt_project.yml` | +| Tests | description, tags, macros | | In-file `config()` block → `.yml file` → `dbt_project.yml` | +| | | | | | +| Configurations | In-file `config()` block → `.yml file` → `dbt_project.yml` | +| | | | | | +| Special Properties | description, tests, docs, columns, quote, source properties, exposure properties, macro properties | | `properties.yml` | + +## Related docs +- [About resource paths](/reference/resource-configs/resource-path) +- [About configs and properties](/reference/configs-and-properties) diff --git a/website/docs/reference/configs-and-properties.md b/website/docs/reference/configs-and-properties.md index 58ab43af136..6f4d1289118 100644 --- a/website/docs/reference/configs-and-properties.md +++ b/website/docs/reference/configs-and-properties.md @@ -84,6 +84,7 @@ These properties are: - [`macro` properties](/reference/macro-properties) (e.g. `arguments`) ## Example + Here's an example that defines both `sources` and `models` for a project: @@ -151,6 +152,8 @@ models: + + ## FAQs @@ -170,7 +174,6 @@ You can find an exhaustive list of each supported property and config, broken do ## Troubleshooting common errors - ### Invalid test config given in [model name] This error occurs when your `.yml` file does not conform to the structure expected by dbt. A full error message might look like: diff --git a/website/scripts/update_node_resource_table.py b/website/scripts/update_node_resource_table.py new file mode 100755 index 00000000000..15878509f07 --- /dev/null +++ b/website/scripts/update_node_resource_table.py @@ -0,0 +1,85 @@ +import requests +import re + +# URLs of the source files +node_types_url = "https://raw.githubusercontent.com/dbt-labs/dbt-core/main/core/dbt/node_types.py" +resource_types_url = "https://raw.githubusercontent.com/dbt-labs/dbt-core/main/core/dbt/artifacts/resources/types.py" + +# Function to fetch and parse the Python files +def fetch_and_parse(url): + response = requests.get(url) + response.raise_for_status() + return response.text + +# Fetch the content of the Python files +node_types_content = fetch_and_parse(node_types_url) +resource_types_content = fetch_and_parse(resource_types_url) + +# Extract resource types from the node_types.py file +node_types_lines = node_types_content.split('\n') +node_types = {} +for line in node_types_lines: + match = re.search(r'(\w+)\s*=\s*"(\w+)"', line) + if match: + node_types[match.group(2)] = {'executable': False, 'refable': False, 'versioned': False} + +# Extract resource types from the resource_types.py file +resource_types_lines = resource_types_content.split('\n') +resource_types = {} +for line in resource_types_lines: + match = re.search(r'(\w+)\s*=\s*"(\w+)"', line) + if match: + resource_type = match.group(2) + resource_types[resource_type] = {'executable': False, 'refable': False, 'versioned': False} + +# Define the properties for each resource type +resource_properties = { + "model": {"executable": True, "refable": True, "versioned": True}, + "analysis": {"executable": True, "refable": False, "versioned": False}, + "test": {"executable": True, "refable": False, "versioned": False}, + "snapshot": {"executable": True, "refable": True, "versioned": False}, + "operation": {"executable": True, "refable": False, "versioned": False}, + "seed": {"executable": True, "refable": True, "versioned": False}, + "rpc": {"executable": True, "refable": False, "versioned": False}, + "sql_operation": {"executable": True, "refable": False, "versioned": False}, + "doc": {"executable": True, "refable": False, "versioned": False}, + "source": {"executable": False, "refable": False, "versioned": False}, + "macro": {"executable": True, "refable": False, "versioned": False}, + "exposure": {"executable": False, "refable": False, "versioned": False}, + "metric": {"executable": False, "refable": False, "versioned": False}, + "group": {"executable": False, "refable": False, "versioned": False}, + "saved_query": {"executable": False, "refable": False, "versioned": False}, + "semantic_model": {"executable": False, "refable": False, "versioned": False}, + "unit_test": {"executable": False, "refable": False, "versioned": False}, + "fixture": {"executable": False, "refable": False, "versioned": False}, +} + +# Update the properties of resource types based on the definitions +for resource_type in resource_types.keys(): + if resource_type in resource_properties: + resource_types[resource_type] = resource_properties[resource_type] + +# Sort the resource types alphabetically +sorted_resource_types = dict(sorted(resource_types.items())) + +# Generate the Markdown table +markdown_table = "| Resource type | Executable | Refable | Versioned |\n" +markdown_table += "|---------------|------------|---------|-----------|\n" +for resource_type, properties in sorted_resource_types.items(): + markdown_table += f"| {resource_type} | {'✅' if properties['executable'] else '❌'} | {'✅' if properties['refable'] else '❌'} | {'✅' if properties['versioned'] else '❌'} |\n" + +# Define the file path +file_path = "../docs/reference/about-resources.md" + +# Read the existing content of the file +with open(file_path, "r") as file: + content = file.read() + +# Replace the content under the "## Resource types" header +new_content = re.sub(r'(## Resource types\s*\n)(.*?)(\n## )', rf'\1\n{markdown_table}\3', content, flags=re.DOTALL) + +# Write the updated content back to the file +with open(file_path, "w") as file: + file.write(new_content) + +print(f"The Markdown table has been added to {file_path} under the '## Resource types' header.") diff --git a/website/sidebars.js b/website/sidebars.js index a5992389ea1..c44202c1829 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -786,8 +786,9 @@ const sidebarSettings = { type: "category", label: "Resource configs and properties", items: [ - "reference/configs-and-properties", + "reference/about-resources", "reference/resource-configs/resource-path", + "reference/configs-and-properties", { type: "category", label: "General properties",