diff --git a/website/blog/maching-learning-dbt-baton-pass.md b/website/blog/maching-learning-dbt-baton-pass.md index 2c38cfd8983..c1cb05fceab 100644 --- a/website/blog/maching-learning-dbt-baton-pass.md +++ b/website/blog/maching-learning-dbt-baton-pass.md @@ -145,20 +145,6 @@ This wouldn’t solve for the ML engineer and her desire to inject custom ML mod It may be worth having python scripts live side by side dbt jobs and configurations. I can get better lineage and have one less tool to context switch to. -#### fal - -- [fal](https://github.com/fal-ai/fal): Makes dbt and python interoperable. Read in a dbt model as a pandas dataframe using a [ref statement](https://github.com/fal-ai/fal/blob/b20874ab957f8eb0f65c56d82e6bb85c717de4c6/examples/write_jupyter_notebook.md#example-9-use-dbt-from-a-jupyter-notebook)! - -#### How would this change my story? - -**A ref statement would mean the same thing to both my ML engineer and me.** - -We would work in the same dbt project for the entire workflow, not just part of it. We would align python scripts to dbt configurations for better lineage (see below). - -![Screenshot of fal installation instructions](/img/blog/2022-02-18-machine-learning-dbt-baton-pass/fal-install-1.png) - -![Second screenshot of fal installation instructions](/img/blog/2022-02-18-machine-learning-dbt-baton-pass/fal-install-2.png) - #### What are the tradeoffs of this tool path? When things would go wrong, it’d still be a messy jumble to figure out how SQL changes inform python changes and vice versa. And I would need to care about which infrastructure my python code is running on. But my gut tells me the tradeoff would be worth it because there’d be less notebooks to schedule, and it’d be easier to align machine learning logic to dbt logic. diff --git a/website/dbt-versions.js b/website/dbt-versions.js index 5ad3a3048c5..7430450da50 100644 --- a/website/dbt-versions.js +++ b/website/dbt-versions.js @@ -169,4 +169,4 @@ exports.versionedCategories = [ "category": "Build your metrics", "firstVersion": "1.6", } -] +] \ No newline at end of file diff --git a/website/docs/docs/community-adapters.md b/website/docs/docs/community-adapters.md index 22dd2404765..3af4e15b32b 100644 --- a/website/docs/docs/community-adapters.md +++ b/website/docs/docs/community-adapters.md @@ -9,7 +9,6 @@ Community adapters are adapter plugins contributed and maintained by members of | ------------------------------------------ | -------------------------------- | ------------------------------------- | | [Clickhouse](/docs/core/connect-data-platform/clickhouse-setup) | [Databend Cloud](/docs/core/connect-data-platform/databend-setup) | [Doris & SelectDB](/docs/core/connect-data-platform/doris-setup) | | [DuckDB](/docs/core/connect-data-platform/duckdb-setup) | [Exasol Analytics](/docs/core/connect-data-platform/exasol-setup) | [Extrica](/docs/core/connect-data-platform/extrica-setup) | -| [fal - Python models](/docs/core/connect-data-platform/fal-setup) | [Firebolt](/docs/core/connect-data-platform/firebolt-setup) | [Greenplum](/docs/core/connect-data-platform/greenplum-setup) | | [Hive](/docs/core/connect-data-platform/hive-setup) | [IBM DB2](/docs/core/connect-data-platform/ibmdb2-setup) | [Impala](/docs/core/connect-data-platform/impala-setup) | | [Infer](/docs/core/connect-data-platform/infer-setup) | [iomete](/docs/core/connect-data-platform/iomete-setup) | [MindsDB](/docs/core/connect-data-platform/mindsdb-setup) | | [MySQL](/docs/core/connect-data-platform/mysql-setup) | [RisingWave](/docs/core/connect-data-platform/risingwave-setup) | [Rockset](/docs/core/connect-data-platform/rockset-setup) | diff --git a/website/docs/docs/core/connect-data-platform/fal-setup.md b/website/docs/docs/core/connect-data-platform/fal-setup.md deleted file mode 100644 index 5dfb5967d3a..00000000000 --- a/website/docs/docs/core/connect-data-platform/fal-setup.md +++ /dev/null @@ -1,51 +0,0 @@ ---- -title: "fal setup (Python models)" -description: "Read this guide to learn about the fal warehouse setup in dbt." -meta: - maintained_by: fal.ai - authors: 'Features & Labels' - github_repo: 'fal-ai/fal' - pypi_package: 'dbt-fal' - min_core_version: 'v1.3.0' - max_core_version: 'v1.5.0' - cloud_support: Not Supported - min_supported_version: 'n/a' - slack_channel_name: '#tools-fal' - slack_channel_link: 'https://getdbt.slack.com/archives/C02V8QW3Q4Q' - platform_name: 'fal' - config_page: '/reference/resource-configs/fal-configs' ---- - -:::info Adapter no longer maintained -The [`dbt-fal` adapter](https://github.com/fal-ai/dbt-fal) is no longer actively maintained. This means although the adapter is still operational, there is no further development or bug fixes planned and it may not be compatible with future versions of dbt. `dbt-fal` was test until dbt v1.5. - -Documentation for `dbt-fal` are kept for reference purposes only and will eventually be removed from the site in the future. -::: - -import SetUpPages from '/snippets/_setup-pages-intro.md'; - - - -## Setting up fal with other adapter - -[fal](http://github.com/fal-ai/fal) offers a Python runtime independent from what database you are using and integrates seamlessly with dbt. It works by downloading the data as a Pandas DataFrame, transforming it in a local Python runtime and uploading it to the database. The only configuration change you need to do is adding it to the `profiles.yml` and setting the `db_profile` property as the database profile you are already using. - -It will run all the SQL dbt models with the main adapter you specified in your `profiles.yml` and all the Python models are executed by the fal adapter. - -Example: - - - -```yaml -jaffle_shop: - target: dev_with_fal - outputs: - dev_with_fal: - type: fal - db_profile: dev_pg # This points to your main adapter - dev_pg: - type: postgres - ... -``` - - diff --git a/website/docs/reference/resource-configs/fal-configs.md b/website/docs/reference/resource-configs/fal-configs.md deleted file mode 100644 index 85befeccdb4..00000000000 --- a/website/docs/reference/resource-configs/fal-configs.md +++ /dev/null @@ -1,84 +0,0 @@ ---- -title: "fal configurations" -id: "fal-configs" ---- - -:::info Adapter no longer maintained -The [`dbt-fal` adapter](https://github.com/fal-ai/dbt-fal) is no longer actively maintained. This means although the adapter is still operational, there is no further development or bug fixes planned and it may not be compatible with future versions of dbt. `dbt-fal` was test until dbt v1.5. - -Documentation for `dbt-fal` are kept for reference purposes only and will eventually be removed from the site in the future. -::: - -## Setting the `db_profile` - -The fal profile configuration needs the `db_profile` property set to the profile configuring your database for SQL models. - -fal will wrap around adapter and just handle Python models while letting all the SQL -needs to the underlying database adapter. - -fal will inherit the `threads` configuration from the `db_profile` unless explicitly specified. - -Example: - - - -```yaml -jaffle_shop: - target: dev_with_fal - outputs: - dev_with_fal: - type: fal - db_profile: dev_pg # This points to your main adapter - dev_pg: - type: postgres - ... -``` - - - -## Using `fal_environment` model configuration - -By creating a `fal_project.yml` in the same location as your `dbt_project.yml` and adding environment definitions in there: - - - -```yaml -environments: - - name: clustering - type: conda - packages: - - kmodes==0.12.2 - - - name: predict - type: venv - requirements: - - prophet -``` - - - -You can now reference any of these environments in your dbt Python models: - - - -```py -def model(dbt, fal): - dbt.config({ - "fal_environment": "clustering" - }) - - import pandas as pd - # kmodes is available because of the `fal_environment` being used - from kmodes.kmodes import KModes - - df: pd.DataFrame = dbt.ref("order_detailed") - df_train = df[["size", "is_vegan", "is_vegetarian", "is_keto", "shape"]] - - km_2 = KModes(n_clusters=3, init="Huang") - km_2.fit_predict(df_train) - df["cluster_label"] = km_2.labels_ - - return df -``` - - diff --git a/website/sidebars.js b/website/sidebars.js index 7bb8a43d85b..589dbd4a27d 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -225,7 +225,6 @@ const sidebarSettings = { "docs/core/connect-data-platform/duckdb-setup", "docs/core/connect-data-platform/exasol-setup", "docs/core/connect-data-platform/extrica-setup", - "docs/core/connect-data-platform/fal-setup", "docs/core/connect-data-platform/firebolt-setup", "docs/core/connect-data-platform/greenplum-setup", "docs/core/connect-data-platform/ibmdb2-setup", @@ -834,7 +833,6 @@ const sidebarSettings = { "reference/resource-configs/impala-configs", "reference/resource-configs/clickhouse-configs", "reference/resource-configs/doris-configs", - "reference/resource-configs/fal-configs", "reference/resource-configs/firebolt-configs", "reference/resource-configs/greenplum-configs", "reference/resource-configs/infer-configs", diff --git a/website/vercel.json b/website/vercel.json index 18d70f9fee0..7cf5bb542dd 100644 --- a/website/vercel.json +++ b/website/vercel.json @@ -2,6 +2,16 @@ "cleanUrls": true, "trailingSlash": false, "redirects": [ + { + "source": "/docs/core/connect-data-platform/fal-setup", + "destination": "https://github.com/fal-ai/dbt-fal", + "permanent": true + }, + { + "source": "/reference/warehouse-setups/fal-setup", + "destination": "https://github.com/fal-ai/dbt-fal", + "permanent": true + }, { "source": "docs/cloud/secure/environment-permissions", "destination": "docs/cloud/manage-access/environment-permissions",