From 817d76e1b2661d05a98162c77e3a673e832aa73d Mon Sep 17 00:00:00 2001 From: nikki everett Date: Sat, 24 Feb 2024 15:43:39 -0600 Subject: [PATCH 1/4] copy edits Signed-off-by: nikki everett --- docs/flyte_agents/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/flyte_agents/index.md b/docs/flyte_agents/index.md index d56c7fed4b..2a819603a4 100644 --- a/docs/flyte_agents/index.md +++ b/docs/flyte_agents/index.md @@ -7,11 +7,11 @@ prev-page-title: Extending Flyte (flyte_agents_guide)= # Flyte agents -Flyte agents are long-running, stateless services that receive execution requests via gRPC and initiate jobs with appropriate external or internal services. They enable two key workflows: asynchronously launching jobs on hosted platforms (e.g. Databbricks or Snowflake) and calling external synchronous services, such as access control, data retrieval, and model inferencing. +Flyte agents are long-running, stateless services that receive execution requests via gRPC and initiate jobs with appropriate external or internal services. They enable two key workflows: asynchronously launching jobs on hosted platforms (e.g. Databricks or Snowflake) and calling external synchronous services, such as access control, data retrieval, and model inferencing. -Each agent service is a Kubernetes deployment that receives gRPC requests from FlytePropeller when users trigger a particular type of task (for example, the BigQuery agent handles BigQuery tasks). The agent service then initiates a job with the appropriate service. Since Agents can be spawned in process, they allow for running all services locally as long as the connection secrets are available. Moreover, Agents use a protobuf interface, thus can be implemented in any language, providing a lot of opportunity for flexibility and reuse of existing libraries, as well as simpler testing. +Each agent service is a Kubernetes deployment that receives gRPC requests from FlytePropeller when users trigger a particular type of task (for example, the BigQuery agent handles BigQuery tasks). The agent service then initiates a job with the appropriate service. Since agents can be spawned in process, they allow for running all services locally as long as the connection secrets are available. Moreover, agents use a protobuf interface, thus can be implemented in any language, enabling flexibility, reuse of existing libraries, and simpler testing. -You can create different agent services that host different agents, e.g., a production and a development agent service. +You can create different agent services that host different agents, e.g., a production and a development agent service: :::{figure} https://i.ibb.co/vXhBDjP/Screen-Shot-2023-05-29-at-2-54-14-PM.png :alt: Agent Service From ef34390cb97515ba076a9d05d716b45e0218d2d8 Mon Sep 17 00:00:00 2001 From: nikki everett Date: Sat, 24 Feb 2024 15:44:24 -0600 Subject: [PATCH 2/4] add anchor link Signed-off-by: nikki everett --- docs/flyte_agents/index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/flyte_agents/index.md b/docs/flyte_agents/index.md index 2a819603a4..293f661be9 100644 --- a/docs/flyte_agents/index.md +++ b/docs/flyte_agents/index.md @@ -18,6 +18,7 @@ You can create different agent services that host different agents, e.g., a prod :class: with-shadow ::: +(using_agents_in_tasks)= ## Using agents in tasks If you need to connect to an external service in your workflow, we recommend using the corresponding agent rather than a web API plugin. Agents are designed to be scalable and can handle large workloads efficiently, and decrease load on FlytePropeller, since they run outside of it. You can also test agents locally without having to change the Flyte backend configuration, streamlining development. From 71bda1558ba50acb8b77951af8204e679da03f3a Mon Sep 17 00:00:00 2001 From: nikki everett Date: Sun, 25 Feb 2024 18:44:17 -0600 Subject: [PATCH 3/4] add note about contribution Signed-off-by: nikki everett --- docs/flyte_agents/developing_agents.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/flyte_agents/developing_agents.md b/docs/flyte_agents/developing_agents.md index bd7d1c7610..688f241852 100644 --- a/docs/flyte_agents/developing_agents.md +++ b/docs/flyte_agents/developing_agents.md @@ -13,6 +13,12 @@ The Flyte agent framework enables rapid agent development, since agents are deco If you need to create a new type of task, we recommend creating a new agent to run it rather than running the task in a pod. After testing the new agent, you can update your FlytePropeller configMap to specify the type of task that the agent should run. +```{note} + +We strongly encourage you to contribute your agent to the Flyte community. To do so, follow the steps in "[Contributing to Flyte](https://docs.flyte.org/en/latest/community/contribute.html)", and reach out to us on [Slack](https://docs.flyte.org/en/latest/community/contribute.html#) if you have any questions. + +``` + There are two types of agents: **async** and **sync**. * **Async agents** enable long-running jobs that execute on an external platform over time. They communicate with external services that have asynchronous APIs that support `create`, `get`, and `delete` operations. The vast majority of agents are async agents. * **Sync agents** enable request/response services that return immediate outputs (e.g. calling an internal API to fetch data or communicating with the OpenAI API). @@ -83,3 +89,9 @@ AgentRegistry.register(CustomAsyncAgent()) ``` For an example implementation, see the [BigQuery agent](https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-bigquery/flytekitplugins/bigquery/agent.py#L43). + +```{note} + +To contribute + +``` From daa8a9feb3fa704bbbd134fa0e67e11b19ca719c Mon Sep 17 00:00:00 2001 From: nikki everett Date: Mon, 26 Feb 2024 11:30:31 -0600 Subject: [PATCH 4/4] add deprecated integrations section, update agents links Signed-off-by: nikki everett --- docs/conf.py | 3 ++- docs/core_use_cases/analytics.md | 4 ++-- docs/core_use_cases/data_engineering.md | 2 +- docs/flyte_fundamentals/optimizing_tasks.md | 2 +- docs/index.md | 1 + 5 files changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/conf.py b/docs/conf.py index 63a1ec9483..00c82afd2a 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -339,7 +339,8 @@ "source_regex_mapping": REPLACE_PATTERNS, "list_table_toc": [ "flytesnacks/tutorials", - "flytesnacks/integrations", + "flytesnacks/integrations", + "flytesnacks/deprecated_integrations" ], "dev_build": bool(int(os.environ.get("MONODOCS_DEV_BUILD", 1))), } diff --git a/docs/core_use_cases/analytics.md b/docs/core_use_cases/analytics.md index 58b6ab770c..886b75618d 100644 --- a/docs/core_use_cases/analytics.md +++ b/docs/core_use_cases/analytics.md @@ -173,7 +173,7 @@ and [DBT](https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-dbt) integrations. If you need to connect to a database, Flyte provides first-party -support for {ref}`AWS Athena `, {ref}`Google Bigquery `, -{ref}`Snowflake `, {ref}`SQLAlchemy `, and +support for {ref}`AWS Athena `, {ref}`Google Bigquery `, +{ref}`Snowflake `, {ref}`SQLAlchemy `, and {ref}`SQLite3 `. ``` diff --git a/docs/core_use_cases/data_engineering.md b/docs/core_use_cases/data_engineering.md index 25eb802fc2..9cbfca430c 100644 --- a/docs/core_use_cases/data_engineering.md +++ b/docs/core_use_cases/data_engineering.md @@ -170,6 +170,6 @@ and [DBT](https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-dbt) integrations. For database connectors, Flyte provides first-party support for {ref}`AWS Athena `, -{ref}`Google Bigquery `, {ref}`Snowflake `, +{ref}`Google BigQuery `, {ref}`Snowflake `, {ref}`SQLAlchemy `, and {ref}`SQLite3 `. ``` diff --git a/docs/flyte_fundamentals/optimizing_tasks.md b/docs/flyte_fundamentals/optimizing_tasks.md index 508767d05f..00c27c881f 100644 --- a/docs/flyte_fundamentals/optimizing_tasks.md +++ b/docs/flyte_fundamentals/optimizing_tasks.md @@ -243,7 +243,7 @@ When this task is executed on a Flyte cluster, it automatically provisions all o the resources that you need. In this case, that need is distributed training, but Flyte also provides integrations for {ref}`Spark `, {ref}`Ray `, {ref}`MPI `, {ref}`Sagemaker `, -{ref}`Snowflake `, and more. +{ref}`Snowflake `, and more. Even though Flyte itself is a powerful compute engine and orchestrator for data engineering, machine learning, and analytics, perhaps you have existing diff --git a/docs/index.md b/docs/index.md index 9df04ccd78..4720be51f7 100644 --- a/docs/index.md +++ b/docs/index.md @@ -151,6 +151,7 @@ Core use cases User Guide Tutorials Integrations +Deprecated integrations ``` ```{toctree}