From 4d9830690e446486c7b24f816ae3f0f531ef1631 Mon Sep 17 00:00:00 2001 From: Nikki Everett Date: Wed, 14 Feb 2024 10:37:30 -0600 Subject: [PATCH] Update docs links for monodocs (#4834) * update docs links in main README Signed-off-by: nikki everett * update links in contribution guide Signed-off-by: nikki everett * update links in concepts docs Signed-off-by: nikki everett * update docs links in community roadmap Signed-off-by: nikki everett * update link in contribution guide Signed-off-by: nikki everett * update link in concept doc Signed-off-by: nikki everett * update links in deployment docs Signed-off-by: nikki everett * update link in API docs landing page Signed-off-by: nikki everett * update CHANGELOG links Signed-off-by: nikki everett * update flyteidl links Signed-off-by: nikki everett * update links in rfcs Signed-off-by: nikki everett * update remaining old docs links Signed-off-by: nikki everett * change flyteidl files back to master state Signed-off-by: nikki everett --------- Signed-off-by: nikki everett --- CHANGELOG/CHANGELOG-v0.11.0.md | 2 +- CHANGELOG/CHANGELOG-v0.14.0.md | 2 +- CHANGELOG/CHANGELOG-v0.17.0.md | 8 +- CHANGELOG/CHANGELOG-v1.1.0.md | 2 +- CHANGELOG/CHANGELOG-v1.10.0.md | 4 +- CHANGELOG/CHANGELOG-v1.2.0.md | 2 +- CHANGELOG/CHANGELOG-v1.3.0.md | 2 +- CHANGELOG/CHANGELOG-v1.5.0.md | 4 +- CHANGELOG/CHANGELOG-v1.9.0.md | 4 +- README.md | 22 ++-- docs/community/contribute.rst | 12 +- docs/community/roadmap.rst | 2 +- .../native_scheduler_architecture.rst | 2 +- docs/concepts/registration.rst | 2 +- docs/concepts/tasks.rst | 4 +- docs/deployment/configuration/auth_setup.rst | 2 +- .../configuration/customizable_resources.rst | 114 ++++++++++++++++++ docs/deployment/configuration/general.rst | 20 +-- docs/deployment/plugins/aws/batch.rst | 2 +- .../registering_workflows.md | 3 +- .../flyte_project_components.md | 2 +- docs/reference/index.rst | 4 +- rfc/system/0001-flyte-execution-tags.md | 2 +- rfc/system/1476-task-resources.md | 6 +- .../1893-caching-of-offloaded-objects.md | 2 +- .../2633-eviction-of-cached-task-outputs.md | 6 +- rfc/system/3902-simplify-retry-behaviour.md | 2 +- 27 files changed, 176 insertions(+), 63 deletions(-) diff --git a/CHANGELOG/CHANGELOG-v0.11.0.md b/CHANGELOG/CHANGELOG-v0.11.0.md index ab19f1e21a..16c450a82e 100644 --- a/CHANGELOG/CHANGELOG-v0.11.0.md +++ b/CHANGELOG/CHANGELOG-v0.11.0.md @@ -4,7 +4,7 @@ * New to flyte? https://start.flyte.org takes you through first run experience. (Thanks to @jeevb) * [Grafana templates](https://docs.flyte.org/en/latest/howto/monitoring/index.html) for monitoring Flyte System and User Workflows. * [Extend Flyte](https://docs.flyte.org/en/latest/plugins/index.html) docs. -* [FlyteIdl Docs](https://docs.flyte.org/projects/flyteidl/en/latest/) are published! You can learn about the core language that makes it all work. +* [FlyteIdl Docs](https://docs.flyte.org/en/latest/) are published! You can learn about the core language that makes it all work. * [Additional knob](https://github.com/flyteorg/flytepropeller/pull/219/files#diff-91657d6448dfbf87f4cecf126ad02bd668ea233edcf74e860ef4f54bdd4cb552R78) for fine tuning flyte propeller performance that speeds up executions drastically. * OidC support for Google Idp (And other OidC compliant Idps) * Various stabilization bugs. diff --git a/CHANGELOG/CHANGELOG-v0.14.0.md b/CHANGELOG/CHANGELOG-v0.14.0.md index 344ccac919..705ab6ac03 100644 --- a/CHANGELOG/CHANGELOG-v0.14.0.md +++ b/CHANGELOG/CHANGELOG-v0.14.0.md @@ -27,6 +27,6 @@ - More use case driven examples in flytesnacks ## flytectl - - flytectl is ready for BETA. check it out - https://docs.flyte.org/projects/flytectl/en/latest/ + - flytectl is ready for BETA. check it out - https://docs.flyte.org/en/latest/flytectl/ Please see the [flytekit release](https://github.com/flyteorg/flytekit/releases/tag/v0.18.0) for the full list and more details. diff --git a/CHANGELOG/CHANGELOG-v0.17.0.md b/CHANGELOG/CHANGELOG-v0.17.0.md index 1a7d951ae2..2605bfae45 100644 --- a/CHANGELOG/CHANGELOG-v0.17.0.md +++ b/CHANGELOG/CHANGELOG-v0.17.0.md @@ -1,16 +1,16 @@ # Flyte v0.17.0 ## Platform -1. Recovery Mode: Executions that fail due to external system failures (e.g. external system being down) can now be rerun in recovery mode ([flytectl --recover docs](https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_create_execution.html)). It's also available in the UI: +1. Recovery Mode: Executions that fail due to external system failures (e.g. external system being down) can now be rerun in recovery mode ([flytectl --recover docs](https://docs.flyte.org/en/latest/flytectl/gen/flytectl_create_execution.html)). It's also available in the UI: Recovery mode in the UI ## Flytekit -1. Great Expectations Integration ([docs](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/flytekit_plugins/greatexpectations/index.html#great-expectations)). +1. Great Expectations Integration ([docs](https://docs.flyte.org/en/latest/flytesnacks/auto/integrations/flytekit_plugins/greatexpectations/index.html#great-expectations)). 1. Access to durable blob stores (AWS/GCS/etc) are now pluggable. 1. Local task execution has been updated to also trigger the type engine. -1. Tasks that have `cache=True` should now be cached when running locally as well ([docs](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/task_cache.html#how-local-caching-works)). +1. Tasks that have `cache=True` should now be cached when running locally as well ([docs](https://docs.flyte.org/en/latest/flytesnacks/auto/core/flyte_basics/task_cache.html#how-local-caching-works)). Please see the [flytekit release](https://github.com/flyteorg/flytekit/releases/tag/v0.22.0) for the full list and more details. @@ -32,7 +32,7 @@ Please see the [flytekit release](https://github.com/flyteorg/flytekit/releases/ ## FlyteCtl -1. `flytectl upgrade` to automatically upgrade itself ([docs](https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_upgrade.html)). +1. `flytectl upgrade` to automatically upgrade itself ([docs](https://docs.flyte.org/en/latest/flytectl/gen/flytectl_upgrade.html)). 1. `--dryRun` is available in most commands with server-side-effects to simulate the operations before committing any changes. And various stabilization [fixes](https://github.com/flyteorg/flyte/milestone/17?closed=1)! diff --git a/CHANGELOG/CHANGELOG-v1.1.0.md b/CHANGELOG/CHANGELOG-v1.1.0.md index f6906a6a5e..9b84ae94d8 100644 --- a/CHANGELOG/CHANGELOG-v1.1.0.md +++ b/CHANGELOG/CHANGELOG-v1.1.0.md @@ -4,7 +4,7 @@ ### User Improvements Support for [Optional types](https://github.com/flyteorg/flyte/issues/2426). With the inclusion of Union types in flytekit, we can now support optional types. -[Flyte Deck](https://github.com/flyteorg/flyte/issues/2175) is now available. Please take a look at the [documentation](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/deck.html#sphx-glr-auto-core-flyte-basics-deck-py) and also the [OSS presentation](https://www.youtube.com/watch?v=KqyBYIaAZ7c) that was done a few weeks back. +[Flyte Deck](https://github.com/flyteorg/flyte/issues/2175) is now available. Please take a look at the [documentation](https://docs.flyte.org/en/latest/flytesnacks/auto/core/flyte_basics/deck.html#sphx-glr-auto-core-flyte-basics-deck-py) and also the [OSS presentation](https://www.youtube.com/watch?v=KqyBYIaAZ7c) that was done a few weeks back. ### Backend Improvements diff --git a/CHANGELOG/CHANGELOG-v1.10.0.md b/CHANGELOG/CHANGELOG-v1.10.0.md index 42c301ac2e..48d298ccf7 100644 --- a/CHANGELOG/CHANGELOG-v1.10.0.md +++ b/CHANGELOG/CHANGELOG-v1.10.0.md @@ -8,9 +8,9 @@ Programmatically consuming inputs and outputs using flyteremote became a lot eas ![Usage snippet](./images/v1.10.0-flyteconsole-programmatic-access.png) -You'll now be able to use offloaded types in [eager workflows](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/advanced_composition/eager_workflows.html#id1). +You'll now be able to use offloaded types in [eager workflows](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/eager_workflows.html). -More ergonomic improvements to [pyflyte](https://docs.flyte.org/projects/flytekit/en/latest/pyflyte.html#pyflyte-cli), including the inclusion of a progress bar, the ability to activate launchplans, and the ability to interact with gate nodes in local executions. +More ergonomic improvements to [pyflyte](https://docs.flyte.org/en/latest/api/flytekit/pyflyte.html), including the inclusion of a progress bar, the ability to activate launchplans, and the ability to interact with gate nodes in local executions. And much more. Here's the exhaustive list of changes: diff --git a/CHANGELOG/CHANGELOG-v1.2.0.md b/CHANGELOG/CHANGELOG-v1.2.0.md index 4dbd2f3a2a..6ceafe8cfe 100644 --- a/CHANGELOG/CHANGELOG-v1.2.0.md +++ b/CHANGELOG/CHANGELOG-v1.2.0.md @@ -18,7 +18,7 @@ - dbt plugin (https://github.com/flyteorg/flyte/issues/2202) - cache overriding behavior is now open to all types (https://github.com/flyteorg/flyte/issues/2912) - Bug: Fallback to pickling in the case of unknown types used Unions (https://github.com/flyteorg/flyte/issues/2823) -- [pyflyte run](https://docs.flyte.org/projects/flytekit/en/latest/design/clis.html#pyflyte-run) now supports [imperative workflows](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/imperative_wf_style.html#sphx-glr-auto-core-flyte-basics-imperative-wf-style-py) +- [pyflyte run](https://docs.flyte.org/en/latest/api/flytekit/design/clis.html#pyflyte-run) now supports [imperative workflows](https://docs.flyte.org/en/latest/flytesnacks/auto/core/flyte_basics/imperative_wf_style.html#sphx-glr-auto-core-flyte-basics-imperative-wf-style-py) - Newlines are now stripped from client secrets (https://github.com/flyteorg/flytekit/pull/1163) - Ensure repeatability in the generation of cache keys in the case of dictionaries (https://github.com/flyteorg/flytekit/pull/1126) - Support for multiple images in the yaml config file (https://github.com/flyteorg/flytekit/pull/1106) diff --git a/CHANGELOG/CHANGELOG-v1.3.0.md b/CHANGELOG/CHANGELOG-v1.3.0.md index c710af9334..44b836313c 100644 --- a/CHANGELOG/CHANGELOG-v1.3.0.md +++ b/CHANGELOG/CHANGELOG-v1.3.0.md @@ -99,7 +99,7 @@ Users can now configure workflow execution to overwrite the cache. Each task in ### Support for Dask -Users will be able to spawn [Dask](https://www.dask.org/) ephemeral clusters as part of their workflows, similar to the support for [Ray](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/ray_example/ray_example.html#sphx-glr-auto-integrations-kubernetes-ray-example-ray-example-py) and [Spark](https://docs.flyte.org/projects/cookbook/en/stable/auto/integrations/kubernetes/k8s_spark/pyspark_pi.html). +Users will be able to spawn [Dask](https://www.dask.org/) ephemeral clusters as part of their workflows, similar to the support for [Ray](https://docs.flyte.org/en/latest/flytesnacks/auto/integrations/kubernetes/ray_example/ray_example.html#sphx-glr-auto-integrations-kubernetes-ray-example-ray-example-py) and [Spark](https://docs.flyte.org/en/latest/flytesnacks/examples/k8s_spark_plugin/index.html). ## Looking Ahead diff --git a/CHANGELOG/CHANGELOG-v1.5.0.md b/CHANGELOG/CHANGELOG-v1.5.0.md index 718cb6f61a..0c4c7eb4eb 100644 --- a/CHANGELOG/CHANGELOG-v1.5.0.md +++ b/CHANGELOG/CHANGELOG-v1.5.0.md @@ -63,7 +63,7 @@ def wf(a: int) -> str: Notice how calls to `t1_fixed_b` do not need to specify the `b` parameter. -This also works for [MapTasks](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/map_task.html#sphx-glr-auto-core-control-flow-map-task-py) in a limited capacity. For example: +This also works for [MapTasks](https://docs.flyte.org/en/latest/flytesnacks/auto/core/control_flow/map_task.html#sphx-glr-auto-core-control-flow-map-task-py) in a limited capacity. For example: ``` from flytekit import task, workflow, partial, map_task @@ -107,5 +107,5 @@ Map tasks do not support partial tasks with lists as inputs. ## Flyteconsole -Multiple bug fixes around [waiting for external inputs](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/waiting_for_external_inputs.html#waiting-for-external-inputs). +Multiple bug fixes around [waiting for external inputs](https://docs.flyte.org/en/latest/flytesnacks/auto/core/control_flow/waiting_for_external_inputs.html#waiting-for-external-inputs). Better support for dataclasses in the launch form. diff --git a/CHANGELOG/CHANGELOG-v1.9.0.md b/CHANGELOG/CHANGELOG-v1.9.0.md index 47c62b92a3..90371e5c11 100644 --- a/CHANGELOG/CHANGELOG-v1.9.0.md +++ b/CHANGELOG/CHANGELOG-v1.9.0.md @@ -5,7 +5,7 @@ In this release we're announcing two experimental features, namely (1) ArrayNode ### ArrayNode map tasks -ArrayNodes are described more fully in [RFC 3346](https://github.com/flyteorg/flyte/blob/master/rfc/system/3346-array-node.md), but the summary is that ArrayNode map tasks are a drop-in replacement for [regular map tasks](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/control_flow/map_task.html#map-tasks), the only difference being the submodule used to import the `map_task` function. +ArrayNodes are described more fully in [RFC 3346](https://github.com/flyteorg/flyte/blob/master/rfc/system/3346-array-node.md), but the summary is that ArrayNode map tasks are a drop-in replacement for [regular map tasks](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/map_task.html), the only difference being the submodule used to import the `map_task` function. More explicitly, let's say you have this code: ```python @@ -42,7 +42,7 @@ def wf(xs: List[int]) -> List[int]: Execution tags allow users to can discover their executions and other flyte entities more easily, by creating smarter groupings. The feature is described in this [RFC](https://github.com/flyteorg/flyte/blob/master/rfc/system/0001-flyte-execution-tags.md). -As mentioned before, this feature is shipped in an experimental capacity, the idea being that we're going to incorporate the feedback of the community as we iterate. More work is expected to give prominence to the feature in flyteconsole, in the meanwhile, the feature is supported via [Remote](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/remote_access/index.html). +As mentioned before, this feature is shipped in an experimental capacity, the idea being that we're going to incorporate the feedback of the community as we iterate. More work is expected to give prominence to the feature in flyteconsole, in the meanwhile, the feature is supported via [Remote](https://docs.flyte.org/en/latest/api/flytekit/remote.html#remote-access). ## Flytekit diff --git a/README.md b/README.md index 24c2b3aff6..43c9e72a82 100644 --- a/README.md +++ b/README.md @@ -99,32 +99,32 @@ Go to the [Deployment guide](https://docs.flyte.org/en/latest/deployment/deploym ## Tutorials - [Fine-tune Code Llama on the Flyte codebase](https://github.com/unionai-oss/llm-fine-tuning/tree/main/flyte_llama#readme) -- [Forecast sales with Horovod and Spark](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/forecasting_sales/index.html) -- [Nucleotide Sequence Querying with BLASTX](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/blast/index.html) +- [Forecast sales with Horovod and Spark](https://docs.flyte.org/en/latest/flytesnacks/examples/forecasting_sales/index.html) +- [Nucleotide Sequence Querying with BLASTX](https://docs.flyte.org/en/latest/flytesnacks/examples/blast/index.html) ## Features ๐Ÿš€ **Strongly typed interfaces**: Validate your data at every step of the workflow by defining data guardrails using Flyte types.
๐ŸŒ **Any language**: Write code in any language using raw containers, or choose [Python](https://github.com/flyteorg/flytekit), [Java](https://github.com/flyteorg/flytekit-java), [Scala](https://github.com/flyteorg/flytekit-java) or [JavaScript](https://github.com/NotMatthewGriffin/pterodactyl) SDKs to develop your Flyte workflows.
๐Ÿ”’ **Immutability**: Immutable executions help ensure reproducibility by preventing any changes to the state of an execution.
๐Ÿงฌ **Data lineage**: Track the movement and transformation of data throughout the lifecycle of your data and ML workflows.
-๐Ÿ“Š **Map tasks**: Achieve parallel code execution with minimal configuration using [map tasks](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/advanced_composition/map_task.html).
+๐Ÿ“Š **Map tasks**: Achieve parallel code execution with minimal configuration using [map tasks](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/map_task.html).
๐ŸŒŽ **Multi-tenancy**: Multiple users can share the same platform while maintaining their own distinct data and configurations.
-๐ŸŒŸ **Dynamic workflows**: [Build flexible and adaptable workflows](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/advanced_composition/dynamics.html) that can change and evolve as needed, making it easier to respond to changing requirements.
-โฏ๏ธ [Wait](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/advanced_composition/waiting_for_external_inputs.html) for **external inputs** before proceeding with the execution.
-๐ŸŒณ **Branching**: [Selectively execute branches](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/advanced_composition/conditions.html) of your workflow based on static or dynamic data produced by other tasks or input data.
+๐ŸŒŸ **Dynamic workflows**: [Build flexible and adaptable workflows](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/dynamic_workflow.html) that can change and evolve as needed, making it easier to respond to changing requirements.
+โฏ๏ธ [Wait](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/waiting_for_external_inputs.html) for **external inputs** before proceeding with the execution.
+๐ŸŒณ **Branching**: [Selectively execute branches](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/conditional.html) of your workflow based on static or dynamic data produced by other tasks or input data.
๐Ÿ“ˆ **Data visualization**: Visualize data, monitor models and view training history through plots.
-๐Ÿ“‚ **FlyteFile & FlyteDirectory**: Transfer [files](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/advanced_composition/files.html) and [directories](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/advanced_composition/folders.html) between local and cloud storage.
-๐Ÿ—ƒ๏ธ **Structured dataset**: Convert dataframes between types and enforce column-level type checking using the abstract 2D representation provided by [Structured Dataset](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/data_types_and_io/structured_dataset.html).
+๐Ÿ“‚ **FlyteFile & FlyteDirectory**: Transfer [files](https://docs.flyte.org/en/latest/flytesnacks/examples/data_types_and_io/file.html#file) and [directories](https://docs.flyte.org/en/latest/flytesnacks/examples/data_types_and_io/folder.html) between local and cloud storage.
+๐Ÿ—ƒ๏ธ **Structured dataset**: Convert dataframes between types and enforce column-level type checking using the abstract 2D representation provided by [Structured Dataset](https://docs.flyte.org/en/latest/flytesnacks/examples/data_types_and_io/structured_dataset.html).
๐Ÿ›ก๏ธ **Recover from failures**: Recover only the failed tasks.
๐Ÿ” **Rerun a single task**: Rerun workflows at the most granular level without modifying the previous state of a data/ML workflow.
๐Ÿ” **Cache outputs**: Cache task outputs by passing `cache=True` to the task decorator.
-๐Ÿšฉ **Intra-task checkpointing**: [Checkpoint progress](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/advanced_composition/checkpoint.html) within a task execution.
+๐Ÿšฉ **Intra-task checkpointing**: [Checkpoint progress](https://docs.flyte.org/en/latest/flytesnacks/examples/advanced_composition/checkpoint.html) within a task execution.
โฐ **Timeout**: Define a timeout period, after which the task is marked as failure.
๐Ÿญ **Dev to prod**: As simple as changing your [domain](https://docs.flyte.org/en/latest/concepts/domains.html) from development or staging to production.
๐Ÿ’ธ **Spot or preemptible instances**: Schedule your workflows on spot instances by setting `interruptible` to `True` in the task decorator.
โ˜๏ธ **Cloud-native deployment**: Deploy Flyte on AWS, GCP, Azure and other cloud services.
-๐Ÿ“… **Scheduling**: [Schedule](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/productionizing/lp_schedules.html) your data and ML workflows to run at a specific time.
-๐Ÿ“ข **Notifications**: Stay informed about changes to your workflow's state by configuring [notifications](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/productionizing/lp_notifications.html) through Slack, PagerDuty or email.
+๐Ÿ“… **Scheduling**: [Schedule](https://docs.flyte.org/en/latest/flytesnacks/examples/productionizing/lp_schedules.html) your data and ML workflows to run at a specific time.
+๐Ÿ“ข **Notifications**: Stay informed about changes to your workflow's state by configuring [notifications](https://docs.flyte.org/en/latest/flytesnacks/examples/productionizing/lp_notifications.html) through Slack, PagerDuty or email.
โŒ›๏ธ **Timeline view**: Evaluate the duration of each of your Flyte tasks and identify potential bottlenecks.
๐Ÿ’จ **GPU acceleration**: Enable and control your tasksโ€™ GPU demands by requesting resources in the task decorator.
๐Ÿณ **Dependency isolation via containers**: Maintain separate sets of dependencies for your tasks so no dependency conflicts arise.
diff --git a/docs/community/contribute.rst b/docs/community/contribute.rst index f99b5adaec..12cbf38b01 100644 --- a/docs/community/contribute.rst +++ b/docs/community/contribute.rst @@ -227,7 +227,7 @@ The resulting ``html`` files will be in ``docs/_build/html``. You can view them * - `Repo `__ * - **Purpose**: Python SDK & Tools * - **Language**: Python - * - **Guidelines**: Refer to the `Flytekit Contribution Guide `__ + * - **Guidelines**: Refer to the `Flytekit Contribution Guide `__ ``flyteconsole`` **************** @@ -281,8 +281,8 @@ The resulting ``html`` files will be in ``docs/_build/html``. You can view them * - `Repo `__ * - **Purpose**: Examples, Tips, and Tricks to use Flytekit SDKs * - **Language**: Python (In the future, Java examples will be added) - * - **Guidelines**: Refer to the `Flytesnacks Contribution Guide `__ - + * - **Guidelines**: Refer to the `Flytesnacks Contribution Guide `__ + ``flytectl`` ************ @@ -291,7 +291,7 @@ The resulting ``html`` files will be in ``docs/_build/html``. You can view them * - `Repo `__ * - **Purpose**: A standalone Flyte CLI * - **Language**: Go - * - **Guidelines**: Refer to the `FlyteCTL Contribution Guide `__ + * - **Guidelines**: Refer to the `FlyteCTL Contribution Guide `__ ๐Ÿ”ฎ Development Environment Setup Guide @@ -505,7 +505,7 @@ If not, we can start backends with a single command. export PATH=$PATH:/home/ubuntu/bin # replace with your path # Step3: Starts the Flyte demo cluster. This will setup a k3s cluster running minio, postgres Pods, and all Flyte components: flyteadmin, flyteplugins, flytepropeller, etc. - # See https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_demo_start.html for more details. + # See https://docs.flyte.org/en/latest/flytectl/gen/flytectl_demo_start.html for more details. flytectl demo start # ๐Ÿ‘จโ€๐Ÿ’ป Flyte is ready! Flyte UI is available at http://localhost:30080/console ๐Ÿš€ ๐Ÿš€ ๐ŸŽ‰ # โ‡๏ธ Run the following command to export demo environment variables for accessing flytectl @@ -677,7 +677,7 @@ You can access it via http://localhost:30080/console. Core Flyte components, such as admin, propeller, and datacatalog, as well as user runtime containers rely on an object store (in this case, minio) to hold files. -During development, you might need to examine files such as `input.pb/output.pb `__, or `deck.html `__ stored in minio. +During development, you might need to examine files such as `input.pb/output.pb `__, or `deck.html `__ stored in minio. Access the minio console at: http://localhost:30080/minio/login. The default credentials are: diff --git a/docs/community/roadmap.rst b/docs/community/roadmap.rst index 3e6bc3f5ae..6ea83ec5d9 100644 --- a/docs/community/roadmap.rst +++ b/docs/community/roadmap.rst @@ -17,7 +17,7 @@ It is extremely important to let the community know about your use cases, so tha Milestones and Release Processes ================================ -Flyte consists of many components and services. Each service is independently iterated and coordinated by maintaining backwards compatible contracts using Protobuf messages defined in `FlyteIDL `__. +Flyte consists of many components and services. Each service is independently iterated and coordinated by maintaining backwards compatible contracts using Protobuf messages defined in `FlyteIDL `__. Release Cadence --------------- diff --git a/docs/concepts/component_architecture/native_scheduler_architecture.rst b/docs/concepts/component_architecture/native_scheduler_architecture.rst index 19f13ef6c7..a923403625 100644 --- a/docs/concepts/component_architecture/native_scheduler_architecture.rst +++ b/docs/concepts/component_architecture/native_scheduler_architecture.rst @@ -35,7 +35,7 @@ Components Schedule Management ------------------- -This component supports creation/activation and deactivation of schedules. Each schedule is tied to a launch plan and is versioned in a similar manner. The schedule is created or its state is changed to activated/deactivated whenever the `admin API `__ is invoked for it with `ACTIVE/INACTIVE state `__. This is done either through `flytectl `__ or through any other client that calls the GRPC API. +This component supports creation/activation and deactivation of schedules. Each schedule is tied to a launch plan and is versioned in a similar manner. The schedule is created or its state is changed to activated/deactivated whenever the `admin API `__ is invoked for it with `ACTIVE/INACTIVE state `__. This is done either through `flytectl `__ or through any other client that calls the GRPC API. The API is similar to a launchplan, ensuring that only one schedule is active for a given launchplan. diff --git a/docs/concepts/registration.rst b/docs/concepts/registration.rst index bc745f7a0f..b7552d9fb9 100644 --- a/docs/concepts/registration.rst +++ b/docs/concepts/registration.rst @@ -16,7 +16,7 @@ The following steps elaborate on the specifics of the registration process: * Define the tasks using the :py:mod:`Flytekit ` Task Definition language. * Define a workflow using the :py:mod:`Flytekit ` Workflow definition language. -* Use `flytectl register CLI `__ to compile the tasks into their serialized representation as described in :std:ref:`Flyte Specification language `. During this, the task representation is bound to a container that constitutes the code for the task. This associated entity is registered with FlyteAdmin using the registerTask API. +* Use `flytectl register CLI `__ to compile the tasks into their serialized representation as described in :std:ref:`Flyte Specification language `. During this, the task representation is bound to a container that constitutes the code for the task. This associated entity is registered with FlyteAdmin using the registerTask API. * Use flytectl register CLI to compile the workflow into their serialized representation as described in :std:ref:`Flyte Specification language `. The referenced tasks are replaced by their FlyteAdmin registered Identifiers, obtained in the previous step. The associated entity is registered with FlyteAdmin using the registerWorkflow API. * Launch an execution using the FlyteAdmin launch execution API, which requires the necessary inputs provided. This is automatically done if the user uses flytectl to launch the execution. * Use the FlyteAdmin read APIs to get details of the execution, monitor it to completion, or retrieve a historical execution. diff --git a/docs/concepts/tasks.rst b/docs/concepts/tasks.rst index 301287fc7b..f3ae87709e 100644 --- a/docs/concepts/tasks.rst +++ b/docs/concepts/tasks.rst @@ -99,7 +99,7 @@ System retry can be of two types: - Or update `max-workflow-retries `__ in helm. -2. User retry: If a task fails to execute, it is retried for a specific number of times, and this number is set by the user in `TaskMetadata `__. The number of retries must be less than or equal to 10. +2. User retry: If a task fails to execute, it is retried for a specific number of times, and this number is set by the user in `TaskMetadata `__. The number of retries must be less than or equal to 10. .. note:: @@ -113,7 +113,7 @@ System retry can be of two types: **Timeouts** -To ensure that the system is always making progress, tasks must be guaranteed to end gracefully/successfully. The system defines a default timeout period for the tasks. It is possible for task authors to define a timeout period, after which the task is marked as ``failure``. Note that a timed-out task will be retried if it has a retry strategy defined. The timeout can be handled in the `TaskMetadata `__. +To ensure that the system is always making progress, tasks must be guaranteed to end gracefully/successfully. The system defines a default timeout period for the tasks. It is possible for task authors to define a timeout period, after which the task is marked as ``failure``. Note that a timed-out task will be retried if it has a retry strategy defined. The timeout can be handled in the `TaskMetadata `__. Caching/Memoization diff --git a/docs/deployment/configuration/auth_setup.rst b/docs/deployment/configuration/auth_setup.rst index 41c10f03b1..2887e830ed 100644 --- a/docs/deployment/configuration/auth_setup.rst +++ b/docs/deployment/configuration/auth_setup.rst @@ -702,7 +702,7 @@ If your organization does any automated registration, then you'll need to authen .. group-tab:: flytectl - Flytectl's `config.yaml `_ can be + Flytectl's `config.yaml `_ can be configured to use either PKCE (`Proof key for code exchange `_) or Client Credentials (`Client Credentials `_) flows. diff --git a/docs/deployment/configuration/customizable_resources.rst b/docs/deployment/configuration/customizable_resources.rst index 2b785d31f6..778059b49b 100644 --- a/docs/deployment/configuration/customizable_resources.rst +++ b/docs/deployment/configuration/customizable_resources.rst @@ -370,3 +370,117 @@ Flytekit For convenience, add a FlyteCTL wrapper to update the new attributes. Refer to `this PR `__ for the entire set of changes required. That's it! You now have a new matchable attribute to configure as the needs of your users evolve. +<<<<<<< HEAD +======= + +Flyte ResourceManager +--------------------- + +**Flyte ResourceManager** is a configurable component that allows plugins to manage resource allocations independently. It helps track resource utilization of tasks that run on Flyte. The default deployments are configured as ``noop``, which indicates that the ResourceManager provided by Flyte is disabled and plugins rely on each independent platform to manage resource utilization. In situations like the K8s plugin, where the platform has a robust mechanism to manage resource scheduling, this may work well. However, in a scenario like a simple web API plugin, the rate at which Flyte sends requests may overwhelm a service and benefit from additional resource management. + +The below attribute is configurable within FlytePropeller, which can be disabled with: + +.. code-block:: yaml + + resourcemanager: + type: noop + +The ResourceManager provides a task-type-specific pooling system for Flyte tasks. Optionally, plugin writers can request resource allocation in their tasks. + +A plugin defines a collection of resource pools using its configuration. Flyte uses tokens as a placeholder to represent a unit of resource. + +How does a Flyte plugin request for resources? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The Flyte plugin registers the resource and the desired quota of every resource with the **ResourceRegistrar** when setting up FlytePropeller. When a plugin is invoked, FlytePropeller provides a proxy for the plugin. This proxy facilitates the plugin's view of the resource pool by controlling operations to allocate and deallocate resources. + +.. dropdown:: :fa:`info-circle` Enabling Redis instance + :title: text-muted + :animate: fade-in-slide-down + + The ResourceManager can use a Redis instance as an external store to track and manage resource pool allocation. By default, it is disabled, and can be enabled with: + + .. code-block:: yaml + + resourcemanager: + type: redis + resourceMaxQuota: 100 + redis: + hostPaths: + - foo + hostKey: bar + maxRetries: 0 + +Once the setup is complete, FlytePropeller builds a ResourceManager based on the previously requested resource registration. Based on the plugin implementation's logic, resources are allocated and deallocated. + +During runtime, the ResourceManager: + +#. Allocates tokens to the plugin. +#. Releases tokens once the task is completed. + +How are resources allocated? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When a Flyte task execution needs to send a request to an external service, the plugin claims a unit of the corresponding resource. This is done using a **ResourceName**, which is a unique token and a fully qualified resource request (which is typically an integer). The execution generates this unique token and registers this token with the ResourceManager by calling the ResourceManagerโ€™s **"AllocateResource function"**. If the resource pool has sufficient capacity to fulfil your request, then the resources requested are allocated, and the plugin proceeds further. + +When the status is **"AllocationGranted"**, the execution moves forward and sends out the request for those resources. + +The granted token is recorded in a token pool which corresponds to the resource that is managed by the ResourceManager. + +How are resources deallocated? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +When the request is completed, the plugin asks the ResourceManager to release the token by calling the ReleaseResource() function present in the ResourceManager. Upon calling the function, the token is eliminated from the token pool. +In this manner, Flyte plugins intelligently throttle resource usage during parallel execution of nodes. + +Example +^^^^^^^^ +Let's take an example to understand resource allocation and deallocation when a plugin requests resources. + +Flyte has a built-in `Qubole `__ plugin. This plugin allows Flyte tasks to send Hive commands to Qubole. In the plugin, a single Qubole cluster is considered a resource, and sending a single Hive command to a Qubole cluster consumes a token of the corresponding resource. +The resource is allocated when the status is **โ€œAllocationGrantedโ€**. Qubole plugin calls: + +.. code-block:: go + + status, err := AllocateResource(ctx, , , ) + +Wherein the placeholders are occupied by: + +.. code-block:: go + + status, err := AllocateResource(ctx, "default_cluster", "flkgiwd13-akjdoe-0", ResourceConstraintsSpec{}) + +The resource is deallocated when the Hive command completes its execution and the corresponding token is released. The plugin calls: + +.. code-block:: go + + status, err := AllocateResource(ctx, , , ) + +Wherein the placeholders are occupied by: + +.. code-block:: go + + err := ReleaseResource(ctx, "default_cluster", "flkgiwd13-akjdoe-0") + +Below is an example interface that shows allocation and deallocation of resources. + +.. code-block:: go + + type ResourceManager interface { + GetID() string + // During execution, the plugin calls AllocateResource() to register a token in the token pool associated with a resource + // If it is granted an allocation, the token is recorded in the token pool until the same plugin releases it. + // When calling AllocateResource, the plugin has to specify a ResourceConstraintsSpec that contains resource capping constraints at different project and namespace levels. + // The ResourceConstraint pointers in ResourceConstraintsSpec can be set to nil to not have a constraint at that level + AllocateResource(ctx context.Context, namespace ResourceNamespace, allocationToken string, constraintsSpec ResourceConstraintsSpec) (AllocationStatus, error) + // During execution, after an outstanding request is completed, the plugin uses ReleaseResource() to release the allocation of the token from the token pool. This way, it redeems the quota taken by the token + ReleaseResource(ctx context.Context, namespace ResourceNamespace, allocationToken string) error + } + +How can you force ResourceManager to force runtime quota allocation constraints? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Runtime quota allocation constraints can be achieved using ResourceConstraintsSpec. It is a contact that a plugin can specify at different project and namespace levels. + +Let's take an example to understand it. + +You can set ResourceConstraintsSpec to ``nil`` objects, which means there would be no allocation constraints at the respective project and namespace level. When ResourceConstraintsSpec specifies ``nil`` ProjectScopeResourceConstraint, and a non-nil NamespaceScopeResourceConstraint, it suggests no constraints specified at any project or namespace level. +>>>>>>> master \ No newline at end of file diff --git a/docs/deployment/configuration/general.rst b/docs/deployment/configuration/general.rst index cf8ab2403a..e02d22b3ef 100644 --- a/docs/deployment/configuration/general.rst +++ b/docs/deployment/configuration/general.rst @@ -31,7 +31,7 @@ Starting with the Flyte 1.4 release, there are two ways of defining `PodTemplate Compile-time PodTemplates ************************* -We can define a compile-time pod template, as part of the definition of a `Task `__, for example: +We can define a compile-time pod template, as part of the definition of a `Task `__, for example: .. code-block:: python @@ -70,7 +70,7 @@ Notice how in this example we are defining a new PodTemplate inline, which allow `V1PodSpec `__ and also define the name of the primary container, labels, and annotations. -The term compile-time here refers to the fact that the pod template definition is part of the `TaskSpec `__. +The term compile-time here refers to the fact that the pod template definition is part of the `TaskSpec `__. ******************** Runtime PodTemplates @@ -92,7 +92,7 @@ initializes a K8s informer internally to track system PodTemplate updates `aware `__ of the latest PodTemplate definitions in the K8s environment. You can find this setting in `FlytePropeller `__ -config map, which is not set by default. +config map, which is not set by default. An example configuration is: @@ -105,14 +105,14 @@ An example configuration is: image: "cr.flyte.org/flyteorg/flytecopilot:v0.0.15" start-timeout: "30s" default-pod-template-name: - + Create a PodTemplate resource ============================= -Flyte recognizes PodTemplate definitions with the ``default-pod-template-name`` at two granularities. +Flyte recognizes PodTemplate definitions with the ``default-pod-template-name`` at two granularities. 1. A system-wide configuration can be created in the same namespace that - FlytePropeller is running in (typically `flyte`). + FlytePropeller is running in (typically `flyte`). 2. PodTemplates can be applied from the same namespace that the Pod will be created in. FlytePropeller always favors the PodTemplate with the more specific namespace. For example, a Pod created in the ``flytesnacks-development`` @@ -197,7 +197,7 @@ where you start the Pod. An example PodTemplate is shown: .. code-block:: yaml - + apiVersion: v1 kind: PodTemplate metadata: @@ -221,7 +221,7 @@ In addition, the K8s plugin configuration in FlytePropeller defines the default Pod Labels, Annotations, and enables the host networking. .. code-block:: yaml - + plugins: k8s: default-labels: @@ -234,7 +234,7 @@ Pod Labels, Annotations, and enables the host networking. To construct a Pod, FlytePropeller initializes a Pod definition using the default PodTemplate. This definition is applied to the K8s plugin configuration values, and any task-specific configuration is overlaid. During the process, when lists -are merged, values are appended and when maps are merged, the values are overridden. +are merged, values are appended and when maps are merged, the values are overridden. The resultant Pod using the above default PodTemplate and K8s Plugin configuration is shown: .. code-block:: yaml @@ -454,4 +454,4 @@ The resultant pod for that task is as follows: image: a.b.c/image:v1 command: cmd args: [] - // remaining container configuration omitted + // remaining container configuration omitted \ No newline at end of file diff --git a/docs/deployment/plugins/aws/batch.rst b/docs/deployment/plugins/aws/batch.rst index 5bb4c77e8c..f640b7907a 100644 --- a/docs/deployment/plugins/aws/batch.rst +++ b/docs/deployment/plugins/aws/batch.rst @@ -146,7 +146,7 @@ These configurations reside within FlytePropeller's configMap. Modify the config .. note:: To register the `map task - `__ on Flyte, + `__ on Flyte, use the command ``pyflyte register ``. Launch the execution through the FlyteConsole by selecting the appropriate ``IAM Role`` and entering the full ``AWS Arn`` of an IAM Role configured according to the above guide. diff --git a/docs/flyte_fundamentals/registering_workflows.md b/docs/flyte_fundamentals/registering_workflows.md index 4e2186df7f..9e503e4127 100644 --- a/docs/flyte_fundamentals/registering_workflows.md +++ b/docs/flyte_fundamentals/registering_workflows.md @@ -23,8 +23,7 @@ If you have custom Python dependencies, update the `requirements.txt` file that ships with the {ref}`project template ` and those changes will be incorporated into the Docker image. -You can also update the Dockerfile (if using a Dockerfile) or the [ImageSpec configuration](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/customizing_dependencies/image_spec.html#image-spec-example) if you -want to use a different base image or if the additional Python dependencies +You can also update the Dockerfile (if using a Dockerfile) or the {ref}`ImageSpec configuration ` if you want to use a different base image or if the additional Python dependencies require installing binaries or packages from other languages. ## Registration patterns diff --git a/docs/getting_started_with_workflow_development/flyte_project_components.md b/docs/getting_started_with_workflow_development/flyte_project_components.md index bbd0f66a79..18fcf86136 100644 --- a/docs/getting_started_with_workflow_development/flyte_project_components.md +++ b/docs/getting_started_with_workflow_development/flyte_project_components.md @@ -52,7 +52,7 @@ The workflow directory also contains an `__init__.py` file to indicate that the ### ImageSpec -The workflow code file in the basic template includes an optional ImageSpec configuration. ImageSpec is a Flyte feature that enables you to build a custom container image without having to write a Dockerfile. To learn more, see the [ImageSpec documentation](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/customizing_dependencies/image_spec.html#image-spec-example) +The workflow code file in the basic template includes an optional ImageSpec configuration. ImageSpec is a Flyte feature that enables you to build a custom container image without having to write a Dockerfile. To learn more, see the {ref}`ImageSpec documentation `. ```python # basic_image = ImageSpec( diff --git a/docs/reference/index.rst b/docs/reference/index.rst index 1e40110b0c..2b4d977a45 100644 --- a/docs/reference/index.rst +++ b/docs/reference/index.rst @@ -44,7 +44,7 @@ API Reference --- - .. link-button:: https://docs.flyte.org/projects/flyteidl/en/latest/protos/docs/service/service.html + .. link-button:: https://docs.flyte.org/en/latest/protos/docs/service/service.html :type: url :text: FlyteAdmin :classes: btn-block stretched-link @@ -71,5 +71,5 @@ API Reference FlyteIDL Flytekit Python Flytekit Java - FlyteAdmin + FlyteAdmin swagger diff --git a/rfc/system/0001-flyte-execution-tags.md b/rfc/system/0001-flyte-execution-tags.md index a30ea7c9de..e1c8f8b82d 100644 --- a/rfc/system/0001-flyte-execution-tags.md +++ b/rfc/system/0001-flyte-execution-tags.md @@ -32,7 +32,7 @@ As a User I want to We propose to solve the problem of discovery by supporting arbitrary metadata association with an entity. This is similar to concept of โ€œtagsโ€ as in AWS. The tags are represented as plain string. -We'll add tags to [ExecutionCreateRequest](https://docs.flyte.org/projects/flyteidl/en/latest/protos/docs/admin/admin.html#executioncreaterequest) -> [ExecutionSpec](https://docs.flyte.org/projects/flyteidl/en/latest/protos/docs/admin/admin.html#executionspec). +We'll add tags to [ExecutionCreateRequest](https://docs.flyte.org/en/latest/protos/docs/admin/admin.html#executioncreaterequest) -> [ExecutionSpec](https://docs.flyte.org/en/latest/protos/docs/admin/admin.html#executionspec). The resultant tags will be persisted in the database, instead of being applied to the execution in Kubernetes. We'll create two new tables in the flyteadmin database. diff --git a/rfc/system/1476-task-resources.md b/rfc/system/1476-task-resources.md index 8889b53752..638cfbfbb5 100644 --- a/rfc/system/1476-task-resources.md +++ b/rfc/system/1476-task-resources.md @@ -6,7 +6,7 @@ ## 1 Executive Summary -Task resource allocation in Flyte includes the process of setting *CPU, memory, GPU* and *ephemeral storage* requests and limits for containers running Flyte tasks on [kubernetes](https://docs.flyte.org/projects/cookbook/en/latest/native_backend_plugins.html#native-backend-plugins). These resource selections affect pod scheduling decisions and as such sensible defaults ought to be applied when a user doesn't specify requests or limits. This fallback behavior currently exists in Flyte but has been configured and modified organically such that the default value assignment has grown convoluted and unfortunately error-prone. +Task resource allocation in Flyte includes the process of setting *CPU, memory, GPU* and *ephemeral storage* requests and limits for containers running Flyte tasks on [kubernetes](https://docs.flyte.org/en/latest/flytesnacks/native_backend_plugins.html#native-backend-plugins). These resource selections affect pod scheduling decisions and as such sensible defaults ought to be applied when a user doesn't specify requests or limits. This fallback behavior currently exists in Flyte but has been configured and modified organically such that the default value assignment has grown convoluted and unfortunately error-prone. ## 2 Motivation @@ -18,7 +18,7 @@ Background ---------- Kubernetes allows users to specify both [requests and limits](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/). **Requests** are used to schedule pods onto nodes. **Limits** are hard stops that running containers are not permitted to exceed. -In the context of what a Flyte user can specify, flytekit [task decorators](https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.task.html#flytekit-task) permit setting both requests and limits. Furthermore, in their workflow definitions, users can specify node-level overrides which supersede static task definition resource values. +In the context of what a Flyte user can specify, flytekit [task decorators](https://docs.flyte.org/en/latest/api/flytekit/generated/flytekit.task.html#flytekit-task) permit setting both requests and limits. Furthermore, in their workflow definitions, users can specify node-level overrides which supersede static task definition resource values. In the Flyte back-end, **default** values can be applied as requests and limits when a user omits them from a task specification. Furthermore, **max** values are used to enforce that either user-specified resource requests or limits do not exceed a configured threshold. @@ -34,7 +34,7 @@ The `ExecutionConfig` already stores the [admin-resolved values](https://github. When building a task container, the default container task plugin will use user-specified resource values and verify they do not exceed the platform-supplied **max** values. When a request and/or limit for a specific resource (e.g. CPU, memory, etc) is not user-supplied the platform-supplied **default** values will be used. After this resolution is performed, the plugin handler will finally make sure that no resource requests exceed resource limits, leading to an impossible-to-schedule situation. ### K8s Pod Plugin Behavior -This will behave similarly to the default container plugin behavior. Platform-supplied **max** values will be enforced for *all* containers defined in a k8s pod spec for a task. However, platform-supplied **default** values will only be substituted for primary container tasks (see more [here](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/pod/pod.html#sphx-glr-auto-integrations-kubernetes-pod-pod-py)). +This will behave similarly to the default container plugin behavior. Platform-supplied **max** values will be enforced for *all* containers defined in a k8s pod spec for a task. However, platform-supplied **default** values will only be substituted for primary container tasks (see more [here](https://docs.flyte.org/en/latest/flytesnacks/auto/integrations/kubernetes/pod/pod.html#sphx-glr-auto-integrations-kubernetes-pod-pod-py)). diff --git a/rfc/system/1893-caching-of-offloaded-objects.md b/rfc/system/1893-caching-of-offloaded-objects.md index 9547cf33c9..c08bf450ba 100644 --- a/rfc/system/1893-caching-of-offloaded-objects.md +++ b/rfc/system/1893-caching-of-offloaded-objects.md @@ -6,7 +6,7 @@ ## 1 Executive Summary -We propose a way to override the default behavior of [caching task executions](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/task_cache.html), enabling cache-by-value semantics for certain categories of objects. +We propose a way to override the default behavior of [caching task executions](https://docs.flyte.org/en/latest/flytesnacks/auto/core/flyte_basics/task_cache.html), enabling cache-by-value semantics for certain categories of objects. ## 2 Motivation diff --git a/rfc/system/2633-eviction-of-cached-task-outputs.md b/rfc/system/2633-eviction-of-cached-task-outputs.md index b60fd56eff..04774d8cbd 100644 --- a/rfc/system/2633-eviction-of-cached-task-outputs.md +++ b/rfc/system/2633-eviction-of-cached-task-outputs.md @@ -70,7 +70,7 @@ Similar to the `interruptible` override currently available, we propose to add a #### `flytekit` -We should extend the [`flytekit.remote`](https://docs.flyte.org/projects/flytekit/en/latest/remote.html#remote-access) functionality to support setting the `skip_cache` flag for a single execution. +We should extend the [`flytekit.remote`](https://docs.flyte.org/en/latest/api/flytekit/remote.html#remote-access) functionality to support setting the `skip_cache` flag for a single execution. #### `flytectl` @@ -112,7 +112,7 @@ Whilst our main intent for this `AdminService` extension is for automated/script #### `flytekit` -[`flytekit.remote`](https://docs.flyte.org/projects/flytekit/en/latest/remote.html#remote-access) could be extended to support eviction of a task/workflow's cached results remotely. +[`flytekit.remote`](https://docs.flyte.org/en/latest/api/flytekit/remote.html#remote-access) could be extended to support eviction of a task/workflow's cached results remotely. #### `flytectl` @@ -158,7 +158,7 @@ The potential for malicious exploitation is deemed non-existent as no access to 3. Which Flyte tools (`flyteconsole`/`flytectl`) should support the proposed `AdminService` API extension for `flyteadmin`, if any? - **RESOLVED**: `flytectl`, `flytekit.remote`, `flyteconsole` 4. Should we support automatic eviction of cached results on workflow archival (opt-out via `flyteconsole`)? -5. Should we evict [Infratask Checkpoints](https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/checkpoint.html) from the cache as well since they might return cached results? If so, should we evict them from the backend side or pass the `cache_override` flag along to `flytekit`/its `Checkpointer` to skip any available entries? +5. Should we evict [Infratask Checkpoints](https://docs.flyte.org/en/latest/flytesnacks/auto/core/control_flow/checkpoint.html) from the cache as well since they might return cached results? If so, should we evict them from the backend side or pass the `cache_override` flag along to `flytekit`/its `Checkpointer` to skip any available entries? - **RESOLVED**: not for the initial implementation. Infratask checkpoints are only relevant for consecutive retries of a task - their results would not be considered when launching another execution with a `cache_override` flag set. ## 9 Conclusion diff --git a/rfc/system/3902-simplify-retry-behaviour.md b/rfc/system/3902-simplify-retry-behaviour.md index c15bc185b6..1e6bd832f7 100644 --- a/rfc/system/3902-simplify-retry-behaviour.md +++ b/rfc/system/3902-simplify-retry-behaviour.md @@ -10,7 +10,7 @@ Flyte implements a retry mechanism to make workflows robust against failure. This retry mechanism has two different budgets, one for which the user defines the maximum number of retries in the `@task` decorator and one for system failures, which is defined on the platform side. -> We distinguish between Flytekit/system-level exceptions and user exceptions. For instance, if Flytekit encounters an issue while uploading outputs, it is considered a system exception. On the other hand, if a user raises a ValueError due to an unexpected input in the task code, it is classified as a user exception. [(Source)](https://docs.flyte.org/projects/flytekit/en/latest/design/authoring.html#exception-handling) +> We distinguish between Flytekit/system-level exceptions and user exceptions. For instance, if Flytekit encounters an issue while uploading outputs, it is considered a system exception. On the other hand, if a user raises a ValueError due to an unexpected input in the task code, it is classified as a user exception. [(Source)](https://docs.flyte.org/en/latest/api/flytekit/design/authoring.html#exception-handling) Especially when it comes to interruptions/node terminations, the details of the retry behavior (which budget a retry counts against and how many retry possibilities are remaining) are intransparent and difficult to understand. The behavior is unfortunately also not consistent between plugins or even within the Pod plugin.