diff --git a/.gitignore b/.gitignore index 81d776231a..7d35d48139 100644 --- a/.gitignore +++ b/.gitignore @@ -34,3 +34,5 @@ dist vendor/ /docker/sandbox-bundled/images/tar rsts/_tags/ +rsts/conf.py +rsts/community/contribute.rst diff --git a/rsts/conf.py b/rsts/conf.py index 7e41a81891..63c2990d27 100644 --- a/rsts/conf.py +++ b/rsts/conf.py @@ -94,7 +94,7 @@ # # This is also used if you do content translation via gettext catalogs. # Usually you set "language" from the command line for these cases. -language = None +language = 'en' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. @@ -130,8 +130,10 @@ "github_username": "flyteorg", "github_commit": "master", "docs_path": "rsts", # path to documentation source - "sphinx_gallery_src_dir": "cookbook", # path to directory of sphinx gallery source files relative to repo root - "sphinx_gallery_dest_dir": "auto", # path to root directory containing auto-generated sphinx gallery examples + # path to directory of sphinx gallery source files relative to repo root + "sphinx_gallery_src_dir": "cookbook", + # path to root directory containing auto-generated sphinx gallery examples + "sphinx_gallery_dest_dir": "auto", } # Theme options are theme-specific and customize the look and feel of a theme diff --git a/rsts/deployment/agents/bigquery.rst b/rsts/deployment/agents/bigquery.rst deleted file mode 100644 index ca2b704986..0000000000 --- a/rsts/deployment/agents/bigquery.rst +++ /dev/null @@ -1,90 +0,0 @@ -.. _deployment-agent-setup-bigquery: - -Google BigQuery Agent -====================== - -This guide provides an overview of setting up BigQuery agent in your Flyte deployment. -Please note that the BigQuery agent requires Flyte deployment in the GCP cloud; -it is not compatible with demo/AWS/Azure. - -Set up the GCP Flyte cluster ----------------------------- - -* Ensure you have a functional Flyte cluster running in `GCP `__. -* Create a service account for BigQuery. For more details, refer to: https://cloud.google.com/bigquery/docs/quickstarts/quickstart-client-libraries. -* Verify that you have the correct kubeconfig and have selected the appropriate Kubernetes context. -* Confirm that you have the correct Flytectl configuration at ``~/.flyte/config.yaml``. - -Specify agent configuration ----------------------------- - -.. tabs:: - - .. group-tab:: Flyte binary - - Edit the relevant YAML file to specify the agent. - - .. code-block:: yaml - :emphasize-lines: 7,11 - - tasks: - task-plugins: - enabled-plugins: - - container - - sidecar - - k8s-array - - bigquery - default-for-task-types: - - container: container - - container_array: k8s-array - - bigquery_query_job_task: agent-service - - .. group-tab:: Flyte core - - Create a file named ``values-override.yaml`` and add the following configuration to it. - - .. code-block:: yaml - - configmap: - enabled_plugins: - # -- Tasks specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig) - tasks: - # -- Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig) - task-plugins: - # -- [Enabled Plugins](https://pkg.go.dev/github.com/flyteorg/flyteplugins/go/tasks/config#Config). Enable sagemaker*, athena if you install the backend - enabled-plugins: - - container - - sidecar - - k8s-array - - bigquery - default-for-task-types: - container: container - sidecar: sidecar - container_array: k8s-array - bigquery_query_job_task: agent-service - -Ensure that the propeller has the correct service account for BigQuery. - -Upgrade the Flyte Helm release ------------------------------- - -.. tabs:: - - .. group-tab:: Flyte binary - - .. code-block:: bash - - helm upgrade flyteorg/flyte-binary -n --values - - Replace ```` with the name of your release (e.g., ``flyte-backend``), - ```` with the name of your namespace (e.g., ``flyte``), - and ```` with the name of your YAML file. - - .. group-tab:: Flyte core - - .. code-block:: bash - - helm upgrade flyte/flyte-core -n --values values-override.yaml - - Replace ```` with the name of your release (e.g., ``flyte``) - and ```` with the name of your namespace (e.g., ``flyte``). diff --git a/rsts/deployment/agents/index.rst b/rsts/deployment/agents/index.rst index 57232864a7..db6f758524 100644 --- a/rsts/deployment/agents/index.rst +++ b/rsts/deployment/agents/index.rst @@ -1,7 +1,7 @@ .. _deployment-agent-setup: Agent Setup -=========== +============== .. tags:: Agent, Integration, Data, Advanced @@ -17,10 +17,22 @@ Discover the process of setting up Agents for Flyte. :classes: btn-block stretched-link ^^^^^^^^^^^^ Guide to setting up the Bigquery agent. + + --- + + .. link-button:: deployment-agent-setup-memverge + :type: ref + :text: MMCloud Agent + :classes: btn-block stretched-link + ^^^^^^^^^^^^ + Guide to setting up the MMCloud agent. + .. toctree:: :maxdepth: 1 :name: Agent setup :hidden: - + bigquery + mmcloud + diff --git a/rsts/deployment/agents/mmcloud.rst b/rsts/deployment/agents/mmcloud.rst new file mode 100644 index 0000000000..ffaac4e4e5 --- /dev/null +++ b/rsts/deployment/agents/mmcloud.rst @@ -0,0 +1,119 @@ +.. _deployment-agent-setup-memverge: + +MMCloud Agent +============= + +MemVerge Memory Machine Cloud (MMCloud) empowers users to continuously optimize cloud resources during runtime, +safely execute stateful tasks on spot instances, +and monitor resource usage in real time. +These capabilities make it an excellent fit for long-running batch workloads. + +This guide provides an overview of how to set up MMCloud in your Flyte deployment. + +Set up MMCloud +-------------- + +To run a Flyte workflow with Memory Machine Cloud, you will need to deploy Memory Machine Cloud. +Check out the `MMCloud User Guide `_ to get started! + +By the end of this step, you should have deployed an MMCloud OpCenter. + +Spin up a cluster +----------------- + +.. tabs:: + + .. group-tab:: Flyte binary + + You can spin up a demo cluster using the following command: + + .. code-block:: bash + + flytectl demo start + + Or install Flyte using the :ref:`flyte-binary helm chart `. + + .. group-tab:: Flyte core + + If you've installed Flyte using the + `flyte-core helm chart `__, please ensure: + + * You have the correct kubeconfig and have selected the correct Kubernetes context. + * You have configured the correct flytectl settings in ``~/.flyte/config.yaml``. + +.. note:: + + Add the Flyte chart repo to Helm if you're installing via the Helm charts. + + .. code-block:: bash + + helm repo add flyteorg https://flyteorg.github.io/flyte + +Specify agent configuration +---------------------------- + +Enable the MMCloud agent by adding the following config to the relevant YAML file(s): + +.. code-block:: yaml + + tasks: + task-agents: + enabled-agents: + - agent-service + default-for-task-types: + - mmcloud_task: agent-service + +.. code-block:: yaml + + agents: + agent-service: + agents: + mmcloud-agent: + endpoint: + insecure: true + supportedTaskTypes: + - mmcloud_task + agentForTaskTypes: + - mmcloud_task: mmcloud-agent + +Substitute ```` with the endpoint of your MMCloud agent. + +Upgrade the deployment +---------------------- + +.. tabs:: + + .. group-tab:: Flyte binary + + .. tabs:: + + .. group-tab:: Demo cluster + + .. code-block:: bash + + kubectl rollout restart deployment flyte-sandbox -n flyte + + .. group-tab:: Helm chart + + .. code-block:: bash + + helm upgrade flyteorg/flyte-binary -n --values + + Replace ```` with the name of your release (e.g., ``flyte-backend``), + ```` with the name of your namespace (e.g., ``flyte``), + and ```` with the name of your YAML file. + + .. group-tab:: Flyte core + + .. code-block:: + + helm upgrade flyte/flyte-core -n --values values-override.yaml + + Replace ```` with the name of your release (e.g., ``flyte``) + and ```` with the name of your namespace (e.g., ``flyte``). + +Wait for the upgrade to complete. You can check the status of the deployment pods by running the following command: + +.. code-block:: + + kubectl get pods -n flyte diff --git a/rsts/deployment/plugins/index.rst b/rsts/deployment/plugins/index.rst index d83bd79c54..937c95365a 100644 --- a/rsts/deployment/plugins/index.rst +++ b/rsts/deployment/plugins/index.rst @@ -48,6 +48,7 @@ to use these plugins in your workflows written in ``flytekit``. ^^^^^^^^^^^^ Guide to setting up GCP-specific Plugins. + .. toctree:: :maxdepth: 1 :name: Plugin Setup