-
Notifications
You must be signed in to change notification settings - Fork 683
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'master' into 4291-separate-buckets
Signed-off-by: David Espejo <[email protected]>
- Loading branch information
Showing
249 changed files
with
965 additions
and
34,836 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| name: Validate helm charts & manifests | ||
|
|
||
| on: | ||
| pull_request: | ||
| branches: | ||
| - master | ||
| paths: | ||
| - deployment | ||
| - docker/sandbox-bundled/manifests | ||
|
|
||
| jobs: | ||
| lint-and-test-charts: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 0 | ||
| sparse-checkout: charts | ||
|
|
||
| - name: Install Helm | ||
| uses: azure/setup-helm@v4 | ||
|
|
||
| - name: Setup Python | ||
| uses: actions/setup-python@v4 | ||
| with: | ||
| python-version: "3.10" | ||
| check-latest: true | ||
|
|
||
| - name: Set up chart-testing | ||
| uses: helm/chart-testing-action@v2 | ||
|
|
||
| - name: Detect charts changed (list-changed) | ||
| id: charts-changed | ||
| run: | | ||
| changed=$(ct list-changed --target-branch ${{ github.event.repository.default_branch }}) | ||
| if [[ -n "$changed" ]]; then | ||
| echo "changed=true" >> "$GITHUB_OUTPUT" | ||
| fi | ||
| - name: Run chart-testing (lint) | ||
| if: steps.charts-changed.outputs.changed == 'true' | ||
| run: ct lint --target-branch ${{ github.event.repository.default_branch }} | ||
|
|
||
| - name: Create KinD cluster | ||
| if: steps.charts-changed.outputs.changed == 'true' | ||
| uses: helm/kind-action@v1 | ||
|
|
||
| - name: Run chart-testing (install) | ||
| if: steps.charts-changed.outputs.changed == 'true' | ||
| run: ct install --target-branch ${{ github.event.repository.default_branch }} | ||
|
|
||
| validate-manifests: | ||
| needs: | ||
| - lint-and-test-charts | ||
| runs-on: ubuntu-latest | ||
| defaults: | ||
| run: | ||
| shell: bash | ||
| strategy: | ||
| matrix: | ||
| k8s_versions: [ "1.29.2", "1.28.7", "1.27.11" ] | ||
|
|
||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| sparse-checkout: | | ||
| deployment | ||
| docker/sandbox-bundled/manifests | ||
| - name: Install Helm | ||
| uses: azure/setup-helm@v4 | ||
|
|
||
| - name: Install kubeconform | ||
| run: | | ||
| curl -L -o kubeconform.tar.gz https://github.com/yannh/kubeconform/releases/download/v0.6.4/kubeconform-linux-amd64.tar.gz | ||
| tar -zvxf kubeconform.tar.gz | ||
| chmod +x kubeconform | ||
| sudo mv kubeconform /usr/local/bin/kubeconform | ||
| - name: Validate manifests | ||
| run: | | ||
| kubeconform -strict -summary -skip CustomResourceDefinition -ignore-filename-pattern "deployment/stats/prometheus/*" -kubernetes-version ${{ matrix.k8s_versions }} ./deployment ./docker/sandbox-bundled/manifests |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| # Flyte v1.11.0 Release Notes | ||
|
|
||
| We're excited to announce the release of Flyte v1.11.0! This version brings a host of improvements, bug fixes, and new features designed to enhance your experience with Flyte. From operational enhancements to documentation updates, this release aims to make Flyte more robust, user-friendly, and feature-rich. | ||
|
|
||
| ## Highlights | ||
|
|
||
| - **Agents hit General Availability (GA):** Agents, now in General Availability, are long-running, stateless services that facilitate asynchronous job launches on platforms like Databricks or Snowflake and enable external service calls. They are versatile, supporting implementations in any language through a protobuf interface, enhancing Flyte's flexibility and operational efficiency. | ||
| - **Improved Caching:** Support for loading cached sublists with multiple data types has been introduced, eliminating issues related to cache retrieval across varied data formats. | ||
| - **Tracing and Observability:** The introduction of opentelemetry BlobstoreClientTracer in flyteadmin enhances observability, allowing for better monitoring and troubleshooting. | ||
| - **Security Enhancements:** Added securityContext configuration to Flyte-core charts, strengthening the security posture of Flyte deployments. | ||
| - **Documentation Overhaul:** Continuous improvements and updates have been made to the documentation, fixing broken links and updating content for better clarity and usability. | ||
| - **Operational Improvements:** This release introduces enhancements such as adding a service account for V1 Ray Jobs, caching console assets in a single binary, and conditional mounting of secrets to improve the operational efficiency of Flyte. Additionally, we are removing `kustomize` from our deployment process to simplify the configuration and management of Flyte instances, making it easier for users to maintain and streamline their deployment workflows. | ||
|
|
||
|
|
||
| ## Bug Fixes | ||
|
|
||
| - **Fixed Literal in Launchplan:** Added fixed_literal to the launchplan template, addressing issues with hardcoded values in workflows. | ||
| - **Corrected Metadata and Resources:** Fixes have been applied to correct IsParent metadata in ArrayNode eventing and to address invalid "resources" scope issues in deployment configurations. | ||
| - **Enhanced Stability and Performance:** Numerous bug fixes have been implemented to address stability and performance issues, including fixes for data catalog errors, yaml comment errors in pod template examples, and more. | ||
|
|
||
| ## Documentation and Guides | ||
|
|
||
| - **Comprehensive Guides:** New guides and documentation updates have been added, including a ChatGPT Agent Setup guide and an Airflow migration guide. Improvements in documentation for developing agents have been integrated into the broader enhancements for this release. | ||
| - **Updated Troubleshooting and Configuration Docs:** New troubleshooting guides for spark task execution and updates to deployment configuration documents enhance the knowledge base for Flyte users. | ||
|
|
||
| ## Contributors | ||
|
|
||
| We extend our deepest gratitude to all the contributors who made this release possible. Special shoutouts to @neilisaur, @lowc1012, @MortalHappiness, @novahow, and @pryce-turner for making their first contributions! | ||
|
|
||
| **For a full list of changes, enhancements, and bug fixes, visit our [changelog](https://github.com/flyteorg/flyte/compare/v1.10.7...v1.11.0).** | ||
|
|
||
| Thank you for your continued support of Flyte. We look forward to hearing your feedback on this release! |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -95,16 +95,16 @@ helm install gateway bitnami/contour -n flyte | |
| | configmap.clusters.clusterConfigs | list | `[]` | | | ||
| | configmap.clusters.labelClusterMap | object | `{}` | | | ||
| | configmap.console | object | `{"BASE_URL":"/console","CONFIG_DIR":"/etc/flyte/config"}` | Configuration for Flyte console UI | | ||
| | configmap.copilot | object | `{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b1","name":"flyte-copilot-","start-timeout":"30s"}}}}` | Copilot configuration | | ||
| | configmap.copilot.plugins.k8s.co-pilot | object | `{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.11.0-b1","name":"flyte-copilot-","start-timeout":"30s"}` | Structure documented [here](https://pkg.go.dev/github.com/lyft/[email protected]/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) | | ||
| | configmap.copilot | object | `{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.11.0","name":"flyte-copilot-","start-timeout":"30s"}}}}` | Copilot configuration | | ||
| | configmap.copilot.plugins.k8s.co-pilot | object | `{"image":"cr.flyte.org/flyteorg/flytecopilot:v1.11.0","name":"flyte-copilot-","start-timeout":"30s"}` | Structure documented [here](https://pkg.go.dev/github.com/lyft/[email protected]/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) | | ||
| | configmap.core | object | `{"manager":{"pod-application":"flytepropeller","pod-template-container-name":"flytepropeller","pod-template-name":"flytepropeller-template"},"propeller":{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"},"webhook":{"certDir":"/etc/webhook/certs","serviceName":"flyte-pod-webhook"}}` | Core propeller configuration | | ||
| | configmap.core.manager | object | `{"pod-application":"flytepropeller","pod-template-container-name":"flytepropeller","pod-template-name":"flytepropeller-template"}` | follows the structure specified [here](https://pkg.go.dev/github.com/flyteorg/flytepropeller/manager/config#Config). | | ||
| | configmap.core.propeller | object | `{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"}` | follows the structure specified [here](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/config). | | ||
| | configmap.datacatalogServer | object | `{"application":{"grpcPort":8089,"grpcServerReflection":true,"httpPort":8080},"datacatalog":{"heartbeat-grace-period-multiplier":3,"max-reservation-heartbeat":"30s","metrics-scope":"datacatalog","profiler-port":10254,"storage-prefix":"metadata/datacatalog"}}` | Datacatalog server config | | ||
| | configmap.domain | object | `{"domains":[{"id":"development","name":"development"},{"id":"staging","name":"staging"},{"id":"production","name":"production"}]}` | Domains configuration for Flyte projects. This enables the specified number of domains across all projects in Flyte. | | ||
| | configmap.enabled_plugins.tasks | object | `{"task-plugins":{"default-for-task-types":{"container":"container","container_array":"k8s-array","sidecar":"sidecar"},"enabled-plugins":["container","sidecar","k8s-array"]}}` | Tasks specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig) | | ||
| | configmap.enabled_plugins.tasks.task-plugins | object | `{"default-for-task-types":{"container":"container","container_array":"k8s-array","sidecar":"sidecar"},"enabled-plugins":["container","sidecar","k8s-array"]}` | Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig) | | ||
| | configmap.enabled_plugins.tasks.task-plugins.enabled-plugins | list | `["container","sidecar","k8s-array"]` | [Enabled Plugins](https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/config#Config). Enable sagemaker*, athena if you install the backend plugins | | ||
| | configmap.enabled_plugins.tasks | object | `{"task-plugins":{"default-for-task-types":{"container":"container","container_array":"k8s-array","sidecar":"sidecar"},"enabled-plugins":["container","sidecar","k8s-array","agent-service"]}}` | Tasks specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig) | | ||
| | configmap.enabled_plugins.tasks.task-plugins | object | `{"default-for-task-types":{"container":"container","container_array":"k8s-array","sidecar":"sidecar"},"enabled-plugins":["container","sidecar","k8s-array","agent-service"]}` | Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig) | | ||
| | configmap.enabled_plugins.tasks.task-plugins.enabled-plugins | list | `["container","sidecar","k8s-array","agent-service"]` | [Enabled Plugins](https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/config#Config). Enable sagemaker*, athena if you install the backend plugins | | ||
| | configmap.k8s | object | `{"plugins":{"k8s":{"default-cpus":"100m","default-env-vars":[],"default-memory":"100Mi"}}}` | Kubernetes specific Flyte configuration | | ||
| | configmap.k8s.plugins.k8s | object | `{"default-cpus":"100m","default-env-vars":[],"default-memory":"100Mi"}` | Configuration section for all K8s specific plugins [Configuration structure](https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/pluginmachinery/flytek8s/config) | | ||
| | configmap.remoteData.remoteData.region | string | `"us-east-1"` | | | ||
|
|
@@ -130,7 +130,7 @@ helm install gateway bitnami/contour -n flyte | |
| | datacatalog.extraArgs | object | `{}` | Appends extra command line arguments to the main command | | ||
| | datacatalog.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy | | ||
| | datacatalog.image.repository | string | `"cr.flyte.org/flyteorg/datacatalog"` | Docker image for Datacatalog deployment | | ||
| | datacatalog.image.tag | string | `"v1.11.0-b1"` | Docker image tag | | ||
| | datacatalog.image.tag | string | `"v1.11.0"` | Docker image tag | | ||
| | datacatalog.nodeSelector | object | `{}` | nodeSelector for Datacatalog deployment | | ||
| | datacatalog.podAnnotations | object | `{}` | Annotations for Datacatalog pods | | ||
| | datacatalog.podEnv | object | `{}` | Additional Datacatalog container environment variables | | ||
|
|
@@ -165,7 +165,7 @@ helm install gateway bitnami/contour -n flyte | |
| | flyteadmin.extraArgs | object | `{}` | Appends extra command line arguments to the serve command | | ||
| | flyteadmin.image.pullPolicy | string | `"IfNotPresent"` | | | ||
| | flyteadmin.image.repository | string | `"cr.flyte.org/flyteorg/flyteadmin"` | Docker image for Flyteadmin deployment | | ||
| | flyteadmin.image.tag | string | `"v1.11.0-b1"` | | | ||
| | flyteadmin.image.tag | string | `"v1.11.0"` | | | ||
| | flyteadmin.initialProjects | list | `["flytesnacks","flytetester","flyteexamples"]` | Initial projects to create | | ||
| | flyteadmin.nodeSelector | object | `{}` | nodeSelector for Flyteadmin deployment | | ||
| | flyteadmin.podAnnotations | object | `{}` | Annotations for Flyteadmin pods | | ||
|
|
@@ -202,7 +202,7 @@ helm install gateway bitnami/contour -n flyte | |
| | flyteconsole.ga.tracking_id | string | `"G-0QW4DJWJ20"` | | | ||
| | flyteconsole.image.pullPolicy | string | `"IfNotPresent"` | | | ||
| | flyteconsole.image.repository | string | `"cr.flyte.org/flyteorg/flyteconsole"` | Docker image for Flyteconsole deployment | | ||
| | flyteconsole.image.tag | string | `"v1.11.0"` | | | ||
| | flyteconsole.image.tag | string | `"v1.10.3"` | | | ||
| | flyteconsole.imagePullSecrets | list | `[]` | ImagePullSecrets to assign to the Flyteconsole deployment | | ||
| | flyteconsole.nodeSelector | object | `{}` | nodeSelector for Flyteconsole deployment | | ||
| | flyteconsole.podAnnotations | object | `{}` | Annotations for Flyteconsole pods | | ||
|
|
@@ -226,7 +226,7 @@ helm install gateway bitnami/contour -n flyte | |
| | flytepropeller.extraArgs | object | `{}` | Appends extra command line arguments to the main command | | ||
| | flytepropeller.image.pullPolicy | string | `"IfNotPresent"` | | | ||
| | flytepropeller.image.repository | string | `"cr.flyte.org/flyteorg/flytepropeller"` | Docker image for Flytepropeller deployment | | ||
| | flytepropeller.image.tag | string | `"v1.11.0-b1"` | | | ||
| | flytepropeller.image.tag | string | `"v1.11.0"` | | | ||
| | flytepropeller.manager | bool | `false` | | | ||
| | flytepropeller.nodeSelector | object | `{}` | nodeSelector for Flytepropeller deployment | | ||
| | flytepropeller.podAnnotations | object | `{}` | Annotations for Flytepropeller pods | | ||
|
|
@@ -256,7 +256,7 @@ helm install gateway bitnami/contour -n flyte | |
| | flytescheduler.configPath | string | `"/etc/flyte/config/*.yaml"` | Default regex string for searching configuration files | | ||
| | flytescheduler.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy | | ||
| | flytescheduler.image.repository | string | `"cr.flyte.org/flyteorg/flytescheduler"` | Docker image for Flytescheduler deployment | | ||
| | flytescheduler.image.tag | string | `"v1.11.0-b1"` | Docker image tag | | ||
| | flytescheduler.image.tag | string | `"v1.11.0"` | Docker image tag | | ||
| | flytescheduler.nodeSelector | object | `{}` | nodeSelector for Flytescheduler deployment | | ||
| | flytescheduler.podAnnotations | object | `{}` | Annotations for Flytescheduler pods | | ||
| | flytescheduler.podEnv | object | `{}` | Additional Flytescheduler container environment variables | | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.