From cd422e09a5fd1e1e1afd0fea126d0256ec4cb3be Mon Sep 17 00:00:00 2001 From: DomFleischmann Date: Mon, 13 Mar 2023 17:21:24 +0100 Subject: [PATCH] Update instructions for Kustomize 5 (#2399) (#2408) * example: Update for kustomize 5 Add the sortOptions in the kustomization.yaml to ensure we are creating MutationWebhooks before Deployments and Pods. This way we'll ensure that Pods will get Istio sidecars in a race-free way. * readme: Update instructions to use Kustomize 5 Update instructions to inform users to use Kustomize 5. We'll also need to do an awk command, to remove a line that ends up in the final output. That is a regression on Kustomize's side. Once a newer version of Kustomize is available with the fix we'll then remove this awk command. * gh-action: Update to Kustomize 5 Update the helper script to use Kustomize 5 for the tests. * gh-action: Use the kustomize script in gh action Update the GH Action that runs the unit test to use the same script we use in the other GH Actions for fetching kustomize. * Add FAQ for earlier versions of Kustomize --------- Signed-off-by: Kimonas Sotirchos Co-authored-by: Kimonas Sotirchos --- .github/workflows/manifests_unittests.yaml | 8 ++---- README.md | 16 ++++++------ example/kustomization.yaml | 30 ++++++++++++++++++++++ tests/gh-actions/install_kustomize.sh | 7 ++--- 4 files changed, 44 insertions(+), 17 deletions(-) diff --git a/.github/workflows/manifests_unittests.yaml b/.github/workflows/manifests_unittests.yaml index 42b0e2e331..ad343478c9 100644 --- a/.github/workflows/manifests_unittests.yaml +++ b/.github/workflows/manifests_unittests.yaml @@ -13,12 +13,8 @@ jobs: - name: Check out repo uses: actions/checkout@v2 - - name: Install Kustomize - working-directory: ./tests/e2e - run: | - curl -Lo ./kustomize https://github.com/kubernetes-sigs/kustomize/releases/download/v3.2.0/kustomize_3.2.0_linux_amd64 - chmod +x ./kustomize - sudo mv kustomize /usr/local/bin + - name: Install kustomize + run: ./tests/gh-actions/install_kustomize.sh - name: Unit Test run: | diff --git a/README.md b/README.md index 2961a48271..7a3721b5fd 100644 --- a/README.md +++ b/README.md @@ -84,8 +84,8 @@ The `example` directory contains an example kustomization for the single command ### Prerequisites - `Kubernetes` (up to `1.25`) with a default [StorageClass](https://kubernetes.io/docs/concepts/storage/storage-classes/) -- `kustomize` (version [`3.2.0`](https://github.com/kubernetes-sigs/kustomize/releases/tag/v3.2.0) or [`4.5.7`](https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv4.5.7)) - - :warning: Kubeflow is compatible with Kustomize 4.5.7 only for [installing the individual components](#install-individual-components). The one-liner will need Kustomize 3.2, since versions from 4 and onwards have a regression on the order resources are sorted and printed. Please see [kubernetes-sigs/kustomize#3794](https://github.com/kubernetes-sigs/kustomize/issues/3794) and [kubeflow/manifests#1797](https://github.com/kubeflow/manifests/issues/1797). We know this is not ideal and are working with the upstream kustomize team to add support for the latest versions of kustomize as soon as we can. +- `kustomize` [5.0.0](https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv5.0.0) + - :warning: Kubeflow is not compatible with earlier versions of Kustomize. This is because we need the [`sortOptions`](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/sortoptions/) field, which is only available in Kustomize 5 and onwards https://github.com/kubeflow/manifests/issues/2388. - `kubectl` --- @@ -93,6 +93,8 @@ The `example` directory contains an example kustomization for the single command `kubectl apply` commands may fail on the first try. This is inherent in how Kubernetes and `kubectl` work (e.g., CR must be created after CRD becomes ready). The solution is to simply re-run the command until it succeeds. For the single-line command, we have included a bash one-liner to retry the command. +The reason we do `awk '!/well-defined/'` is because there's a regression in Kustomize 5 and a line is printed in stdout and not stderr https://github.com/kubernetes-sigs/kustomize/issues/5039. We'll remove this command once a future patch version of Kustomize is available. + --- ### Install with a single command @@ -100,7 +102,7 @@ The `example` directory contains an example kustomization for the single command You can install all Kubeflow official components (residing under `apps`) and all common services (residing under `common`) using the following command: ```sh -while ! kustomize build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done +while ! kustomize build example | awk '!/well-defined/' | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done ``` Once, everything is installed successfully, you can access the Kubeflow Central Dashboard [by logging in to your cluster](#connect-to-your-kubeflow-cluster). @@ -242,7 +244,7 @@ kustomize build common/istio-1-16/kubeflow-istio-resources/base | kubectl apply Install the [Multi-User Kubeflow Pipelines](https://www.kubeflow.org/docs/components/pipelines/multi-user/) official Kubeflow component: ```sh -kustomize build apps/pipeline/upstream/env/cert-manager/platform-agnostic-multi-user | kubectl apply -f - +kustomize build apps/pipeline/upstream/env/cert-manager/platform-agnostic-multi-user | awk '!/well-defined/' | kubectl apply -f - ``` This installs argo with the safe-to use runasnonroot emissary executor. Please note that the installer is still responsible to analyze the security issues that arise when containers are run with root access and to decide if the kubeflow pipeline main containers are run as runasnonroot. It is strongly recommended that the pipelines main containers are installed and run as runasnonroot and without any special capabilities to mitigate security risks. @@ -438,7 +440,5 @@ For security reasons, we don't want to use the default password for the default - **Q:** What versions of Istio, Knative, Cert-Manager, Argo, ... are compatible with Kubeflow? \ **A:** Please refer to each individual component's documentation for a dependency compatibility range. For Istio, Knative, Dex, Cert-Manager and OIDC-AuthService, the versions in `common` are the ones we have validated. - -- **Q:** Can I use the latest Kustomize version (`v4.x`)? \ - **A:** Kubeflow is compatible with Kustomize 4.5.7 only for [installing the individual components](#install-individual-components). The one-liner will need Kustomize 3.2, since versions from 4 and onwards have a regression on the order resources are sorted and printed. Please see [kubernetes-sigs/kustomize#3794](https://github.com/kubernetes-sigs/kustomize/issues/3794) and [kubeflow/manifests#1797](https://github.com/kubeflow/manifests/issues/1797). We know this is not ideal and are working with the upstream kustomize team to add support for the latest versions of kustomize as soon as we can. - +- **Q:** Can I use earlier version of Kustomize with Kubeflow manifests? + **A:** The manual installation instructions work with Kustomize 3.2. To use the one-liner installation you'll need to comment out the `sortOptions` section in the `example/kustomization.yaml`. diff --git a/example/kustomization.yaml b/example/kustomization.yaml index 172e9bf9e1..3ba71d71e5 100644 --- a/example/kustomization.yaml +++ b/example/kustomization.yaml @@ -1,6 +1,36 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization +sortOptions: + order: legacy + legacySortOptions: + orderFirst: + - Namespace + - ResourceQuota + - StorageClass + - CustomResourceDefinition + - MutatingWebhookConfiguration + - ServiceAccount + - PodSecurityPolicy + - Role + - ClusterRole + - RoleBinding + - ClusterRoleBinding + - ConfigMap + - Secret + - Endpoints + - Service + - LimitRange + - PriorityClass + - PersistentVolume + - PersistentVolumeClaim + - Deployment + - StatefulSet + - CronJob + - PodDisruptionBudget + orderLast: + - ValidatingWebhookConfiguration + resources: # Cert-Manager - ../common/cert-manager/cert-manager/base diff --git a/tests/gh-actions/install_kustomize.sh b/tests/gh-actions/install_kustomize.sh index 45196faf60..e82bf61011 100755 --- a/tests/gh-actions/install_kustomize.sh +++ b/tests/gh-actions/install_kustomize.sh @@ -1,5 +1,6 @@ #!/bin/bash set -e -curl --silent --location --remote-name "https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/v3.2.3/kustomize_kustomize.v3.2.3_linux_amd64" -chmod a+x kustomize_kustomize.v3.2.3_linux_amd64 -sudo mv kustomize_kustomize.v3.2.3_linux_amd64 /usr/local/bin/kustomize \ No newline at end of file +curl --silent --location --remote-name "https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv5.0.0/kustomize_v5.0.0_linux_amd64.tar.gz" +tar -xzvf kustomize_v5.0.0_linux_amd64.tar.gz +chmod a+x kustomize +sudo mv kustomize /usr/local/bin/kustomize