From 029b269e3919d85f883ce561087e51d1da9507d3 Mon Sep 17 00:00:00 2001 From: biswajit-9776 <115724497+biswajit-9776@users.noreply.github.com> Date: Mon, 22 Jul 2024 03:46:00 +0530 Subject: [PATCH 01/35] Documented about how to change default password in README.md (#2799) * Documented about how to change default password in README.md Signed-off-by: biswajit-9776 * Made requested changes Signed-off-by: biswajit-9776 --------- Signed-off-by: biswajit-9776 --- README.md | 44 ++++++++++++++++++++++++++++++++++++-------- 1 file changed, 36 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 3d0e4ca2c6..e431fb5f0d 100644 --- a/README.md +++ b/README.md @@ -465,25 +465,53 @@ If you absolutely need to expose Kubeflow over HTTP, you can disable the `Secure ### Change default user password -For security reasons, we don't want to use the default password for the default Kubeflow user when installing in security-sensitive environments. Instead, you should define your own password before deploying. To define a password for the default user: +For security reasons, we don't want to use the default password for the default Kubeflow user when installing in security-sensitive environments. Instead, you should define your own password and apply it either **before creating the cluster** or **after creating the cluster**. -1. Pick a password for the default user, with email `user@example.com`, and hash it using `bcrypt`: - -TODO this changed slightly in https://github.com/kubeflow/manifests/pull/2669 and https://github.com/kubeflow/manifests/pull/2229 +Pick a password for the default user, with email `user@example.com`, and hash it using `bcrypt`: ```sh python3 -c 'from passlib.hash import bcrypt; import getpass; print(bcrypt.using(rounds=12, ident="2y").hash(getpass.getpass()))' ``` -2. Edit `common/dex/base/config-map.yaml` and fill the relevant field with the hash of the password you chose: +For example, running the above command locally with required packages like _passlib_ would look as follows: + ```sh + python3 -c 'from passlib.hash import bcrypt; import getpass; print(bcrypt.using(rounds=12, ident="2y").hash(getpass.getpass()))' + Password: <--- Enter the password here + $2y$12$vIm8CANhuWui0J1p3jYeGeuM28Qcn76IFMaFWvZCG5ZkKZ4MjTF4u <--- GENERATED_HASH_FOR_ENTERED_PASSWORD + ``` + +#### Before creating the cluster: + +1. Edit `common/dex/base/dex-passwords.yaml` and fill the relevant field with the hash of the password you chose: ```yaml ... - staticPasswords: - - email: user@example.com - hash: + stringData: + DEX_USER_PASSWORD: ``` +#### After creating the cluster: + +1. Delete the existing secret _dex-passwords_ in auth namespace using the following command: + + ```sh + kubectl delete secret dex-passwords -n auth + ``` + +2. Create secret dex-passwords with new hash using the following command: + + ```sh + kubectl create secret generic dex-passwords --from-literal=DEX_USER_PASSWORD='REPLACE_WITH_HASH' -n auth + ``` + +3. Recreate the _dex_ pod in auth namespace using the following command: + + ```sh + kubectl delete pods --all -n auth + ``` + +4. Try to login using the new dex password. + ## Upgrading and extending For modifications and in place upgrades of the Kubeflow platform we provide a rough description for advanced users: From d5e1e012223b5b6fec7b2fd664b24a4c4b4389c5 Mon Sep 17 00:00:00 2001 From: Ricardo Martinelli de Oliveira Date: Mon, 22 Jul 2024 07:22:01 -0300 Subject: [PATCH 02/35] Update kubeflow/katib manifests from v0.17.0 (#2801) Signed-off-by: Ricardo M. Oliveira --- .../controller/trial-templates.yaml | 8 ++--- .../katib-cert-manager/katib-config.yaml | 30 +++++++++---------- .../katib-cert-manager/kustomization.yaml | 6 ++-- .../katib-external-db/katib-config.yaml | 30 +++++++++---------- .../katib-external-db/kustomization.yaml | 6 ++-- .../katib-leader-election/katib-config.yaml | 30 +++++++++---------- .../katib-openshift/katib-config.yaml | 30 +++++++++---------- .../katib-openshift/kustomization.yaml | 6 ++-- .../katib-config.yaml | 30 +++++++++---------- .../kustomization.yaml | 6 ++-- .../katib-standalone/katib-config.yaml | 30 +++++++++---------- .../katib-standalone/kustomization.yaml | 6 ++-- .../katib-with-kubeflow/kustomization.yaml | 6 ++-- 13 files changed, 112 insertions(+), 112 deletions(-) diff --git a/apps/katib/upstream/components/controller/trial-templates.yaml b/apps/katib/upstream/components/controller/trial-templates.yaml index 371d63849e..13193d2149 100644 --- a/apps/katib/upstream/components/controller/trial-templates.yaml +++ b/apps/katib/upstream/components/controller/trial-templates.yaml @@ -15,7 +15,7 @@ data: spec: containers: - name: training-container - image: docker.io/kubeflowkatib/pytorch-mnist-cpu:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/pytorch-mnist-cpu:v0.17.0 command: - "python3" - "/opt/pytorch-mnist/mnist.py" @@ -33,7 +33,7 @@ data: spec: containers: - name: training-container - image: docker.io/kubeflowkatib/enas-cnn-cifar10-cpu:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/enas-cnn-cifar10-cpu:v0.17.0 command: - python3 - -u @@ -54,7 +54,7 @@ data: spec: containers: - name: pytorch - image: docker.io/kubeflowkatib/pytorch-mnist-cpu:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/pytorch-mnist-cpu:v0.17.0 command: - "python3" - "/opt/pytorch-mnist/mnist.py" @@ -68,7 +68,7 @@ data: spec: containers: - name: pytorch - image: docker.io/kubeflowkatib/pytorch-mnist-cpu:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/pytorch-mnist-cpu:v0.17.0 command: - "python3" - "/opt/pytorch-mnist/mnist.py" diff --git a/apps/katib/upstream/installs/katib-cert-manager/katib-config.yaml b/apps/katib/upstream/installs/katib-cert-manager/katib-config.yaml index 419dc60ab6..08464d17ed 100644 --- a/apps/katib/upstream/installs/katib-cert-manager/katib-config.yaml +++ b/apps/katib/upstream/installs/katib-cert-manager/katib-config.yaml @@ -14,40 +14,40 @@ init: runtime: metricsCollectors: - kind: StdOut - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: File - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: TensorFlowEvent - image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0 resources: limits: memory: 1Gi suggestions: - algorithmName: random - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: tpe - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: grid - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: hyperband - image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0 - algorithmName: bayesianoptimization - image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0 - algorithmName: cmaes - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: sobol - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: multivariate-tpe - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: enas - image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0 resources: limits: memory: 400Mi - algorithmName: darts - image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0 - algorithmName: pbt - image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0 persistentVolumeClaimSpec: accessModes: - ReadWriteMany @@ -56,4 +56,4 @@ runtime: storage: 5Gi earlyStoppings: - algorithmName: medianstop - image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0 diff --git a/apps/katib/upstream/installs/katib-cert-manager/kustomization.yaml b/apps/katib/upstream/installs/katib-cert-manager/kustomization.yaml index 87d217ffe9..670b72d0b6 100644 --- a/apps/katib/upstream/installs/katib-cert-manager/kustomization.yaml +++ b/apps/katib/upstream/installs/katib-cert-manager/kustomization.yaml @@ -22,13 +22,13 @@ resources: images: - name: docker.io/kubeflowkatib/katib-controller newName: docker.io/kubeflowkatib/katib-controller - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-db-manager newName: docker.io/kubeflowkatib/katib-db-manager - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-ui newName: docker.io/kubeflowkatib/katib-ui - newTag: v0.17.0-rc.1 + newTag: v0.17.0 patchesStrategicMerge: - patches/katib-cert-injection.yaml diff --git a/apps/katib/upstream/installs/katib-external-db/katib-config.yaml b/apps/katib/upstream/installs/katib-external-db/katib-config.yaml index 7fc6860950..1773b0123f 100644 --- a/apps/katib/upstream/installs/katib-external-db/katib-config.yaml +++ b/apps/katib/upstream/installs/katib-external-db/katib-config.yaml @@ -16,40 +16,40 @@ init: runtime: metricsCollectors: - kind: StdOut - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: File - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: TensorFlowEvent - image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0 resources: limits: memory: 1Gi suggestions: - algorithmName: random - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: tpe - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: grid - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: hyperband - image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0 - algorithmName: bayesianoptimization - image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0 - algorithmName: cmaes - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: sobol - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: multivariate-tpe - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: enas - image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0 resources: limits: memory: 400Mi - algorithmName: darts - image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0 - algorithmName: pbt - image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0 persistentVolumeClaimSpec: accessModes: - ReadWriteMany @@ -58,4 +58,4 @@ runtime: storage: 5Gi earlyStoppings: - algorithmName: medianstop - image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0 diff --git a/apps/katib/upstream/installs/katib-external-db/kustomization.yaml b/apps/katib/upstream/installs/katib-external-db/kustomization.yaml index 05e239a78b..e8f9a95cdd 100644 --- a/apps/katib/upstream/installs/katib-external-db/kustomization.yaml +++ b/apps/katib/upstream/installs/katib-external-db/kustomization.yaml @@ -18,13 +18,13 @@ resources: images: - name: docker.io/kubeflowkatib/katib-controller newName: docker.io/kubeflowkatib/katib-controller - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-db-manager newName: docker.io/kubeflowkatib/katib-db-manager - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-ui newName: docker.io/kubeflowkatib/katib-ui - newTag: v0.17.0-rc.1 + newTag: v0.17.0 patchesStrategicMerge: - patches/db-manager.yaml # Modify katib-mysql-secrets with parameters for the DB. diff --git a/apps/katib/upstream/installs/katib-leader-election/katib-config.yaml b/apps/katib/upstream/installs/katib-leader-election/katib-config.yaml index 901e633adc..31d51bbf64 100644 --- a/apps/katib/upstream/installs/katib-leader-election/katib-config.yaml +++ b/apps/katib/upstream/installs/katib-leader-election/katib-config.yaml @@ -17,40 +17,40 @@ init: runtime: metricsCollectors: - kind: StdOut - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: File - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: TensorFlowEvent - image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0 resources: limits: memory: 1Gi suggestions: - algorithmName: random - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: tpe - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: grid - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: hyperband - image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0 - algorithmName: bayesianoptimization - image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0 - algorithmName: cmaes - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: sobol - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: multivariate-tpe - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: enas - image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0 resources: limits: memory: 400Mi - algorithmName: darts - image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0 - algorithmName: pbt - image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0 persistentVolumeClaimSpec: accessModes: - ReadWriteMany @@ -59,4 +59,4 @@ runtime: storage: 5Gi earlyStoppings: - algorithmName: medianstop - image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0 diff --git a/apps/katib/upstream/installs/katib-openshift/katib-config.yaml b/apps/katib/upstream/installs/katib-openshift/katib-config.yaml index 419dc60ab6..08464d17ed 100644 --- a/apps/katib/upstream/installs/katib-openshift/katib-config.yaml +++ b/apps/katib/upstream/installs/katib-openshift/katib-config.yaml @@ -14,40 +14,40 @@ init: runtime: metricsCollectors: - kind: StdOut - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: File - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: TensorFlowEvent - image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0 resources: limits: memory: 1Gi suggestions: - algorithmName: random - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: tpe - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: grid - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: hyperband - image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0 - algorithmName: bayesianoptimization - image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0 - algorithmName: cmaes - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: sobol - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: multivariate-tpe - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: enas - image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0 resources: limits: memory: 400Mi - algorithmName: darts - image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0 - algorithmName: pbt - image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0 persistentVolumeClaimSpec: accessModes: - ReadWriteMany @@ -56,4 +56,4 @@ runtime: storage: 5Gi earlyStoppings: - algorithmName: medianstop - image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0 diff --git a/apps/katib/upstream/installs/katib-openshift/kustomization.yaml b/apps/katib/upstream/installs/katib-openshift/kustomization.yaml index af40e9ac51..f98c9638a2 100644 --- a/apps/katib/upstream/installs/katib-openshift/kustomization.yaml +++ b/apps/katib/upstream/installs/katib-openshift/kustomization.yaml @@ -30,13 +30,13 @@ resources: images: - name: docker.io/kubeflowkatib/katib-controller newName: docker.io/kubeflowkatib/katib-controller - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-db-manager newName: docker.io/kubeflowkatib/katib-db-manager - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-ui newName: docker.io/kubeflowkatib/katib-ui - newTag: v0.17.0-rc.1 + newTag: v0.17.0 patchesJson6902: # Annotate Service to delegate TLS-secret generation to OpenShift service controller diff --git a/apps/katib/upstream/installs/katib-standalone-postgres/katib-config.yaml b/apps/katib/upstream/installs/katib-standalone-postgres/katib-config.yaml index 7fc6860950..1773b0123f 100644 --- a/apps/katib/upstream/installs/katib-standalone-postgres/katib-config.yaml +++ b/apps/katib/upstream/installs/katib-standalone-postgres/katib-config.yaml @@ -16,40 +16,40 @@ init: runtime: metricsCollectors: - kind: StdOut - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: File - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: TensorFlowEvent - image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0 resources: limits: memory: 1Gi suggestions: - algorithmName: random - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: tpe - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: grid - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: hyperband - image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0 - algorithmName: bayesianoptimization - image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0 - algorithmName: cmaes - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: sobol - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: multivariate-tpe - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: enas - image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0 resources: limits: memory: 400Mi - algorithmName: darts - image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0 - algorithmName: pbt - image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0 persistentVolumeClaimSpec: accessModes: - ReadWriteMany @@ -58,4 +58,4 @@ runtime: storage: 5Gi earlyStoppings: - algorithmName: medianstop - image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0 diff --git a/apps/katib/upstream/installs/katib-standalone-postgres/kustomization.yaml b/apps/katib/upstream/installs/katib-standalone-postgres/kustomization.yaml index 48054c5f44..fa70306aff 100644 --- a/apps/katib/upstream/installs/katib-standalone-postgres/kustomization.yaml +++ b/apps/katib/upstream/installs/katib-standalone-postgres/kustomization.yaml @@ -20,13 +20,13 @@ resources: images: - name: docker.io/kubeflowkatib/katib-controller newName: docker.io/kubeflowkatib/katib-controller - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-db-manager newName: docker.io/kubeflowkatib/katib-db-manager - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-ui newName: docker.io/kubeflowkatib/katib-ui - newTag: v0.17.0-rc.1 + newTag: v0.17.0 patchesJson6902: - target: group: apps diff --git a/apps/katib/upstream/installs/katib-standalone/katib-config.yaml b/apps/katib/upstream/installs/katib-standalone/katib-config.yaml index 7fc6860950..1773b0123f 100644 --- a/apps/katib/upstream/installs/katib-standalone/katib-config.yaml +++ b/apps/katib/upstream/installs/katib-standalone/katib-config.yaml @@ -16,40 +16,40 @@ init: runtime: metricsCollectors: - kind: StdOut - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: File - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: TensorFlowEvent - image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0 resources: limits: memory: 1Gi suggestions: - algorithmName: random - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: tpe - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: grid - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: hyperband - image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0 - algorithmName: bayesianoptimization - image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0 - algorithmName: cmaes - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: sobol - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: multivariate-tpe - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: enas - image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0 resources: limits: memory: 400Mi - algorithmName: darts - image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0 - algorithmName: pbt - image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0 persistentVolumeClaimSpec: accessModes: - ReadWriteMany @@ -58,4 +58,4 @@ runtime: storage: 5Gi earlyStoppings: - algorithmName: medianstop - image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0 diff --git a/apps/katib/upstream/installs/katib-standalone/kustomization.yaml b/apps/katib/upstream/installs/katib-standalone/kustomization.yaml index 24857c5733..146a5e6877 100644 --- a/apps/katib/upstream/installs/katib-standalone/kustomization.yaml +++ b/apps/katib/upstream/installs/katib-standalone/kustomization.yaml @@ -20,13 +20,13 @@ resources: images: - name: docker.io/kubeflowkatib/katib-controller newName: docker.io/kubeflowkatib/katib-controller - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-db-manager newName: docker.io/kubeflowkatib/katib-db-manager - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-ui newName: docker.io/kubeflowkatib/katib-ui - newTag: v0.17.0-rc.1 + newTag: v0.17.0 configMapGenerator: - name: katib-config behavior: create diff --git a/apps/katib/upstream/installs/katib-with-kubeflow/kustomization.yaml b/apps/katib/upstream/installs/katib-with-kubeflow/kustomization.yaml index 018c9ef43e..621651b15d 100644 --- a/apps/katib/upstream/installs/katib-with-kubeflow/kustomization.yaml +++ b/apps/katib/upstream/installs/katib-with-kubeflow/kustomization.yaml @@ -11,13 +11,13 @@ resources: images: - name: docker.io/kubeflowkatib/katib-controller newName: docker.io/kubeflowkatib/katib-controller - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-db-manager newName: docker.io/kubeflowkatib/katib-db-manager - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-ui newName: docker.io/kubeflowkatib/katib-ui - newTag: v0.17.0-rc.1 + newTag: v0.17.0 patchesStrategicMerge: - patches/remove-namespace.yaml From 858107b90a54987765acb68a889301f40b8f972b Mon Sep 17 00:00:00 2001 From: Ricardo Martinelli de Oliveira Date: Mon, 22 Jul 2024 07:23:00 -0300 Subject: [PATCH 03/35] Update kubeflow/training-operator manifests from v1.8.0 (#2802) Signed-off-by: Ricardo M. Oliveira --- README.md | 2 +- .../upstream/overlays/kubeflow/kustomization.yaml | 2 +- .../upstream/overlays/standalone/kustomization.yaml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index e431fb5f0d..d9131fbda0 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ This repo periodically syncs all official Kubeflow components from their respect | Component | Local Manifests Path | Upstream Revision | | - | - | - | -| Training Operator | apps/training-operator/upstream | [v1.8.0-rc.1](https://github.com/kubeflow/training-operator/tree/v1.8.0-rc.1/manifests) | +| Training Operator | apps/training-operator/upstream | [v1.8.0](https://github.com/kubeflow/training-operator/tree/v1.8.0/manifests) | | Notebook Controller | apps/jupyter/notebook-controller/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/notebook-controller/config) | | PVC Viewer Controller | apps/pvcviewer-roller/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/pvcviewer-controller/config) | | Tensorboard Controller | apps/tensorboard/tensorboard-controller/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/tensorboard-controller/config) | diff --git a/apps/training-operator/upstream/overlays/kubeflow/kustomization.yaml b/apps/training-operator/upstream/overlays/kubeflow/kustomization.yaml index 71ce5ef7b5..f4dabc273e 100644 --- a/apps/training-operator/upstream/overlays/kubeflow/kustomization.yaml +++ b/apps/training-operator/upstream/overlays/kubeflow/kustomization.yaml @@ -6,7 +6,7 @@ resources: - kubeflow-training-roles.yaml images: - name: kubeflow/training-operator - newTag: "v1-4485b0a" + newTag: "v1-9e52eb7" # TODO (tenzen-y): Once we support cert-manager, we need to remove this secret generation. # REF: https://github.com/kubeflow/training-operator/issues/2049 secretGenerator: diff --git a/apps/training-operator/upstream/overlays/standalone/kustomization.yaml b/apps/training-operator/upstream/overlays/standalone/kustomization.yaml index 0ecb165a23..0a6eb6ffbe 100644 --- a/apps/training-operator/upstream/overlays/standalone/kustomization.yaml +++ b/apps/training-operator/upstream/overlays/standalone/kustomization.yaml @@ -6,7 +6,7 @@ resources: - namespace.yaml images: - name: kubeflow/training-operator - newTag: "v1-4485b0a" + newTag: "v1-9e52eb7" secretGenerator: - name: training-operator-webhook-cert options: From c29b4deb3f08241626276a645c9ff99e0521aabd Mon Sep 17 00:00:00 2001 From: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com> Date: Mon, 22 Jul 2024 12:42:01 +0200 Subject: [PATCH 04/35] Synchronize training operator and katib manifests fixes (#2806) * fix the version numbers in the scripts Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * fix the katib directory Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * Update kubeflow/katib manifests from v0.17.0 Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --------- Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --- README.md | 2 +- hack/synchronize-katib-manifests.sh | 6 +++--- hack/synchronize-training-operator-manifests.sh | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index d9131fbda0..d686670719 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,7 @@ This repo periodically syncs all official Kubeflow components from their respect | Jupyter Web App | apps/jupyter/jupyter-web-app/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/crud-web-apps/jupyter/manifests) | | Tensorboards Web App | apps/tensorboard/tensorboards-web-app/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/crud-web-apps/tensorboards/manifests) | | Volumes Web App | apps/volumes-web-app/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/crud-web-apps/volumes/manifests) | -| Katib | apps/katib/upstream | [v0.17.0-rc.0](https://github.com/kubeflow/katib/tree/v0.17.0-rc.0/manifests/v1beta1) | +| Katib | apps/katib/upstream | [v0.17.0](https://github.com/kubeflow/katib/tree/v0.17.0/manifests/v1beta1) | | KServe | contrib/kserve/kserve | [0.13.0](https://github.com/kserve/kserve/releases/tag/v0.13.0) | | KServe Models Web App | contrib/kserve/models-web-app | [0.13.0-rc.0](https://github.com/kserve/models-web-app/tree/0.13.0-rc.0/config) | | Kubeflow Pipelines | apps/pipeline/upstream | [2.2.0](https://github.com/kubeflow/pipelines/tree/2.2.0/manifests/kustomize) | diff --git a/hack/synchronize-katib-manifests.sh b/hack/synchronize-katib-manifests.sh index d1cd5409c8..1edcdbaaa1 100644 --- a/hack/synchronize-katib-manifests.sh +++ b/hack/synchronize-katib-manifests.sh @@ -15,7 +15,7 @@ set -euxo pipefail IFS=$'\n\t' -COMMIT="v0.17.0-rc.0" # You can use tags as well +COMMIT="v0.17.0" # You can use tags as well SRC_DIR=${SRC_DIR:=/tmp/kubeflow-katib} BRANCH=${BRANCH:=synchronize-kubeflow-katib-manifests-${COMMIT?}} @@ -69,8 +69,8 @@ cp $SRC_DIR/katib/manifests/v1beta1 $DST_DIR -r echo "Successfully copied all manifests." echo "Updating README..." -SRC_TXT="\[.*\](https://github.com/kubeflow/katib/tree/.*/manifests/kustomize)" -DST_TXT="\[$COMMIT\](https://github.com/kubeflow/katib/tree/$COMMIT/manifests/kustomize)" +SRC_TXT="\[.*\](https://github.com/kubeflow/katib/tree/.*/manifests/v1beta1)" +DST_TXT="\[$COMMIT\](https://github.com/kubeflow/katib/tree/$COMMIT/manifests/v1beta1)" sed -i "s|$SRC_TXT|$DST_TXT|g" ${MANIFESTS_DIR}/README.md diff --git a/hack/synchronize-training-operator-manifests.sh b/hack/synchronize-training-operator-manifests.sh index 49f69080fb..4ea9aafb48 100644 --- a/hack/synchronize-training-operator-manifests.sh +++ b/hack/synchronize-training-operator-manifests.sh @@ -15,7 +15,7 @@ set -euxo pipefail IFS=$'\n\t' -COMMIT="v1.8.0-rc.1" # You can use tags as well +COMMIT="v1.8.0" # You can use tags as well SRC_DIR=${SRC_DIR:=/tmp/kubeflow-training-operator} BRANCH=${BRANCH:=synchronize-kubeflow-training-operator-manifests-${COMMIT?}} From 43a0e6ad69975c60a81d54deddeecd851b8eb808 Mon Sep 17 00:00:00 2001 From: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com> Date: Mon, 22 Jul 2024 13:22:01 +0200 Subject: [PATCH 05/35] Update kubeflow/kubeflow manifests from v1.9.0 (#2807) fix tags fix tags fix tags fix tags fix tags Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --- README.md | 18 +++++++++--------- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- apps/profiles/upstream/base/kustomization.yaml | 2 +- .../overlays/kubeflow/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- hack/synchronize-kubeflow-manifests.sh | 2 +- 12 files changed, 20 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index d686670719..b928b389f5 100644 --- a/README.md +++ b/README.md @@ -44,15 +44,15 @@ This repo periodically syncs all official Kubeflow components from their respect | Component | Local Manifests Path | Upstream Revision | | - | - | - | | Training Operator | apps/training-operator/upstream | [v1.8.0](https://github.com/kubeflow/training-operator/tree/v1.8.0/manifests) | -| Notebook Controller | apps/jupyter/notebook-controller/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/notebook-controller/config) | -| PVC Viewer Controller | apps/pvcviewer-roller/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/pvcviewer-controller/config) | -| Tensorboard Controller | apps/tensorboard/tensorboard-controller/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/tensorboard-controller/config) | -| Central Dashboard | apps/centraldashboard/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/centraldashboard/manifests) | -| Profiles + KFAM | apps/profiles/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/profile-controller/config) | -| PodDefaults Webhook | apps/admission-webhook/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/admission-webhook/manifests) | -| Jupyter Web App | apps/jupyter/jupyter-web-app/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/crud-web-apps/jupyter/manifests) | -| Tensorboards Web App | apps/tensorboard/tensorboards-web-app/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/crud-web-apps/tensorboards/manifests) | -| Volumes Web App | apps/volumes-web-app/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/crud-web-apps/volumes/manifests) | +| Notebook Controller | apps/jupyter/notebook-controller/upstream | [v1.9.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0/components/notebook-controller/config) | +| PVC Viewer Controller | apps/pvcviewer-roller/upstream | [v1.9.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0/components/pvcviewer-controller/config) | +| Tensorboard Controller | apps/tensorboard/tensorboard-controller/upstream | [v1.9.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0/components/tensorboard-controller/config) | +| Central Dashboard | apps/centraldashboard/upstream | [v1.9.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0/components/centraldashboard/manifests) | +| Profiles + KFAM | apps/profiles/upstream | [v1.9.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0/components/profile-controller/config) | +| PodDefaults Webhook | apps/admission-webhook/upstream | [v1.9.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0/components/admission-webhook/manifests) | +| Jupyter Web App | apps/jupyter/jupyter-web-app/upstream | [v1.9.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0/components/crud-web-apps/jupyter/manifests) | +| Tensorboards Web App | apps/tensorboard/tensorboards-web-app/upstream | [v1.9.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0/components/crud-web-apps/tensorboards/manifests) | +| Volumes Web App | apps/volumes-web-app/upstream | [v1.9.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0/components/crud-web-apps/volumes/manifests) | | Katib | apps/katib/upstream | [v0.17.0](https://github.com/kubeflow/katib/tree/v0.17.0/manifests/v1beta1) | | KServe | contrib/kserve/kserve | [0.13.0](https://github.com/kserve/kserve/releases/tag/v0.13.0) | | KServe Models Web App | contrib/kserve/models-web-app | [0.13.0-rc.0](https://github.com/kserve/models-web-app/tree/0.13.0-rc.0/config) | diff --git a/apps/admission-webhook/upstream/base/kustomization.yaml b/apps/admission-webhook/upstream/base/kustomization.yaml index 2b70cdaeb4..04fe85de8c 100644 --- a/apps/admission-webhook/upstream/base/kustomization.yaml +++ b/apps/admission-webhook/upstream/base/kustomization.yaml @@ -16,7 +16,7 @@ commonLabels: images: - name: docker.io/kubeflownotebookswg/poddefaults-webhook newName: docker.io/kubeflownotebookswg/poddefaults-webhook - newTag: v1.9.0-rc.2 + newTag: v1.9.0 namespace: kubeflow generatorOptions: disableNameSuffixHash: true diff --git a/apps/centraldashboard/upstream/base/kustomization.yaml b/apps/centraldashboard/upstream/base/kustomization.yaml index 99cdab21b1..08f70ab264 100644 --- a/apps/centraldashboard/upstream/base/kustomization.yaml +++ b/apps/centraldashboard/upstream/base/kustomization.yaml @@ -13,7 +13,7 @@ resources: images: - name: docker.io/kubeflownotebookswg/centraldashboard newName: docker.io/kubeflownotebookswg/centraldashboard - newTag: v1.9.0-rc.2 + newTag: v1.9.0 configMapGenerator: - envs: - params.env diff --git a/apps/jupyter/jupyter-web-app/upstream/base/kustomization.yaml b/apps/jupyter/jupyter-web-app/upstream/base/kustomization.yaml index c03654976b..ec62bdd747 100644 --- a/apps/jupyter/jupyter-web-app/upstream/base/kustomization.yaml +++ b/apps/jupyter/jupyter-web-app/upstream/base/kustomization.yaml @@ -23,7 +23,7 @@ commonLabels: images: - name: docker.io/kubeflownotebookswg/jupyter-web-app newName: docker.io/kubeflownotebookswg/jupyter-web-app - newTag: v1.9.0-rc.2 + newTag: v1.9.0 # We need the name to be unique without the suffix because the original name is what # gets used with patches configMapGenerator: diff --git a/apps/jupyter/notebook-controller/upstream/base/kustomization.yaml b/apps/jupyter/notebook-controller/upstream/base/kustomization.yaml index e1d4830ea2..a6b02fd248 100644 --- a/apps/jupyter/notebook-controller/upstream/base/kustomization.yaml +++ b/apps/jupyter/notebook-controller/upstream/base/kustomization.yaml @@ -5,4 +5,4 @@ resources: images: - name: docker.io/kubeflownotebookswg/notebook-controller newName: docker.io/kubeflownotebookswg/notebook-controller - newTag: v1.9.0-rc.2 + newTag: v1.9.0 diff --git a/apps/profiles/upstream/base/kustomization.yaml b/apps/profiles/upstream/base/kustomization.yaml index 0d29098f03..c6a15a0bf9 100644 --- a/apps/profiles/upstream/base/kustomization.yaml +++ b/apps/profiles/upstream/base/kustomization.yaml @@ -12,7 +12,7 @@ patchesStrategicMerge: images: - name: docker.io/kubeflownotebookswg/profile-controller newName: docker.io/kubeflownotebookswg/profile-controller - newTag: v1.9.0-rc.2 + newTag: v1.9.0 configMapGenerator: - name: namespace-labels-data diff --git a/apps/profiles/upstream/overlays/kubeflow/kustomization.yaml b/apps/profiles/upstream/overlays/kubeflow/kustomization.yaml index e68ee70492..1b8fba691c 100644 --- a/apps/profiles/upstream/overlays/kubeflow/kustomization.yaml +++ b/apps/profiles/upstream/overlays/kubeflow/kustomization.yaml @@ -29,4 +29,4 @@ vars: images: - name: docker.io/kubeflownotebookswg/kfam newName: docker.io/kubeflownotebookswg/kfam - newTag: v1.9.0-rc.2 + newTag: v1.9.0 diff --git a/apps/pvcviewer-controller/upstream/base/kustomization.yaml b/apps/pvcviewer-controller/upstream/base/kustomization.yaml index 4eb0ae99ac..259b701ef7 100644 --- a/apps/pvcviewer-controller/upstream/base/kustomization.yaml +++ b/apps/pvcviewer-controller/upstream/base/kustomization.yaml @@ -6,4 +6,4 @@ resources: images: - name: docker.io/kubeflownotebookswg/pvcviewer-controller newName: docker.io/kubeflownotebookswg/pvcviewer-controller - newTag: v1.9.0-rc.2 + newTag: v1.9.0 diff --git a/apps/tensorboard/tensorboard-controller/upstream/base/kustomization.yaml b/apps/tensorboard/tensorboard-controller/upstream/base/kustomization.yaml index c16af71f59..0a651adf84 100644 --- a/apps/tensorboard/tensorboard-controller/upstream/base/kustomization.yaml +++ b/apps/tensorboard/tensorboard-controller/upstream/base/kustomization.yaml @@ -14,4 +14,4 @@ patchesStrategicMerge: images: - name: docker.io/kubeflownotebookswg/tensorboard-controller newName: docker.io/kubeflownotebookswg/tensorboard-controller - newTag: v1.9.0-rc.2 + newTag: v1.9.0 diff --git a/apps/tensorboard/tensorboards-web-app/upstream/base/kustomization.yaml b/apps/tensorboard/tensorboards-web-app/upstream/base/kustomization.yaml index 3d6596f1f3..366638beec 100644 --- a/apps/tensorboard/tensorboards-web-app/upstream/base/kustomization.yaml +++ b/apps/tensorboard/tensorboards-web-app/upstream/base/kustomization.yaml @@ -14,7 +14,7 @@ commonLabels: images: - name: docker.io/kubeflownotebookswg/tensorboards-web-app newName: docker.io/kubeflownotebookswg/tensorboards-web-app - newTag: v1.9.0-rc.2 + newTag: v1.9.0 # We need the name to be unique without the suffix because the original name is what # gets used with patches configMapGenerator: diff --git a/apps/volumes-web-app/upstream/base/kustomization.yaml b/apps/volumes-web-app/upstream/base/kustomization.yaml index c2343baa1c..e81cdf755a 100644 --- a/apps/volumes-web-app/upstream/base/kustomization.yaml +++ b/apps/volumes-web-app/upstream/base/kustomization.yaml @@ -14,7 +14,7 @@ commonLabels: images: - name: docker.io/kubeflownotebookswg/volumes-web-app newName: docker.io/kubeflownotebookswg/volumes-web-app - newTag: v1.9.0-rc.2 + newTag: v1.9.0 # We need the name to be unique without the suffix because the original name is what # gets used with patches configMapGenerator: diff --git a/hack/synchronize-kubeflow-manifests.sh b/hack/synchronize-kubeflow-manifests.sh index 96c5154f47..71bbe3e4a8 100644 --- a/hack/synchronize-kubeflow-manifests.sh +++ b/hack/synchronize-kubeflow-manifests.sh @@ -14,7 +14,7 @@ set -euo pipefail IFS=$'\n\t' -COMMIT="v1.9.0-rc.2" # You can use tags as well +COMMIT="v1.9.0" # You can use tags as well SRC_DIR=${SRC_DIR:=/tmp/kubeflow-kubeflow} BRANCH=${BRANCH:=synchronize-kubeflow-kubeflow-manifests-${COMMIT?}} From a38c2be88fbafb0844c0231f0062e4b3719d4737 Mon Sep 17 00:00:00 2001 From: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com> Date: Mon, 22 Jul 2024 13:30:01 +0200 Subject: [PATCH 06/35] Update kserve models web application manifests from 0.13.0 (#2808) fix tags Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --- README.md | 2 +- hack/synchronize-kserve-web-app-manifests.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index b928b389f5..7b85df7d35 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,7 @@ This repo periodically syncs all official Kubeflow components from their respect | Volumes Web App | apps/volumes-web-app/upstream | [v1.9.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0/components/crud-web-apps/volumes/manifests) | | Katib | apps/katib/upstream | [v0.17.0](https://github.com/kubeflow/katib/tree/v0.17.0/manifests/v1beta1) | | KServe | contrib/kserve/kserve | [0.13.0](https://github.com/kserve/kserve/releases/tag/v0.13.0) | -| KServe Models Web App | contrib/kserve/models-web-app | [0.13.0-rc.0](https://github.com/kserve/models-web-app/tree/0.13.0-rc.0/config) | +| KServe Models Web App | contrib/kserve/models-web-app | [0.13.0](https://github.com/kserve/models-web-app/tree/0.13.0/config) | | Kubeflow Pipelines | apps/pipeline/upstream | [2.2.0](https://github.com/kubeflow/pipelines/tree/2.2.0/manifests/kustomize) | | Kubeflow Tekton Pipelines | apps/kfp-tekton/upstream | [2.0.5](https://github.com/kubeflow/kfp-tekton/tree/2.0.5/manifests/kustomize) | | Kubeflow Model Registry | apps/model-registry/upstream | [v0.2.1-alpha](https://github.com/kubeflow/model-registry/tree/v0.2.1-alpha/manifests/kustomize) | diff --git a/hack/synchronize-kserve-web-app-manifests.sh b/hack/synchronize-kserve-web-app-manifests.sh index beabded784..20954bd3cb 100644 --- a/hack/synchronize-kserve-web-app-manifests.sh +++ b/hack/synchronize-kserve-web-app-manifests.sh @@ -11,7 +11,7 @@ # repository, based on that local branch -COMMIT="0.13.0-rc.0" # You can use tags as well +COMMIT="0.13.0" # You can use tags as well SRC_DIR=${SRC_DIR:=/tmp/kserve-models-web-app} BRANCH=${BRANCH:=synchronize-kserve-web-app-manifests-${COMMIT?}} From c6099a9bc1cb1cf25e0b0689025cbb5478aa5091 Mon Sep 17 00:00:00 2001 From: Hansini Karunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Date: Mon, 22 Jul 2024 23:04:01 +0530 Subject: [PATCH 07/35] Proper message display when no bash, python and YAML files changed in the PR (#2803) * Proper message display when no bash files chnaged in the PR Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Proper message display when no Yaml files chnaged in the PR Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Proper message display when no Yaml files chnaged and styled in the PR Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Proper message display when no need to do style in Yaml files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Check the linting working by changing a yaml file Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Testing working of Bash linting Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Make some fix in formatting bash files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * checking functionalities of linting Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Combined Yaml,Bash and Python formatting yaml files into one workflow Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Deleted seperate files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Check the linting working by changing a yaml file Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Testing working of Bash linting Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * fix the mistake of changing unwanted file Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> --------- Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> --- .github/workflows/bash_formatter.yaml | 40 ----- .../linting_bash_python_yaml_files.yaml | 144 ++++++++++++++++++ .github/workflows/python_formatter.yaml | 32 ---- .github/workflows/yaml_formatter.yaml | 59 ------- README.md | 2 +- 5 files changed, 145 insertions(+), 132 deletions(-) delete mode 100644 .github/workflows/bash_formatter.yaml create mode 100644 .github/workflows/linting_bash_python_yaml_files.yaml delete mode 100644 .github/workflows/python_formatter.yaml delete mode 100644 .github/workflows/yaml_formatter.yaml diff --git a/.github/workflows/bash_formatter.yaml b/.github/workflows/bash_formatter.yaml deleted file mode 100644 index fd1d254bf6..0000000000 --- a/.github/workflows/bash_formatter.yaml +++ /dev/null @@ -1,40 +0,0 @@ -name: Proper Formatting on bash files - -on: [push, pull_request] - -jobs: - format_bash_files: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v3 - with: - fetch-depth: 0 - - - name: Install ShellCheck - run: sudo apt install -y shellcheck - - - name: Bash Formatting Guidelines - run: | - echo "### Bash Files Formatting Guidelines ### - If there are errors and warnings regarding your bash files, - You can check that error code definitions in https://www.shellcheck.net/wiki/ site. - You can correct them using the https://www.shellcheck.net/ site. - You have to ignore disable errors in .shellcheckrc file. - " - - - name: Fetch master branch - run: git fetch origin master - - - name: Set up changed files - run: | - git diff --name-only origin/master...HEAD | grep -E '^.*\.sh$' | grep -v '^apps/' > changed_files_in_PR.txt || true - - - name: Display changed files - run: cat changed_files_in_PR.txt - - - name: Run ShellCheck on changed files - run: | - cat changed_files_in_PR.txt | xargs -I {} shellcheck {} - shell: bash - - diff --git a/.github/workflows/linting_bash_python_yaml_files.yaml b/.github/workflows/linting_bash_python_yaml_files.yaml new file mode 100644 index 0000000000..33e5e6914c --- /dev/null +++ b/.github/workflows/linting_bash_python_yaml_files.yaml @@ -0,0 +1,144 @@ +name: Proper linting on Bash, Python, and YAML files + +on: [push, pull_request] + +jobs: + format_python_files: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + + - name: Python Files Formatting Guidelines + run: | + echo "### Python Files Formatting Guidelines ### + If there is a formatting error in your python files, + 1. First install black + It requires Python 3.8+ to run. + Install with 'pip install black' and if you use pipx, install Black with 'pipx install black'. + If you want to format Jupyter Notebooks, install with 'pip install black[jupyter]'. + + 2. Run the command + 'python -m black {source_file_or_directory}' or + 'black {source_file_or_directory}' + to format python files. + " + + - uses: psf/black@stable + with: + src: | + ./common + ./example + + format_YAML_files: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + with: + fetch-depth: 0 + + - name: Install yamllint + run: pip install yamllint + + - name: YAML Formatting Guidelines + run: | + echo "### YAML Formatting Guidelines ### + If there is a formatting error in your YAML file, you will see errors like the one below: + 'Error: 6:4 [indentation] wrong indentation: expected 2 but found 3' + + To fix these errors, refer to the YAML formatting rules at: + https://yamllint.readthedocs.io/en/stable/rules.html# + + Search for the keyword inside the brackets [] in the error message. In this example, it's 'indentation'. + + Note: Some rules have been customized in the '.yamllint.yaml' file. Below is the content of that file: + + extends: default + + rules: + document-start: + present: false + document-end: + present: false + indentation: + indent-sequences: false + line-length: + max: 400 + " + + - name: Fetch master branch + run: git fetch origin master + + - name: Set up changed files + id: changed_files + run: | + git diff --name-only origin/master...HEAD | grep -E '^common/.*\.ya?ml$|^example/.*\.ya?ml$' > changed_files_in_PR.txt || true + if [ ! -s changed_files_in_PR.txt ]; then + echo "No YAML files have changed in this PR." > changed_files_in_PR.txt + fi + + - name: Display changed files + run: cat changed_files_in_PR.txt + + - name: Run yamllint on changed files + id: lint + run: | + if grep -q 'No YAML files have changed in this PR.' changed_files_in_PR.txt; then + echo "No YAML files have changed in this PR." + else + cat changed_files_in_PR.txt | xargs -I {} yamllint {} || exit 1 + fi + shell: bash + + - name: Check YAML lint results + if: success() && steps.lint.outcome == 'success' + run: echo "No styling issues with YAML files." + shell: bash + + format_bash_files: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + with: + fetch-depth: 0 + + - name: Install ShellCheck + run: sudo apt install -y shellcheck + + - name: Bash Formatting Guidelines + run: | + echo "### Bash Files Formatting Guidelines ### + If there are errors and warnings regarding your bash files, + You can check the error code definitions at https://www.shellcheck.net/wiki/. + You can correct them using the https://www.shellcheck.net/ site. + You have to ignore disable errors in the .shellcheckrc file. + " + + - name: Fetch master branch + run: git fetch origin master + + - name: Set up changed files + id: changed_files + run: | + git diff --name-only origin/master...HEAD | grep -E '^.*\.sh$' | grep -v '^apps/' > changed_files_in_PR.txt || true + if [ ! -s changed_files_in_PR.txt ]; then + echo "No bash files have changed in this PR." + fi + + - name: Display changed files + if: always() # Always run this step + run: cat changed_files_in_PR.txt || echo "No bash files have changed in this PR." + + - name: Run ShellCheck on changed files + id: lint + run: | + if grep -q 'No bash files have changed in this PR.' changed_files_in_PR.txt; then + echo "No bash files have changed in this PR." + else + cat changed_files_in_PR.txt | xargs -I {} shellcheck {} || exit 1 + fi + shell: bash + + - name: Check Bash lint results + if: success() && steps.lint.outcome == 'success' + run: echo "No styling issues with Bash files." + shell: bash diff --git a/.github/workflows/python_formatter.yaml b/.github/workflows/python_formatter.yaml deleted file mode 100644 index ba470cad84..0000000000 --- a/.github/workflows/python_formatter.yaml +++ /dev/null @@ -1,32 +0,0 @@ -name: Proper Formatting on Python files - -on: [push, pull_request] - -jobs: - format_python_files: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v3 - - - name: Python Files Formatting Guidelines - run: | - echo "### Python Files Formatting Guidelines ### - If there is a formatting errors in your python files, - 1. First install black - It requires Python 3.8+ to run. - Install with "pip install black" and if you use pipx, install Black with "pipx install black" - If you want to format Jupyter Notebooks, install with pip install "black[jupyter]" - - 2. Run the command - "python -m black {source_file_or_directory}" or - "black {source_file_or_directory}" - to format python files. - " - - - uses: psf/black@stable - with: - src: | - ./common - ./example - - \ No newline at end of file diff --git a/.github/workflows/yaml_formatter.yaml b/.github/workflows/yaml_formatter.yaml deleted file mode 100644 index dca73409d4..0000000000 --- a/.github/workflows/yaml_formatter.yaml +++ /dev/null @@ -1,59 +0,0 @@ -name: Proper Formatting on YAML files - -on: [push, pull_request] - -jobs: - format_YAML_files: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v3 - with: - fetch-depth: 0 - - - name: Install yamllint - run: pip install yamllint - - - name: YAML Formatting Guidelines - run: | - echo "### YAML Formatting Guidelines ### - If there is a formatting error in your YAML file, you will see errors like the one below: - 'Error: 6:4 [indentation] wrong indentation: expected 2 but found 3' - - To fix these errors, refer to the YAML formatting rules at: - https://yamllint.readthedocs.io/en/stable/rules.html# - - Search for the keyword inside the brackets [] in the error message. In this example, it's 'indentation'. - - Note: Some rules have been customized in the '.yamllint.yaml' file. Below is the content of that file: - - extends: default - - rules: - document-start: - present: false - document-end: - present: false - indentation: - indent-sequences: false - line-length: - max: 400 - " - - - name: Fetch master branch - run: git fetch origin master - - - name: Set up changed files - run: | - git diff --name-only origin/master...HEAD | grep -E '^common/.*\.ya?ml$|^example/.*\.ya?ml$' > changed_files_in_PR.txt || true - - - name: Display changed files - run: cat changed_files_in_PR.txt - - - name: Run yamllint on changed files - run: | - chmod +x ./run_yamllint.sh - ./run_yamllint.sh - shell: bash - - - diff --git a/README.md b/README.md index 7b85df7d35..a94653e123 100644 --- a/README.md +++ b/README.md @@ -550,4 +550,4 @@ The Kubeflow security working group follows a responsible disclosure policy for - **Q:** What versions of Istio, Knative, Cert-Manager, Argo, ... are compatible with Kubeflow? \ **A:** Please refer to each individual component's documentation for a dependency compatibility range. For Istio, Knative, Dex, Cert-Manager and OIDC-AuthService, the versions in `common` are the ones we have validated. - **Q:** Can I use earlier version of Kustomize with Kubeflow manifests? - **A:** No, it is not supported anymore, although it might be possible with manual effort. + **A:** No, it is not supported anymore, although it might be possible with manual effort. \ No newline at end of file From ea91efc805389876aed611c0cdcee4db3646cf8b Mon Sep 17 00:00:00 2001 From: Hansini Karunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Date: Tue, 23 Jul 2024 18:05:02 +0530 Subject: [PATCH 08/35] Deprecate optional Knative Eventing components (#2788) Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> --- example/kustomization.yaml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/example/kustomization.yaml b/example/kustomization.yaml index e6a347652f..3591a8bd18 100644 --- a/example/kustomization.yaml +++ b/example/kustomization.yaml @@ -46,7 +46,8 @@ resources: - ../common/dex/overlays/oauth2-proxy # KNative - ../common/knative/knative-serving/overlays/gateways -- ../common/knative/knative-eventing/base +# Uncomment the following line if `knative-eventing` is required +# - ../common/knative/knative-eventing/base - ../common/istio-1-22/cluster-local-gateway/base # Kubeflow namespace - ../common/kubeflow-namespace/base From a0468b89b6d8d04d8d343dc9f0a81777688355d5 Mon Sep 17 00:00:00 2001 From: biswajit-9776 <115724497+biswajit-9776@users.noreply.github.com> Date: Tue, 23 Jul 2024 19:04:02 +0530 Subject: [PATCH 09/35] Added PSS profile seccompProfile to pods of istio (#2787) * Added PSS profile seccompProfile to pods of istio Signed-off-by: biswajit-9776 * Added seccompProfile to istio as kustomize component Signed-off-by: biswajit-9776 * Undone changes made to istio/base Signed-off-by: biswajit-9776 * Added seccomp files to common/istio-1-22/kubeflow-istio-resources/ Signed-off-by: biswajit-9776 * Moved the seccompProfile patches to istio/base/patches Signed-off-by: biswajit-9776 * Added newline character to a file Signed-off-by: biswajit-9776 --------- Signed-off-by: biswajit-9776 --- .../istio-install/base/kustomization.yaml | 2 ++ .../patches/seccomp-istio-ingressgateway.yaml | 16 ++++++++++++++++ .../base/patches/seccomp-istiod.yaml | 16 ++++++++++++++++ .../istio-install/base/kustomization.yaml | 2 ++ .../patches/seccomp-istio-ingressgateway.yaml | 16 ++++++++++++++++ .../base/patches/seccomp-istiod.yaml | 16 ++++++++++++++++ .../PSS/static/baseline/kustomization.yaml | 2 +- 7 files changed, 69 insertions(+), 1 deletion(-) create mode 100644 common/istio-1-22/istio-install/base/patches/seccomp-istio-ingressgateway.yaml create mode 100644 common/istio-1-22/istio-install/base/patches/seccomp-istiod.yaml create mode 100644 common/istio-cni-1-22/istio-install/base/patches/seccomp-istio-ingressgateway.yaml create mode 100644 common/istio-cni-1-22/istio-install/base/patches/seccomp-istiod.yaml diff --git a/common/istio-1-22/istio-install/base/kustomization.yaml b/common/istio-1-22/istio-install/base/kustomization.yaml index da19a0353e..fdb6319b10 100644 --- a/common/istio-1-22/istio-install/base/kustomization.yaml +++ b/common/istio-1-22/istio-install/base/kustomization.yaml @@ -16,3 +16,5 @@ patches: - path: patches/disable-debugging.yaml - path: patches/istio-ingressgateway-remove-pdb.yaml - path: patches/istiod-remove-pdb.yaml +- path: patches/seccomp-istio-ingressgateway.yaml +- path: patches/seccomp-istiod.yaml diff --git a/common/istio-1-22/istio-install/base/patches/seccomp-istio-ingressgateway.yaml b/common/istio-1-22/istio-install/base/patches/seccomp-istio-ingressgateway.yaml new file mode 100644 index 0000000000..f8825a0263 --- /dev/null +++ b/common/istio-1-22/istio-install/base/patches/seccomp-istio-ingressgateway.yaml @@ -0,0 +1,16 @@ +# Seccomp profile must be explicitly set to one of the allowed values. Both the Unconfined profile and the absence of a profile are prohibited. +# According to https://kubernetes.io/docs/concepts/security/pod-security-standards/#:~:text=undefined/null-,Seccomp%20(v1.19%2B),-Seccomp%20profile%20must +# This is done to enable 'restricted' level security standards for the pods. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: istio-ingressgateway + namespace: istio-system +spec: + template: + spec: + containers: + - name: istio-proxy + securityContext: + seccompProfile: + type: RuntimeDefault diff --git a/common/istio-1-22/istio-install/base/patches/seccomp-istiod.yaml b/common/istio-1-22/istio-install/base/patches/seccomp-istiod.yaml new file mode 100644 index 0000000000..3174889f85 --- /dev/null +++ b/common/istio-1-22/istio-install/base/patches/seccomp-istiod.yaml @@ -0,0 +1,16 @@ +# Seccomp profile must be explicitly set to one of the allowed values. Both the Unconfined profile and the absence of a profile are prohibited. +# According to https://kubernetes.io/docs/concepts/security/pod-security-standards/#:~:text=undefined/null-,Seccomp%20(v1.19%2B),-Seccomp%20profile%20must +# This is done to enable 'restricted' level security standards for the pods. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: istiod + namespace: istio-system +spec: + template: + spec: + containers: + - name: discovery + securityContext: + seccompProfile: + type: RuntimeDefault diff --git a/common/istio-cni-1-22/istio-install/base/kustomization.yaml b/common/istio-cni-1-22/istio-install/base/kustomization.yaml index 79d1fe5817..c3d6f9e06f 100644 --- a/common/istio-cni-1-22/istio-install/base/kustomization.yaml +++ b/common/istio-cni-1-22/istio-install/base/kustomization.yaml @@ -14,3 +14,5 @@ patches: - path: patches/disable-debugging.yaml - path: patches/istio-ingressgateway-remove-pdb.yaml - path: patches/istiod-remove-pdb.yaml +- path: patches/seccomp-istio-ingressgateway.yaml +- path: patches/seccomp-istiod.yaml diff --git a/common/istio-cni-1-22/istio-install/base/patches/seccomp-istio-ingressgateway.yaml b/common/istio-cni-1-22/istio-install/base/patches/seccomp-istio-ingressgateway.yaml new file mode 100644 index 0000000000..f8825a0263 --- /dev/null +++ b/common/istio-cni-1-22/istio-install/base/patches/seccomp-istio-ingressgateway.yaml @@ -0,0 +1,16 @@ +# Seccomp profile must be explicitly set to one of the allowed values. Both the Unconfined profile and the absence of a profile are prohibited. +# According to https://kubernetes.io/docs/concepts/security/pod-security-standards/#:~:text=undefined/null-,Seccomp%20(v1.19%2B),-Seccomp%20profile%20must +# This is done to enable 'restricted' level security standards for the pods. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: istio-ingressgateway + namespace: istio-system +spec: + template: + spec: + containers: + - name: istio-proxy + securityContext: + seccompProfile: + type: RuntimeDefault diff --git a/common/istio-cni-1-22/istio-install/base/patches/seccomp-istiod.yaml b/common/istio-cni-1-22/istio-install/base/patches/seccomp-istiod.yaml new file mode 100644 index 0000000000..3174889f85 --- /dev/null +++ b/common/istio-cni-1-22/istio-install/base/patches/seccomp-istiod.yaml @@ -0,0 +1,16 @@ +# Seccomp profile must be explicitly set to one of the allowed values. Both the Unconfined profile and the absence of a profile are prohibited. +# According to https://kubernetes.io/docs/concepts/security/pod-security-standards/#:~:text=undefined/null-,Seccomp%20(v1.19%2B),-Seccomp%20profile%20must +# This is done to enable 'restricted' level security standards for the pods. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: istiod + namespace: istio-system +spec: + template: + spec: + containers: + - name: discovery + securityContext: + seccompProfile: + type: RuntimeDefault diff --git a/contrib/security/PSS/static/baseline/kustomization.yaml b/contrib/security/PSS/static/baseline/kustomization.yaml index 3eea89f2c3..ec133f809c 100644 --- a/contrib/security/PSS/static/baseline/kustomization.yaml +++ b/contrib/security/PSS/static/baseline/kustomization.yaml @@ -6,4 +6,4 @@ patches: - path: patches/istio-labels.yaml - path: patches/cert-manager-labels.yaml - path: patches/dex-labels.yaml -- path: patches/oauth2-proxy-labels.yaml +- path: patches/oauth2-proxy-labels.yaml \ No newline at end of file From ace875bbed9e284483c545cdc1c2cb60ad5b24a3 Mon Sep 17 00:00:00 2001 From: biswajit-9776 <115724497+biswajit-9776@users.noreply.github.com> Date: Tue, 23 Jul 2024 19:21:02 +0530 Subject: [PATCH 10/35] PSS labels for the profile controller (#2778) * Patched PSS labels with profiles Signed-off-by: biswajit-9776 * Patched profiles/upstream/base/namespace-labels.yaml and updated example.yaml Signed-off-by: biswajit-9776 * Tried to override the base configMap Signed-off-by: biswajit-9776 * Made changes for overriding base configMap with kustomize component Signed-off-by: biswajit-9776 * Undone changes to profiles/upstream/base Signed-off-by: biswajit-9776 * Added comments for duplicated file Signed-off-by: biswajit-9776 * Resolved conflict Signed-off-by: biswajit-9776 * Fixed yaml lint to example Signed-off-by: biswajit-9776 --------- Signed-off-by: biswajit-9776 --- .../PSS/dynamic/baseline/kustomization.yaml | 9 ++++++++ .../dynamic/baseline/namespace-labels.yaml | 23 +++++++++++++++++++ .../PSS/dynamic/restricted/kustomization.yaml | 9 ++++++++ .../dynamic/restricted/namespace-labels.yaml | 23 +++++++++++++++++++ .../PSS/static/restricted/kustomization.yaml | 3 ++- example/kustomization.yaml | 7 +++++- 6 files changed, 72 insertions(+), 2 deletions(-) create mode 100644 contrib/security/PSS/dynamic/baseline/kustomization.yaml create mode 100644 contrib/security/PSS/dynamic/baseline/namespace-labels.yaml create mode 100644 contrib/security/PSS/dynamic/restricted/kustomization.yaml create mode 100644 contrib/security/PSS/dynamic/restricted/namespace-labels.yaml diff --git a/contrib/security/PSS/dynamic/baseline/kustomization.yaml b/contrib/security/PSS/dynamic/baseline/kustomization.yaml new file mode 100644 index 0000000000..8d8ea77e00 --- /dev/null +++ b/contrib/security/PSS/dynamic/baseline/kustomization.yaml @@ -0,0 +1,9 @@ +apiVersion: kustomize.config.k8s.io/v1alpha1 +kind: Component + +configMapGenerator: +- name: namespace-labels-data + namespace: kubeflow + behavior: merge + files: + - namespace-labels.yaml diff --git a/contrib/security/PSS/dynamic/baseline/namespace-labels.yaml b/contrib/security/PSS/dynamic/baseline/namespace-labels.yaml new file mode 100644 index 0000000000..08f6690272 --- /dev/null +++ b/contrib/security/PSS/dynamic/baseline/namespace-labels.yaml @@ -0,0 +1,23 @@ +# This file is a duplicate of apps/profiles/upstream/base/namespace-labels.yaml +# as using only the required label to merge it with the existing config map of profiles +# deployment to enable PSS for profile namespaces, leads to creation of a new config map +# with just the PSS label and replaces the pre-exisiting labels in the deployed config map. +# Below is a list of labels to be set by default. +# +# To add a namespace label, use `key: 'value'`, for example: +# istio.io/rev: 'asm-191-1' +# +# To remove a namespace label, use `key: ''`. For example: +# istio-injection: '' +# +# Profile controller will not replace a namespace label if its key already +# exists. If you want to override the value of a previously applied label, you +# need to: +# 1. Remove the label by using `key: ''` and deploy. +# 2. Add the label by using `key: 'value'` and deploy. +# +katib.kubeflow.org/metrics-collector-injection: "enabled" +serving.kubeflow.org/inferenceservice: "enabled" +pipelines.kubeflow.org/enabled: "true" +app.kubernetes.io/part-of: "kubeflow-profile" +pod-security.kubernetes.io/enforce: "baseline" diff --git a/contrib/security/PSS/dynamic/restricted/kustomization.yaml b/contrib/security/PSS/dynamic/restricted/kustomization.yaml new file mode 100644 index 0000000000..8d8ea77e00 --- /dev/null +++ b/contrib/security/PSS/dynamic/restricted/kustomization.yaml @@ -0,0 +1,9 @@ +apiVersion: kustomize.config.k8s.io/v1alpha1 +kind: Component + +configMapGenerator: +- name: namespace-labels-data + namespace: kubeflow + behavior: merge + files: + - namespace-labels.yaml diff --git a/contrib/security/PSS/dynamic/restricted/namespace-labels.yaml b/contrib/security/PSS/dynamic/restricted/namespace-labels.yaml new file mode 100644 index 0000000000..7d1fc7d114 --- /dev/null +++ b/contrib/security/PSS/dynamic/restricted/namespace-labels.yaml @@ -0,0 +1,23 @@ +# This file is a duplicate of apps/profiles/upstream/base/namespace-labels.yaml +# as using only the required label to merge it with the existing config map of profiles +# deployment to enable PSS for profile namespaces, leads to creation of a new config map +# with just the PSS label and replaces the pre-exisiting labels in the deployed config map. +# Below is a list of labels to be set by default. +# +# To add a namespace label, use `key: 'value'`, for example: +# istio.io/rev: 'asm-191-1' +# +# To remove a namespace label, use `key: ''`. For example: +# istio-injection: '' +# +# Profile controller will not replace a namespace label if its key already +# exists. If you want to override the value of a previously applied label, you +# need to: +# 1. Remove the label by using `key: ''` and deploy. +# 2. Add the label by using `key: 'value'` and deploy. +# +katib.kubeflow.org/metrics-collector-injection: "enabled" +serving.kubeflow.org/inferenceservice: "enabled" +pipelines.kubeflow.org/enabled: "true" +app.kubernetes.io/part-of: "kubeflow-profile" +pod-security.kubernetes.io/enforce: "restricted" diff --git a/contrib/security/PSS/static/restricted/kustomization.yaml b/contrib/security/PSS/static/restricted/kustomization.yaml index ec133f809c..f42ff9746b 100644 --- a/contrib/security/PSS/static/restricted/kustomization.yaml +++ b/contrib/security/PSS/static/restricted/kustomization.yaml @@ -6,4 +6,5 @@ patches: - path: patches/istio-labels.yaml - path: patches/cert-manager-labels.yaml - path: patches/dex-labels.yaml -- path: patches/oauth2-proxy-labels.yaml \ No newline at end of file +- path: patches/oauth2-proxy-labels.yaml +- path: patches/istio-labels.yaml diff --git a/example/kustomization.yaml b/example/kustomization.yaml index 3591a8bd18..3e028a16df 100644 --- a/example/kustomization.yaml +++ b/example/kustomization.yaml @@ -90,9 +90,14 @@ resources: - ../contrib/kserve/kserve - ../contrib/kserve/models-web-app/overlays/kubeflow +components: # Pod Security Standards # https://kubernetes.io/docs/concepts/security/pod-security-standards/ # Uncomment to enable baseline level standards # - ../contrib/security/PSS/static/baseline # Uncomment to enable restricted level standards -# - ../contrib/security/PSS/static/restricted \ No newline at end of file +# - ../contrib/security/PSS/static/restricted +# Uncomment to enable baseline level standards for dynamic namespaces +# - ../contrib/security/PSS/dynamic/baseline +# Uncomment to enable restricted level standards for dynamic namespaces +# - ../contrib/security/PSS/dynamic/restricted From 591349d20c5240934f6a70734381d417a7c47385 Mon Sep 17 00:00:00 2001 From: Hansini Karunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Date: Thu, 25 Jul 2024 15:11:54 +0530 Subject: [PATCH 11/35] Typographical changes (#2818) Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> --- .github/workflows/kserve_m2m_test.yaml | 2 +- .github/workflows/model_registry_test.yaml | 4 ++-- .github/workflows/notebook_controller_m2m_test.yaml | 2 +- .github/workflows/pipeline_run_from_notebook.yaml | 2 +- .github/workflows/pipeline_test.yaml | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/kserve_m2m_test.yaml b/.github/workflows/kserve_m2m_test.yaml index f1b484af4f..3b79215e57 100644 --- a/.github/workflows/kserve_m2m_test.yaml +++ b/.github/workflows/kserve_m2m_test.yaml @@ -38,7 +38,7 @@ jobs: - name: Create kubeflow namespace run: kustomize build common/kubeflow-namespace/base | kubectl apply -f - - - name: Install Istio with ext auth + - name: Install Istio with external authentication run: ./tests/gh-actions/install_istio_with_ext_auth.sh - name: Install cert-manager diff --git a/.github/workflows/model_registry_test.yaml b/.github/workflows/model_registry_test.yaml index b8f3ad847b..18345afb5f 100644 --- a/.github/workflows/model_registry_test.yaml +++ b/.github/workflows/model_registry_test.yaml @@ -32,7 +32,7 @@ jobs: - name: Create kubeflow namespace run: kustomize build common/kubeflow-namespace/base | kubectl apply -f - - - name: Install Istio with ext auth + - name: Install Istio with external authentication run: ./tests/gh-actions/install_istio_with_ext_auth.sh* - name: Install cert-manager @@ -61,7 +61,7 @@ jobs: 'http://localhost:8081/api/model_registry/v1alpha3/registered_models?pageSize=100&orderBy=ID&sortOrder=DESC' \ -H 'accept: application/json' - # for these steps below ensure same steps as kserve (ie: Istio with ext auth, cert0manager, knative) so to achieve same setup + # for these steps below ensure same steps as kserve (ie: Istio with ext external authentication, cert-manager, knative) so to achieve same setup - name: Port forward Istio gateway run: | INGRESS_GATEWAY_SERVICE=$(kubectl get svc --namespace istio-system --selector="app=istio-ingressgateway" --output jsonpath='{.items[0].metadata.name}') diff --git a/.github/workflows/notebook_controller_m2m_test.yaml b/.github/workflows/notebook_controller_m2m_test.yaml index 5a3688b039..8c49bf40a3 100644 --- a/.github/workflows/notebook_controller_m2m_test.yaml +++ b/.github/workflows/notebook_controller_m2m_test.yaml @@ -34,7 +34,7 @@ jobs: - name: Create kubeflow namespace run: kustomize build common/kubeflow-namespace/base | kubectl apply -f - - - name: Install Istio with ext auth + - name: Install Istio with external authentication run: ./tests/gh-actions/install_istio_with_ext_auth.sh - name: Install kubeflow-istio-resources diff --git a/.github/workflows/pipeline_run_from_notebook.yaml b/.github/workflows/pipeline_run_from_notebook.yaml index 52c80bd021..8e965bb375 100644 --- a/.github/workflows/pipeline_run_from_notebook.yaml +++ b/.github/workflows/pipeline_run_from_notebook.yaml @@ -32,7 +32,7 @@ jobs: - name: Install kustomize run: ./tests/gh-actions/install_kustomize.sh - - name: Install Istio with ext auth + - name: Install Istio with external authentication run: ./tests/gh-actions/install_istio_with_ext_auth.sh - name: Install cert-manager diff --git a/.github/workflows/pipeline_test.yaml b/.github/workflows/pipeline_test.yaml index 0fbf4b3e21..2df9ee79c0 100644 --- a/.github/workflows/pipeline_test.yaml +++ b/.github/workflows/pipeline_test.yaml @@ -33,7 +33,7 @@ jobs: - name: Install kubectl run: ./tests/gh-actions/install_kubectl.sh - - name: Install Istio with ext auth + - name: Install Istio with external authentication run: ./tests/gh-actions/install_istio_with_ext_auth.sh - name: Install cert-manager From 5ac0da52cb1b524cfed79a4287bae27b9c352826 Mon Sep 17 00:00:00 2001 From: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com> Date: Fri, 26 Jul 2024 15:56:55 +0200 Subject: [PATCH 12/35] Training operator CICD improvements (#2779) * Add the networkpolicies Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * rework the training operator tests Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * fix the comments Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * fix filename Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * try to fix the permissions Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * try to fix the permissions Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * change to the user namespace Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * update the image to rc.1 Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * fixes Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * fixes Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * fixes Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * fixes Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * fixes Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --------- Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --- .../linting_bash_python_yaml_files.yaml | 2 +- .github/workflows/model_registry_test.yaml | 2 +- .github/workflows/train_operator_test.yaml | 43 -------------- .github/workflows/training_operator_test.yaml | 57 +++++++++++++++++++ .../base/training-operator-webhook.yaml | 8 +-- tests/gh-actions/install_multi_tenancy.sh | 3 + tests/gh-actions/install_training_operator.sh | 9 +++ ...tebook.test.kubeflow-user-example.com.yaml | 2 +- tests/gh-actions/kf-objects/tfjob.yaml | 21 ------- .../kf-objects/training_operator_job.yaml | 40 +++++++++++++ 10 files changed, 116 insertions(+), 71 deletions(-) delete mode 100644 .github/workflows/train_operator_test.yaml create mode 100644 .github/workflows/training_operator_test.yaml create mode 100755 tests/gh-actions/install_training_operator.sh delete mode 100644 tests/gh-actions/kf-objects/tfjob.yaml create mode 100644 tests/gh-actions/kf-objects/training_operator_job.yaml diff --git a/.github/workflows/linting_bash_python_yaml_files.yaml b/.github/workflows/linting_bash_python_yaml_files.yaml index 33e5e6914c..2a7e1c6921 100644 --- a/.github/workflows/linting_bash_python_yaml_files.yaml +++ b/.github/workflows/linting_bash_python_yaml_files.yaml @@ -1,6 +1,6 @@ name: Proper linting on Bash, Python, and YAML files -on: [push, pull_request] +on: [pull_request] jobs: format_python_files: diff --git a/.github/workflows/model_registry_test.yaml b/.github/workflows/model_registry_test.yaml index 18345afb5f..082d9b540b 100644 --- a/.github/workflows/model_registry_test.yaml +++ b/.github/workflows/model_registry_test.yaml @@ -61,7 +61,7 @@ jobs: 'http://localhost:8081/api/model_registry/v1alpha3/registered_models?pageSize=100&orderBy=ID&sortOrder=DESC' \ -H 'accept: application/json' - # for these steps below ensure same steps as kserve (ie: Istio with ext external authentication, cert-manager, knative) so to achieve same setup + # for these steps below ensure same steps as kserve (ie: Istio with external authentication, cert-manager, knative) so to achieve same setup - name: Port forward Istio gateway run: | INGRESS_GATEWAY_SERVICE=$(kubectl get svc --namespace istio-system --selector="app=istio-ingressgateway" --output jsonpath='{.items[0].metadata.name}') diff --git a/.github/workflows/train_operator_test.yaml b/.github/workflows/train_operator_test.yaml deleted file mode 100644 index 23d891f627..0000000000 --- a/.github/workflows/train_operator_test.yaml +++ /dev/null @@ -1,43 +0,0 @@ -name: Build & Apply Training Operator manifests in KinD -on: - pull_request: - paths: - - .github/workflows/train_operator_test.yaml - - apps/training-operator/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - - tests/gh-actions/install_istio.sh - - common/istio*/** - - tests/gh-actions/kf-objects/tfjob.yaml - -jobs: - build: - runs-on: ubuntu-latest - steps: - - name: Checkout - uses: actions/checkout@v4 - - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh - - - name: Install Istio - run: ./tests/gh-actions/install_istio.sh - - - name: Build & Apply manifests - run: | - cd apps/training-operator/upstream - kubectl create ns kubeflow - kustomize build overlays/kubeflow | kubectl apply -f - - kubectl wait --for=condition=Ready pods --all --all-namespaces --timeout 180s - - - name: Create a TFJob - run: | - kubectl create -f tests/gh-actions/kf-objects/tfjob.yaml - kubectl wait --for=condition=Succeeded tfjobs tfjob-simple -n kubeflow --timeout 600s diff --git a/.github/workflows/training_operator_test.yaml b/.github/workflows/training_operator_test.yaml new file mode 100644 index 0000000000..5be7d5dffe --- /dev/null +++ b/.github/workflows/training_operator_test.yaml @@ -0,0 +1,57 @@ +name: Build & Apply Training Operator manifests in KinD +on: + pull_request: + paths: + - .github/workflows/training_operator_test.yaml + - apps/training-operator/upstream/** + - tests/gh-actions/kind-cluster.yaml + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh + - tests/gh-actions/install_istio.sh + - common/istio*/** + - tests/gh-actions/kf-objects/tfjob.yaml + +jobs: + build: + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Install KinD + run: ./tests/gh-actions/install_kind.sh + + - name: Create KinD Cluster + run: kind create cluster --config tests/gh-actions/kind-cluster.yaml + + - name: Install kustomize + run: ./tests/gh-actions/install_kustomize.sh + + - name: Install kubectl + run: ./tests/gh-actions/install_kubectl.sh + + - name: Install Istio with external authentication + run: ./tests/gh-actions/install_istio_with_ext_auth.sh + + - name: Install cert-manager + run: ./tests/gh-actions/install_cert_manager.sh + + - name: Create kubeflow namespace + run: kustomize build common/kubeflow-namespace/base | kubectl apply -f - + + - name: Install KF Multi Tenancy + run: ./tests/gh-actions/install_multi_tenancy.sh + + - name: Install kubeflow-istio-resources + run: kustomize build common/istio-1-22/kubeflow-istio-resources/base | kubectl apply -f - + + - name: Create KF Profile + run: kustomize build common/user-namespace/base | kubectl apply -f - + + - name: Install training operator + run: ./tests/gh-actions/install_training_operator.sh + + - name: Create a PyTorchJob + run: | + kubectl create -f tests/gh-actions/kf-objects/training_operator_job.yaml -n kubeflow-user-example-com + kubectl wait --for=condition=Succeeded PyTorchJob pytorch-simple -n kubeflow-user-example-com --timeout 600s diff --git a/common/networkpolicies/base/training-operator-webhook.yaml b/common/networkpolicies/base/training-operator-webhook.yaml index bbf6e373a3..8f9a6a7ea2 100644 --- a/common/networkpolicies/base/training-operator-webhook.yaml +++ b/common/networkpolicies/base/training-operator-webhook.yaml @@ -13,8 +13,8 @@ spec: # https://www.elastic.co/guide/en/cloud-on-k8s/1.1/k8s-webhook-network-policies.html # The kubernetes api server must reach the webhook ingress: - - ports: - - protocol: TCP - port: 9443 + - ports: + - protocol: TCP + port: 9443 policyTypes: - - Ingress \ No newline at end of file + - Ingress diff --git a/tests/gh-actions/install_multi_tenancy.sh b/tests/gh-actions/install_multi_tenancy.sh index ac9048ff68..29cc4d67a4 100755 --- a/tests/gh-actions/install_multi_tenancy.sh +++ b/tests/gh-actions/install_multi_tenancy.sh @@ -7,3 +7,6 @@ kubectl -n kubeflow wait --for=condition=Ready pods -l kustomize.component=profi echo "Installing Multitenancy Kubeflow Roles" kustomize build common/kubeflow-roles/base | kubectl apply -f - + +echo "Installing Multitenancy Network policies" +kustomize build common/networkpolicies/base | kubectl apply -f - diff --git a/tests/gh-actions/install_training_operator.sh b/tests/gh-actions/install_training_operator.sh new file mode 100755 index 0000000000..a0d50c1b1c --- /dev/null +++ b/tests/gh-actions/install_training_operator.sh @@ -0,0 +1,9 @@ +#!/bin/bash +set -euo pipefail +echo "Installing training operator ..." + +cd apps/training-operator/upstream +kustomize build overlays/kubeflow | kubectl apply -f - +kubectl wait --for=condition=Ready pods --all --all-namespaces --timeout=600s \ + --field-selector=status.phase!=Succeeded +cd - diff --git a/tests/gh-actions/kf-objects/notebook.test.kubeflow-user-example.com.yaml b/tests/gh-actions/kf-objects/notebook.test.kubeflow-user-example.com.yaml index 8e5ee2bb81..fdc8e53e0c 100644 --- a/tests/gh-actions/kf-objects/notebook.test.kubeflow-user-example.com.yaml +++ b/tests/gh-actions/kf-objects/notebook.test.kubeflow-user-example.com.yaml @@ -15,7 +15,7 @@ spec: spec: containers: - name: test - image: kubeflownotebookswg/jupyter-scipy:v1.9.0-rc.1 + image: kubeflownotebookswg/jupyter-scipy:v1.9.0 imagePullPolicy: IfNotPresent resources: limits: diff --git a/tests/gh-actions/kf-objects/tfjob.yaml b/tests/gh-actions/kf-objects/tfjob.yaml deleted file mode 100644 index 313c3312c0..0000000000 --- a/tests/gh-actions/kf-objects/tfjob.yaml +++ /dev/null @@ -1,21 +0,0 @@ -apiVersion: "kubeflow.org/v1" -kind: TFJob -metadata: - name: tfjob-simple - namespace: kubeflow -spec: - tfReplicaSpecs: - Worker: - replicas: 2 - restartPolicy: OnFailure - template: - spec: - containers: - - name: tensorflow - image: gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0 - command: - - "python" - - "/var/tf_mnist/mnist_with_summaries.py" - - "--log_dir=/train/logs" - - "--learning_rate=0.01" - - "--batch_size=150" \ No newline at end of file diff --git a/tests/gh-actions/kf-objects/training_operator_job.yaml b/tests/gh-actions/kf-objects/training_operator_job.yaml new file mode 100644 index 0000000000..68ee31c4be --- /dev/null +++ b/tests/gh-actions/kf-objects/training_operator_job.yaml @@ -0,0 +1,40 @@ +# from https://github.com/kubeflow/training-operator/blob/master/examples/pytorch/simple.yaml +# and disabled istio as stated in the documentation https://www.kubeflow.org/docs/components/training/user-guides/pytorch/ +apiVersion: "kubeflow.org/v1" +kind: PyTorchJob +metadata: + name: pytorch-simple +spec: + pytorchReplicaSpecs: + Master: + replicas: 1 + restartPolicy: OnFailure + template: + metadata: + annotations: + sidecar.istio.io/inject: "false" + spec: + containers: + - name: pytorch + image: docker.io/kubeflowkatib/pytorch-mnist:v1beta1-45c5727 + imagePullPolicy: Always + command: + - "python3" + - "/opt/pytorch-mnist/mnist.py" + - "--epochs=1" + Worker: + replicas: 1 + restartPolicy: OnFailure + template: + metadata: + annotations: + sidecar.istio.io/inject: "false" + spec: + containers: + - name: pytorch + image: docker.io/kubeflowkatib/pytorch-mnist:v1beta1-45c5727 + imagePullPolicy: Always + command: + - "python3" + - "/opt/pytorch-mnist/mnist.py" + - "--epochs=1" \ No newline at end of file From f226d3020f9f870f8532b566f8e249de03a6e45c Mon Sep 17 00:00:00 2001 From: biswajit-9776 <115724497+biswajit-9776@users.noreply.github.com> Date: Mon, 29 Jul 2024 20:12:58 +0530 Subject: [PATCH 13/35] Document how to change the default username for dex (#2810) Signed-off-by: biswajit-9776 --- README.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/README.md b/README.md index a94653e123..e88f320ed7 100644 --- a/README.md +++ b/README.md @@ -463,6 +463,19 @@ If you absolutely need to expose Kubeflow over HTTP, you can disable the `Secure --- +### Change default user name + +For security reasons, we don't want to use the default username and email for the default Kubeflow user when installing in security-sensitive environments. Instead, you should define your own username and email before deploying. To define it for the default user: + +1. Edit `common/dex/overlays/oauth2-proxy/config-map.yaml` and fill the relevant field with your email and preferred username: + + ```yaml + ... + staticPasswords: + - email: + username: + ``` + ### Change default user password For security reasons, we don't want to use the default password for the default Kubeflow user when installing in security-sensitive environments. Instead, you should define your own password and apply it either **before creating the cluster** or **after creating the cluster**. From 2d77438d62f5cc2607d4528dfb5e434fd334e182 Mon Sep 17 00:00:00 2001 From: Hansini Karunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Date: Tue, 30 Jul 2024 16:16:56 +0530 Subject: [PATCH 14/35] change ray version and enable auto scaling (#2821) * change ray version and enable auto scaling Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * fix a issue Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Add autoscaler options to ray Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> --------- Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> --- contrib/ray/raycluster_example.yaml | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/contrib/ray/raycluster_example.yaml b/contrib/ray/raycluster_example.yaml index c147a75a69..0de4047263 100644 --- a/contrib/ray/raycluster_example.yaml +++ b/contrib/ray/raycluster_example.yaml @@ -1,15 +1,27 @@ -apiVersion: ray.io/v1alpha1 +apiVersion: ray.io/v1 kind: RayCluster metadata: name: kubeflow-raycluster spec: rayVersion: '2.23.0' + # If `enableInTreeAutoscaling` is true, the Autoscaler sidecar will be added to the Ray head pod. + enableInTreeAutoscaling: true + # `autoscalerOptions` is an OPTIONAL field specifying configuration overrides for the Ray Autoscaler. + # The example configuration shown below below represents the DEFAULT values. + # (You may delete autoscalerOptions if the defaults are suitable.) + autoscalerOptions: + # Default: Upscaling is not rate-limited. This mode adds new worker pods to handle increased workload as quiclky as possible. + upscalingMode: Default + # `idleTimeoutSeconds` is the number of seconds to wait before scaling down a worker pod which is not using Ray resources. + idleTimeoutSeconds: 60 # Ray head pod configuration headGroupSpec: # Kubernetes Service Type. serviceType: ClusterIP # The following params are used to complete the ray start: ray start --head --block --dashboard-host: '0.0.0.0' ... rayStartParams: + # Setting "num-cpus: 0" to avoid any Ray actors or tasks being scheduled on the Ray head Pod. + num-cpus: "0" dashboard-host: '0.0.0.0' block: 'true' # pod template From 08f217c1bfb232ebeb05d12f9aa17d5f848c19bb Mon Sep 17 00:00:00 2001 From: Hansini Karunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Date: Tue, 30 Jul 2024 19:44:55 +0530 Subject: [PATCH 15/35] fix issue of trying to linting deleted files in PR (#2823) * fix issue of trying to linting deleted files in PR Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * fix a issue with linting in yaml Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * fix a issue with linting in yaml Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * fix a issue with linting in yaml trailing space errors Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * fix a issue with linting in yaml trailing space errors Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * fix a trailing issues in YAML files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * fix a trailing issues in YAML files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> --------- Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> --- .../linting_bash_python_yaml_files.yaml | 256 +++++++++--------- 1 file changed, 128 insertions(+), 128 deletions(-) diff --git a/.github/workflows/linting_bash_python_yaml_files.yaml b/.github/workflows/linting_bash_python_yaml_files.yaml index 2a7e1c6921..3d168fde5f 100644 --- a/.github/workflows/linting_bash_python_yaml_files.yaml +++ b/.github/workflows/linting_bash_python_yaml_files.yaml @@ -6,139 +6,139 @@ jobs: format_python_files: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 - - - name: Python Files Formatting Guidelines - run: | - echo "### Python Files Formatting Guidelines ### - If there is a formatting error in your python files, - 1. First install black - It requires Python 3.8+ to run. - Install with 'pip install black' and if you use pipx, install Black with 'pipx install black'. - If you want to format Jupyter Notebooks, install with 'pip install black[jupyter]'. - - 2. Run the command - 'python -m black {source_file_or_directory}' or - 'black {source_file_or_directory}' - to format python files. - " - - - uses: psf/black@stable - with: - src: | - ./common - ./example + - uses: actions/checkout@v3 + + - name: Python Files Formatting Guidelines + run: | + echo "### Python Files Formatting Guidelines ### + If there is a formatting error in your python files, + 1. First install black + It requires Python 3.8+ to run. + Install with 'pip install black' and if you use pipx, install Black with 'pipx install black'. + If you want to format Jupyter Notebooks, install with 'pip install black[jupyter]'. + + 2. Run the command + 'python -m black {source_file_or_directory}' or + 'black {source_file_or_directory}' + to format python files. + " + - uses: psf/black@stable + with: + src: | + ./common + ./example format_YAML_files: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 - with: - fetch-depth: 0 - - - name: Install yamllint - run: pip install yamllint - - - name: YAML Formatting Guidelines - run: | - echo "### YAML Formatting Guidelines ### - If there is a formatting error in your YAML file, you will see errors like the one below: - 'Error: 6:4 [indentation] wrong indentation: expected 2 but found 3' - - To fix these errors, refer to the YAML formatting rules at: - https://yamllint.readthedocs.io/en/stable/rules.html# - - Search for the keyword inside the brackets [] in the error message. In this example, it's 'indentation'. - - Note: Some rules have been customized in the '.yamllint.yaml' file. Below is the content of that file: - - extends: default - - rules: - document-start: - present: false - document-end: - present: false - indentation: - indent-sequences: false - line-length: - max: 400 - " - - - name: Fetch master branch - run: git fetch origin master - - - name: Set up changed files - id: changed_files - run: | - git diff --name-only origin/master...HEAD | grep -E '^common/.*\.ya?ml$|^example/.*\.ya?ml$' > changed_files_in_PR.txt || true - if [ ! -s changed_files_in_PR.txt ]; then - echo "No YAML files have changed in this PR." > changed_files_in_PR.txt - fi - - - name: Display changed files - run: cat changed_files_in_PR.txt - - - name: Run yamllint on changed files - id: lint - run: | - if grep -q 'No YAML files have changed in this PR.' changed_files_in_PR.txt; then - echo "No YAML files have changed in this PR." - else - cat changed_files_in_PR.txt | xargs -I {} yamllint {} || exit 1 - fi - shell: bash - - - name: Check YAML lint results - if: success() && steps.lint.outcome == 'success' - run: echo "No styling issues with YAML files." - shell: bash + - uses: actions/checkout@v3 + with: + fetch-depth: 0 + + - name: Install yamllint + run: pip install yamllint + + - name: YAML Formatting Guidelines + run: | + echo "### YAML Formatting Guidelines ### + If there is a formatting error in your YAML file, you will see errors like the one below: + 'Error: 6:4 [indentation] wrong indentation: expected 2 but found 3' + + 6:4 means line 6, column 4. + + To fix these errors, refer to the YAML formatting rules at: + https://yamllint.readthedocs.io/en/stable/rules.html# + + Search for the keyword inside the brackets [] in the error message. In this example, it's 'indentation'. + Note: Some rules have been customized in the '.yamllint.yaml' file. Below is the content of that file: + + extends: default + + rules: + document-start: + present: false + document-end: + present: false + indentation: + indent-sequences: false + line-length: + max: 400 + " + + - name: Fetch master branch + run: git fetch origin master + + - name: Set up changed files + id: changed_files + run: | + git diff --name-status origin/master...HEAD | grep -E '^[AM].*\.(yaml|yml)$' | awk '{print $2}' > changed_files_in_PR.txt || true + if [ ! -s changed_files_in_PR.txt ]; then + echo "No YAML files have changed in this PR." > changed_files_in_PR.txt + fi + + - name: Display changed files + run: cat changed_files_in_PR.txt + + - name: Run yamllint on changed files + id: lint + run: | + if grep -q 'No YAML files have changed in this PR.' changed_files_in_PR.txt; then + echo "No YAML files have changed in this PR." + else + cat changed_files_in_PR.txt | xargs -I {} yamllint {} || exit 1 + fi + shell: bash + + - name: Check YAML lint results + if: success() && steps.lint.outcome == 'success' + run: echo "No styling issues with YAML files." + shell: bash format_bash_files: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 - with: - fetch-depth: 0 - - - name: Install ShellCheck - run: sudo apt install -y shellcheck - - - name: Bash Formatting Guidelines - run: | - echo "### Bash Files Formatting Guidelines ### - If there are errors and warnings regarding your bash files, - You can check the error code definitions at https://www.shellcheck.net/wiki/. - You can correct them using the https://www.shellcheck.net/ site. - You have to ignore disable errors in the .shellcheckrc file. - " - - - name: Fetch master branch - run: git fetch origin master - - - name: Set up changed files - id: changed_files - run: | - git diff --name-only origin/master...HEAD | grep -E '^.*\.sh$' | grep -v '^apps/' > changed_files_in_PR.txt || true - if [ ! -s changed_files_in_PR.txt ]; then - echo "No bash files have changed in this PR." - fi - - - name: Display changed files - if: always() # Always run this step - run: cat changed_files_in_PR.txt || echo "No bash files have changed in this PR." - - - name: Run ShellCheck on changed files - id: lint - run: | - if grep -q 'No bash files have changed in this PR.' changed_files_in_PR.txt; then - echo "No bash files have changed in this PR." - else - cat changed_files_in_PR.txt | xargs -I {} shellcheck {} || exit 1 - fi - shell: bash - - - name: Check Bash lint results - if: success() && steps.lint.outcome == 'success' - run: echo "No styling issues with Bash files." - shell: bash + - uses: actions/checkout@v3 + with: + fetch-depth: 0 + + - name: Install ShellCheck + run: sudo apt install -y shellcheck + + - name: Bash Formatting Guidelines + run: | + echo "### Bash Files Formatting Guidelines ### + If there are errors and warnings regarding your bash files, + You can check the error code definitions at https://www.shellcheck.net/wiki/. + You can correct them using the https://www.shellcheck.net/ site. + You have to ignore disable errors in the .shellcheckrc file. + " + + - name: Fetch master branch + run: git fetch origin master + + - name: Set up changed files + id: changed_files + run: | + git diff --name-only origin/master...HEAD | grep -E '^[AM].*\.sh$' | grep -v '^apps/' | awk '{print $2}' > changed_files_in_PR.txt || true + if [ ! -s changed_files_in_PR.txt ]; then + echo "No bash files have changed in this PR." + fi + + - name: Display changed files + if: always() # Always run this step + run: cat changed_files_in_PR.txt || echo "No bash files have changed in this PR." + + - name: Run ShellCheck on changed files + id: lint + run: | + if grep -q 'No bash files have changed in this PR.' changed_files_in_PR.txt; then + echo "No bash files have changed in this PR." + else + cat changed_files_in_PR.txt | xargs -I {} shellcheck {} || exit 1 + fi + shell: bash + + - name: Check Bash lint results + if: success() && steps.lint.outcome == 'success' + run: echo "No styling issues with Bash files." + shell: bash From 1c464be6f5e13fbd4ed51406f74919e61619a018 Mon Sep 17 00:00:00 2001 From: Tom Zaspel <40226087+tzabbi@users.noreply.github.com> Date: Thu, 1 Aug 2024 11:00:06 +0200 Subject: [PATCH 16/35] Adding tests for dex plus restructuring and formatting (#2815) * fix redirectURIs for in dex config since kubeflow uses oauth2 Signed-off-by: Tom Zaspel * remove authservice and refactore code Signed-off-by: Tom Zaspel * remove authservice from github actions and refactore code Signed-off-by: Tom Zaspel * adjust name of oauth2 in FAQ Signed-off-by: Tom Zaspel * create new directory in central dashboard called manuel-patches for oauth2-proxy Signed-off-by: Tom Zaspel * change manuel-patches to overlay Signed-off-by: Tom Zaspel * Test if user can login to dex Signed-off-by: Tom Zaspel * Add kubeflow authentication oidc-authservice README.md Signed-off-by: Tom Zaspel <40226087+tzabbi@users.noreply.github.com> * Fix yaml linter error Signed-off-by: Tom Zaspel <40226087+tzabbi@users.noreply.github.com> * Fix yaml linting Signed-off-by: Tom Zaspel <40226087+tzabbi@users.noreply.github.com> * Add changes from comit: 08f217c1bfb232ebeb05d12f9aa17d5f848c19bb Signed-off-by: Tom Zaspel <40226087+tzabbi@users.noreply.github.com> * Fix even more lint issues Signed-off-by: Tom Zaspel --------- Signed-off-by: Tom Zaspel Signed-off-by: Tom Zaspel <40226087+tzabbi@users.noreply.github.com> Co-authored-by: Hansini Karunarathne <107214435+hansinikarunarathne@users.noreply.github.com> --- .github/workflows/admission_webhook_test.yaml | 16 +- .github/workflows/bentoml_test.yaml | 14 +- .github/workflows/centraldashboard_test.yaml | 12 +- .github/workflows/dex_test.yaml | 23 +- .../jupyter_web_application_test.yaml | 12 +- .github/workflows/katib_test.yaml | 16 +- .github/workflows/kserve_cni_test.yaml | 94 +++---- .github/workflows/kserve_m2m_test.yaml | 128 +++++----- .github/workflows/kserve_test.yaml | 94 +++---- .../linting_bash_python_yaml_files.yaml | 2 +- .github/workflows/manifests_example_test.yaml | 5 +- .github/workflows/metacontroller_test.yaml | 18 +- .github/workflows/model_registry_test.yaml | 14 +- .../notebook_controller_m2m_test.yaml | 18 +- .../workflows/notebook_controller_test.yaml | 14 +- .../workflows/pipeline_run_from_notebook.yaml | 26 +- .github/workflows/pipeline_test.yaml | 234 +++++++++--------- .github/workflows/profiles_test.yaml | 14 +- .github/workflows/ray_test.yaml | 8 +- .github/workflows/seldon_test.yaml | 18 +- .github/workflows/stale.yaml | 68 +++-- .../tensorboard_controller_test.yaml | 14 +- .../tensorboards_web_application_test.yaml | 14 +- .github/workflows/training_operator_test.yaml | 16 +- .github/workflows/triage_issues.yaml | 15 +- .github/workflows/trivy.yaml | 78 +++--- .../volumes_web_application_test.yaml | 14 +- README.md | 6 +- .../overlays/oauth2-proxy/kustomization.yaml | 4 +- common/dex/base/config-map.yaml | 2 +- .../overlays/oauth2-proxy/kustomization.yaml | 2 +- .../overlays/oauth2-proxy/kustomization.yaml | 2 +- common/{oidc-client => }/oauth2-proxy/OWNERS | 0 .../{oidc-client => oauth2-proxy}/README.md | 4 +- .../oauth2-proxy/base/README.md | 0 common/oauth2-proxy/base/deployment.yaml | 85 +++++++ .../oauth2-proxy/base/kustomization.yaml | 2 +- .../oauth2-proxy/base/namespace.yaml | 0 .../base/oauth2-proxy-config.yaml | 0 .../oauth2-proxy/base/params.yaml | 0 .../oauth2-proxy/base/rbac.tokenreviews.yaml | 16 +- .../oauth2-proxy/base/service.yaml | 6 +- .../oauth2-proxy/base/serviceaccount.yaml | 0 .../components}/README.md | 12 +- .../README.md | 0 ...lebinding.unauthenticated-oidc-viewer.yaml | 0 .../kustomization.yaml | 0 .../central-dashboard/kustomization.yaml | 0 .../patches/deployment.logout-url.yaml | 4 +- .../README.md | 0 ...ronjob.kubeflow-m2m-oidc-configurator.yaml | 39 +++ .../kustomization.yaml | 0 .../rbac.yaml | 20 +- .../script.sh | 0 .../istio-external-auth-patches}/README.md | 0 .../kustomization.yaml | 0 .../patches/cm.enable-oauth2-proxy.yaml | 0 .../deployment.jwt-refresh-interval.yaml | 18 ++ .../components/istio-external-auth}/README.md | 0 ...ngressgateway-oauth2-proxy-cloudflare.yaml | 0 ...icy.istio-ingressgateway-oauth2-proxy.yaml | 0 .../istio-external-auth/kustomization.yaml | 4 +- .../requestauthentication.dex-jwt.yaml | 19 ++ .../components/istio-m2m/README.md | 6 +- .../components/istio-m2m/kustomization.yaml | 0 .../components/istio-m2m/params.yaml | 0 .../istio-m2m/requestauthentication.yaml | 14 ++ .../components}/kubeflow_auth_diagram.svg | 0 .../m2m-self-signed/kustomization.yaml | 0 .../oauth2-proxy/overlays/m2m/README.md | 0 .../kustomization.yaml | 0 .../overlays/m2m/kustomization.yaml | 0 .../oauth2-proxy/base/deployment.yaml | 85 ------- ...ronjob.kubeflow-m2m-oidc-configurator.yaml | 39 --- .../deployment.jwt-refresh-interval.yaml | 18 -- .../components/istio-external-auth/README.md | 0 .../requestauthentication.dex-jwt.yaml | 19 -- .../istio-m2m/requestauthentication.yaml | 14 -- .../oidc-authservice/base/envoy-filter.yaml | 40 --- .../oidc-authservice/base/kustomization.yaml | 47 ---- .../oidc-authservice/base/params.env | 11 - .../oidc-authservice/base/params.yaml | 5 - .../oidc-authservice/base/pvc.yaml | 10 - .../oidc-authservice/base/rbac.yaml | 30 --- .../oidc-authservice/base/secret_params.env | 2 - .../oidc-authservice/base/service.yaml | 13 - .../oidc-authservice/base/statefulset.yaml | 43 ---- .../ibm-storage-config/kustomization.yaml | 10 - .../ibm-storage-config/statefulset.yaml | 15 -- example/kustomization.yaml | 4 +- hack/trivy_scan.py | 6 +- proposals/20240606-jwt-handling.md | 2 +- .../gh-actions/install_istio_with_ext_auth.sh | 2 +- tests/gh-actions/test_dex_login.py | 108 ++++++++ 94 files changed, 833 insertions(+), 954 deletions(-) rename apps/centraldashboard/{upstream => }/overlays/oauth2-proxy/kustomization.yaml (62%) rename common/{oidc-client => }/oauth2-proxy/OWNERS (100%) rename common/{oidc-client => oauth2-proxy}/README.md (99%) rename common/{oidc-client => }/oauth2-proxy/base/README.md (100%) create mode 100644 common/oauth2-proxy/base/deployment.yaml rename common/{oidc-client => }/oauth2-proxy/base/kustomization.yaml (96%) rename common/{oidc-client => }/oauth2-proxy/base/namespace.yaml (100%) rename common/{oidc-client => }/oauth2-proxy/base/oauth2-proxy-config.yaml (100%) rename common/{oidc-client => }/oauth2-proxy/base/params.yaml (100%) rename common/{oidc-client => }/oauth2-proxy/base/rbac.tokenreviews.yaml (66%) rename common/{oidc-client => }/oauth2-proxy/base/service.yaml (76%) rename common/{oidc-client => }/oauth2-proxy/base/serviceaccount.yaml (100%) rename common/{oidc-client/oauth2-proxy => oauth2-proxy/components}/README.md (97%) rename common/{oidc-client => }/oauth2-proxy/components/allow-unauthenticated-issuer-discovery/README.md (100%) rename common/{oidc-client => }/oauth2-proxy/components/allow-unauthenticated-issuer-discovery/clusterrolebinding.unauthenticated-oidc-viewer.yaml (100%) rename common/{oidc-client => }/oauth2-proxy/components/allow-unauthenticated-issuer-discovery/kustomization.yaml (100%) rename common/{oidc-client => }/oauth2-proxy/components/central-dashboard/kustomization.yaml (100%) rename common/{oidc-client => }/oauth2-proxy/components/central-dashboard/patches/deployment.logout-url.yaml (68%) rename common/{oidc-client => }/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/README.md (100%) create mode 100644 common/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/cronjob.kubeflow-m2m-oidc-configurator.yaml rename common/{oidc-client => }/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/kustomization.yaml (100%) rename common/{oidc-client => }/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/rbac.yaml (69%) rename common/{oidc-client => }/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/script.sh (100%) rename common/{oidc-client/oauth2-proxy/components => oauth2-proxy/components/istio-external-auth-patches}/README.md (100%) rename common/{oidc-client => }/oauth2-proxy/components/istio-external-auth-patches/kustomization.yaml (100%) rename common/{oidc-client => }/oauth2-proxy/components/istio-external-auth-patches/patches/cm.enable-oauth2-proxy.yaml (100%) create mode 100644 common/oauth2-proxy/components/istio-external-auth-patches/patches/deployment.jwt-refresh-interval.yaml rename common/{oidc-client/oauth2-proxy/components/istio-external-auth-patches => oauth2-proxy/components/istio-external-auth}/README.md (100%) rename common/{oidc-client => }/oauth2-proxy/components/istio-external-auth/authorizationpolicy.istio-ingressgateway-oauth2-proxy-cloudflare.yaml (100%) rename common/{oidc-client => }/oauth2-proxy/components/istio-external-auth/authorizationpolicy.istio-ingressgateway-oauth2-proxy.yaml (100%) rename common/{oidc-client => }/oauth2-proxy/components/istio-external-auth/kustomization.yaml (81%) create mode 100644 common/oauth2-proxy/components/istio-external-auth/requestauthentication.dex-jwt.yaml rename common/{oidc-client => }/oauth2-proxy/components/istio-m2m/README.md (84%) rename common/{oidc-client => }/oauth2-proxy/components/istio-m2m/kustomization.yaml (100%) rename common/{oidc-client => }/oauth2-proxy/components/istio-m2m/params.yaml (100%) create mode 100644 common/oauth2-proxy/components/istio-m2m/requestauthentication.yaml rename common/{oidc-client => oauth2-proxy/components}/kubeflow_auth_diagram.svg (100%) rename common/{oidc-client => }/oauth2-proxy/overlays/m2m-self-signed/kustomization.yaml (100%) rename common/{oidc-client => }/oauth2-proxy/overlays/m2m/README.md (100%) rename common/{oidc-client => }/oauth2-proxy/overlays/m2m/component-overwrite-m2m-token-issuer/kustomization.yaml (100%) rename common/{oidc-client => }/oauth2-proxy/overlays/m2m/kustomization.yaml (100%) delete mode 100644 common/oidc-client/oauth2-proxy/base/deployment.yaml delete mode 100644 common/oidc-client/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/cronjob.kubeflow-m2m-oidc-configurator.yaml delete mode 100644 common/oidc-client/oauth2-proxy/components/istio-external-auth-patches/patches/deployment.jwt-refresh-interval.yaml delete mode 100644 common/oidc-client/oauth2-proxy/components/istio-external-auth/README.md delete mode 100644 common/oidc-client/oauth2-proxy/components/istio-external-auth/requestauthentication.dex-jwt.yaml delete mode 100644 common/oidc-client/oauth2-proxy/components/istio-m2m/requestauthentication.yaml delete mode 100644 common/oidc-client/oidc-authservice/base/envoy-filter.yaml delete mode 100644 common/oidc-client/oidc-authservice/base/kustomization.yaml delete mode 100644 common/oidc-client/oidc-authservice/base/params.env delete mode 100644 common/oidc-client/oidc-authservice/base/params.yaml delete mode 100644 common/oidc-client/oidc-authservice/base/pvc.yaml delete mode 100644 common/oidc-client/oidc-authservice/base/rbac.yaml delete mode 100644 common/oidc-client/oidc-authservice/base/secret_params.env delete mode 100644 common/oidc-client/oidc-authservice/base/service.yaml delete mode 100644 common/oidc-client/oidc-authservice/base/statefulset.yaml delete mode 100644 common/oidc-client/oidc-authservice/overlays/ibm-storage-config/kustomization.yaml delete mode 100644 common/oidc-client/oidc-authservice/overlays/ibm-storage-config/statefulset.yaml create mode 100644 tests/gh-actions/test_dex_login.py diff --git a/.github/workflows/admission_webhook_test.yaml b/.github/workflows/admission_webhook_test.yaml index 3a4d8e04af..2660a70e7a 100644 --- a/.github/workflows/admission_webhook_test.yaml +++ b/.github/workflows/admission_webhook_test.yaml @@ -2,14 +2,14 @@ name: Build & Apply PodDefaults manifests in KinD on: pull_request: paths: - - .github/workflows/admission_webhook_test.yaml - - apps/admission-webhook/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - - tests/gh-actions/install_istio.sh - - tests/gh-actions/install_cert_manager.sh - - common/cert-manager/** + - .github/workflows/admission_webhook_test.yaml + - apps/admission-webhook/upstream/** + - tests/gh-actions/kind-cluster.yaml + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh + - tests/gh-actions/install_istio.sh + - tests/gh-actions/install_cert_manager.sh + - common/cert-manager/** jobs: build: diff --git a/.github/workflows/bentoml_test.yaml b/.github/workflows/bentoml_test.yaml index 71b59956ff..41a8fc276b 100644 --- a/.github/workflows/bentoml_test.yaml +++ b/.github/workflows/bentoml_test.yaml @@ -2,13 +2,13 @@ name: Build & Apply BentoML Yatai Stack manifests in KinD on: pull_request: paths: - - .github/workflows/bentoml_test.yaml - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - - tests/gh-actions/install_cert_manager.sh - - common/cert-manager/** - - contrib/bentoml/** + - .github/workflows/bentoml_test.yaml + - tests/gh-actions/kind-cluster.yaml + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh + - tests/gh-actions/install_cert_manager.sh + - common/cert-manager/** + - contrib/bentoml/** jobs: build: diff --git a/.github/workflows/centraldashboard_test.yaml b/.github/workflows/centraldashboard_test.yaml index 6fd9337da1..f5373d1277 100644 --- a/.github/workflows/centraldashboard_test.yaml +++ b/.github/workflows/centraldashboard_test.yaml @@ -2,12 +2,12 @@ name: Build & Apply CentralDashboard manifests in KinD on: pull_request: paths: - - .github/workflows/centraldashboard_test.yaml - - apps/centraldashboard/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - - tests/gh-actions/install_istio.sh + - .github/workflows/centraldashboard_test.yaml + - apps/centraldashboard/upstream/** + - tests/gh-actions/kind-cluster.yaml + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh + - tests/gh-actions/install_istio.sh jobs: build: diff --git a/.github/workflows/dex_test.yaml b/.github/workflows/dex_test.yaml index 140772cf79..f06bd0e7a4 100644 --- a/.github/workflows/dex_test.yaml +++ b/.github/workflows/dex_test.yaml @@ -2,12 +2,12 @@ name: Build & Apply Dex manifests in KinD on: pull_request: paths: - - .github/workflows/dex_test.yaml - - common/dex/base/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - - tests/gh-actions/install_istio.sh + - .github/workflows/dex_test.yaml + - common/dex/base/** + - tests/gh-actions/kind-cluster.yaml + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh + - tests/gh-actions/install_istio.sh jobs: build: @@ -33,3 +33,14 @@ jobs: cd common/dex kustomize build overlays/istio | kubectl apply -f - kubectl wait --for=condition=Ready pods --all --all-namespaces --timeout 180s + + - name: port forward + run: | + ingress_gateway_service=$(kubectl get svc --namespace istio-system --selector="app=istio-ingressgateway" --output jsonpath='{.items[0].metadata.name}') + nohup kubectl port-forward --namespace istio-system svc/${ingress_gateway_service} 8080:80 & + while ! curl localhost:8080; do echo waiting for port-forwarding; sleep 1; done; echo port-forwarding ready + + - name: test dex login + run: | + pip3 install requests + ./tests/gh-actions/test_dex_login.py diff --git a/.github/workflows/jupyter_web_application_test.yaml b/.github/workflows/jupyter_web_application_test.yaml index 6cd555d8f9..13715e1740 100644 --- a/.github/workflows/jupyter_web_application_test.yaml +++ b/.github/workflows/jupyter_web_application_test.yaml @@ -2,12 +2,12 @@ name: Build & Apply JWA manifests in KinD on: pull_request: paths: - - .github/workflows/jupyter_web_application_test.yaml - - apps/jupyter/jupyter-web-app/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - - tests/gh-actions/install_istio.sh + - .github/workflows/jupyter_web_application_test.yaml + - apps/jupyter/jupyter-web-app/upstream/** + - tests/gh-actions/kind-cluster.yaml + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh + - tests/gh-actions/install_istio.sh jobs: build: diff --git a/.github/workflows/katib_test.yaml b/.github/workflows/katib_test.yaml index c61e2237b3..9edace0aa7 100644 --- a/.github/workflows/katib_test.yaml +++ b/.github/workflows/katib_test.yaml @@ -2,14 +2,14 @@ name: Build & Apply Katib manifests in KinD on: pull_request: paths: - - .github/workflows/katib_test.yaml - - apps/katib/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - - tests/gh-actions/install_istio.sh - - tests/gh-actions/install_cert_manager.sh - - common/cert-manager/** + - .github/workflows/katib_test.yaml + - apps/katib/upstream/** + - tests/gh-actions/kind-cluster.yaml + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh + - tests/gh-actions/install_istio.sh + - tests/gh-actions/install_cert_manager.sh + - common/cert-manager/** jobs: build: diff --git a/.github/workflows/kserve_cni_test.yaml b/.github/workflows/kserve_cni_test.yaml index eb48f8fb34..c9ac27229e 100644 --- a/.github/workflows/kserve_cni_test.yaml +++ b/.github/workflows/kserve_cni_test.yaml @@ -2,69 +2,69 @@ name: Build & Apply KServe manifests in KinD, using istio CNI on: pull_request: paths: - - .github/workflows/kserve_cni_test.yaml - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - - common/istio-cni-1-22/** - - tests/gh-actions/install_cert_manager.sh - - common/cert-manager/** - - tests/gh-actions/install_knative-cni.sh - - common/knative/** - - tests/gh-actions/install_kserve.sh + - .github/workflows/kserve_cni_test.yaml + - tests/gh-actions/kind-cluster.yaml + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh + - common/istio-cni-1-22/** + - tests/gh-actions/install_cert_manager.sh + - common/cert-manager/** + - tests/gh-actions/install_knative-cni.sh + - common/knative/** + - tests/gh-actions/install_kserve.sh jobs: build: runs-on: ubuntu-latest steps: - - name: Checkout - uses: actions/checkout@v4 + - name: Checkout + uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh + - name: Install KinD + run: ./tests/gh-actions/install_kind.sh - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml + - name: Create KinD Cluster + run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install kustomize + run: ./tests/gh-actions/install_kustomize.sh - - name: Create kubeflow namespace - run: kustomize build common/kubeflow-namespace/base | kubectl apply -f - + - name: Create kubeflow namespace + run: kustomize build common/kubeflow-namespace/base | kubectl apply -f - - - name: Install Istio CNI - run: ./tests/gh-actions/install_istio-cni.sh + - name: Install Istio CNI + run: ./tests/gh-actions/install_istio-cni.sh - - name: Install cert-manager - run: ./tests/gh-actions/install_cert_manager.sh + - name: Install cert-manager + run: ./tests/gh-actions/install_cert_manager.sh - - name: Install knative CNI - run: ./tests/gh-actions/install_knative-cni.sh + - name: Install knative CNI + run: ./tests/gh-actions/install_knative-cni.sh - - name: Build & Apply manifests - run: ./tests/gh-actions/install_kserve.sh + - name: Build & Apply manifests + run: ./tests/gh-actions/install_kserve.sh - - name: Create test namespace - run: kubectl create ns kserve-test + - name: Create test namespace + run: kubectl create ns kserve-test - - name: Setup python 3.9 - uses: actions/setup-python@v4 - with: - python-version: 3.9 + - name: Setup python 3.9 + uses: actions/setup-python@v4 + with: + python-version: 3.9 - - name: Install test dependencies - run: pip install -r ./contrib/kserve/tests/requirements.txt + - name: Install test dependencies + run: pip install -r ./contrib/kserve/tests/requirements.txt - - name: Port forward - run: | - INGRESS_GATEWAY_SERVICE=$(kubectl get svc --namespace istio-system --selector="app=istio-ingressgateway" --output jsonpath='{.items[0].metadata.name}') - nohup kubectl port-forward --namespace istio-system svc/${INGRESS_GATEWAY_SERVICE} 8080:80 & + - name: Port forward + run: | + INGRESS_GATEWAY_SERVICE=$(kubectl get svc --namespace istio-system --selector="app=istio-ingressgateway" --output jsonpath='{.items[0].metadata.name}') + nohup kubectl port-forward --namespace istio-system svc/${INGRESS_GATEWAY_SERVICE} 8080:80 & - - name: Run kserve tests - run: | - export KSERVE_INGRESS_HOST_PORT=localhost:8080 - cd ./contrib/kserve/tests && pytest . + - name: Run kserve tests + run: | + export KSERVE_INGRESS_HOST_PORT=localhost:8080 + cd ./contrib/kserve/tests && pytest . - - name: Run kserve models webapp test - run: | - kubectl wait --for=condition=Available --timeout=300s -n kubeflow deployment/kserve-models-web-app + - name: Run kserve models webapp test + run: | + kubectl wait --for=condition=Available --timeout=300s -n kubeflow deployment/kserve-models-web-app diff --git a/.github/workflows/kserve_m2m_test.yaml b/.github/workflows/kserve_m2m_test.yaml index 3b79215e57..b468702f1a 100644 --- a/.github/workflows/kserve_m2m_test.yaml +++ b/.github/workflows/kserve_m2m_test.yaml @@ -2,88 +2,88 @@ name: Deploy and test KServe with m2m auth in KinD on: pull_request: paths: - - .github/workflows/kserve_m2m_test.yaml - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - - contrib/kserve/** - - common/oidc-client/oauth2-proxy/** - - common/istio*/** - - tests/gh-actions/install_istio_with_ext_auth.sh* - - tests/gh-actions/install_cert_manager.sh - - common/cert-manager/** - - tests/gh-actions/install_knative.sh - - common/knative/** - - tests/gh-actions/install_kserve.sh + - .github/workflows/kserve_m2m_test.yaml + - tests/gh-actions/kind-cluster.yaml + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh + - contrib/kserve/** + - common/oauth2-proxy/** + - common/istio*/** + - tests/gh-actions/install_istio_with_ext_auth.sh* + - tests/gh-actions/install_cert_manager.sh + - common/cert-manager/** + - tests/gh-actions/install_knative.sh + - common/knative/** + - tests/gh-actions/install_kserve.sh jobs: build: runs-on: ubuntu-latest steps: - - name: Checkout - uses: actions/checkout@v4 + - name: Checkout + uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh + - name: Install KinD + run: ./tests/gh-actions/install_kind.sh - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml + - name: Create KinD Cluster + run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - name: Install kubectl - run: ./tests/gh-actions/install_kubectl.sh + - name: Install kubectl + run: ./tests/gh-actions/install_kubectl.sh - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install kustomize + run: ./tests/gh-actions/install_kustomize.sh - - name: Create kubeflow namespace - run: kustomize build common/kubeflow-namespace/base | kubectl apply -f - + - name: Create kubeflow namespace + run: kustomize build common/kubeflow-namespace/base | kubectl apply -f - - - name: Install Istio with external authentication - run: ./tests/gh-actions/install_istio_with_ext_auth.sh + - name: Install Istio with external authentication + run: ./tests/gh-actions/install_istio_with_ext_auth.sh - - name: Install cert-manager - run: ./tests/gh-actions/install_cert_manager.sh + - name: Install cert-manager + run: ./tests/gh-actions/install_cert_manager.sh - - name: Install knative - run: ./tests/gh-actions/install_knative.sh + - name: Install knative + run: ./tests/gh-actions/install_knative.sh - - name: Build & Apply manifests - run: ./tests/gh-actions/install_kserve.sh + - name: Build & Apply manifests + run: ./tests/gh-actions/install_kserve.sh - - name: Create test namespace - run: kubectl create ns kserve-test + - name: Create test namespace + run: kubectl create ns kserve-test - - name: Setup python 3.9 - uses: actions/setup-python@v4 - with: - python-version: 3.9 + - name: Setup python 3.9 + uses: actions/setup-python@v4 + with: + python-version: 3.9 - - name: Install test dependencies - run: pip install -r ./contrib/kserve/tests/requirements.txt + - name: Install test dependencies + run: pip install -r ./contrib/kserve/tests/requirements.txt - - name: Port forward - run: | - INGRESS_GATEWAY_SERVICE=$(kubectl get svc --namespace istio-system --selector="app=istio-ingressgateway" --output jsonpath='{.items[0].metadata.name}') - nohup kubectl port-forward --namespace istio-system svc/${INGRESS_GATEWAY_SERVICE} 8080:80 & - while ! curl localhost:8080; do echo waiting for port-forwarding; sleep 1; done; echo port-forwarding ready + - name: Port forward + run: | + INGRESS_GATEWAY_SERVICE=$(kubectl get svc --namespace istio-system --selector="app=istio-ingressgateway" --output jsonpath='{.items[0].metadata.name}') + nohup kubectl port-forward --namespace istio-system svc/${INGRESS_GATEWAY_SERVICE} 8080:80 & + while ! curl localhost:8080; do echo waiting for port-forwarding; sleep 1; done; echo port-forwarding ready - - name: Wait for the kubeflow-m2m-oidc-configurator Job - run: | - ./tests/gh-actions/wait_for_kubeflow_m2m_oidc_configurator.sh + - name: Wait for the kubeflow-m2m-oidc-configurator Job + run: | + ./tests/gh-actions/wait_for_kubeflow_m2m_oidc_configurator.sh - - name: Run kserve tests with m2m token from SA default/default - run: | - export KSERVE_INGRESS_HOST_PORT=localhost:8080 - export KSERVE_M2M_TOKEN="$(kubectl -n default create token default)" - cd ./contrib/kserve/tests && pytest . -vs --log-level info + - name: Run kserve tests with m2m token from SA default/default + run: | + export KSERVE_INGRESS_HOST_PORT=localhost:8080 + export KSERVE_M2M_TOKEN="$(kubectl -n default create token default)" + cd ./contrib/kserve/tests && pytest . -vs --log-level info - - name: Run and fail kserve tests without kserve m2m token - run: | - export KSERVE_INGRESS_HOST_PORT=localhost:8080 - cd ./contrib/kserve/tests - if pytest . -vs --log-level info; then - echo "This test should fail with an HTTP redirect to oauth2-proxy/dex auth."; exit 1 - else - echo "Task failed successfully!" - echo "This is a provisional way of testing that m2m is enabled for kserve." - fi + - name: Run and fail kserve tests without kserve m2m token + run: | + export KSERVE_INGRESS_HOST_PORT=localhost:8080 + cd ./contrib/kserve/tests + if pytest . -vs --log-level info; then + echo "This test should fail with an HTTP redirect to oauth2-proxy/dex auth."; exit 1 + else + echo "Task failed successfully!" + echo "This is a provisional way of testing that m2m is enabled for kserve." + fi diff --git a/.github/workflows/kserve_test.yaml b/.github/workflows/kserve_test.yaml index e4d5ef7cfe..81b9825115 100644 --- a/.github/workflows/kserve_test.yaml +++ b/.github/workflows/kserve_test.yaml @@ -2,69 +2,69 @@ name: Build & Apply KServe manifests in KinD on: pull_request: paths: - - .github/workflows/kserve_test.yaml - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - - contrib/kserve/** - - tests/gh-actions/install_cert_manager.sh - - common/cert-manager/** - - tests/gh-actions/install_knative.sh - - common/knative/** - - tests/gh-actions/install_kserve.sh + - .github/workflows/kserve_test.yaml + - tests/gh-actions/kind-cluster.yaml + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh + - contrib/kserve/** + - tests/gh-actions/install_cert_manager.sh + - common/cert-manager/** + - tests/gh-actions/install_knative.sh + - common/knative/** + - tests/gh-actions/install_kserve.sh jobs: build: runs-on: ubuntu-latest steps: - - name: Checkout - uses: actions/checkout@v4 + - name: Checkout + uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh + - name: Install KinD + run: ./tests/gh-actions/install_kind.sh - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml + - name: Create KinD Cluster + run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install kustomize + run: ./tests/gh-actions/install_kustomize.sh - - name: Create kubeflow namespace - run: kustomize build common/kubeflow-namespace/base | kubectl apply -f - + - name: Create kubeflow namespace + run: kustomize build common/kubeflow-namespace/base | kubectl apply -f - - - name: Install Istio - run: ./tests/gh-actions/install_istio.sh + - name: Install Istio + run: ./tests/gh-actions/install_istio.sh - - name: Install cert-manager - run: ./tests/gh-actions/install_cert_manager.sh + - name: Install cert-manager + run: ./tests/gh-actions/install_cert_manager.sh - - name: Install knative - run: ./tests/gh-actions/install_knative.sh + - name: Install knative + run: ./tests/gh-actions/install_knative.sh - - name: Build & Apply manifests - run: ./tests/gh-actions/install_kserve.sh + - name: Build & Apply manifests + run: ./tests/gh-actions/install_kserve.sh - - name: Create test namespace - run: kubectl create ns kserve-test + - name: Create test namespace + run: kubectl create ns kserve-test - - name: Setup python 3.9 - uses: actions/setup-python@v4 - with: - python-version: 3.9 + - name: Setup python 3.9 + uses: actions/setup-python@v4 + with: + python-version: 3.9 - - name: Install test dependencies - run: pip install -r ./contrib/kserve/tests/requirements.txt + - name: Install test dependencies + run: pip install -r ./contrib/kserve/tests/requirements.txt - - name: Port forward - run: | - INGRESS_GATEWAY_SERVICE=$(kubectl get svc --namespace istio-system --selector="app=istio-ingressgateway" --output jsonpath='{.items[0].metadata.name}') - nohup kubectl port-forward --namespace istio-system svc/${INGRESS_GATEWAY_SERVICE} 8080:80 & + - name: Port forward + run: | + INGRESS_GATEWAY_SERVICE=$(kubectl get svc --namespace istio-system --selector="app=istio-ingressgateway" --output jsonpath='{.items[0].metadata.name}') + nohup kubectl port-forward --namespace istio-system svc/${INGRESS_GATEWAY_SERVICE} 8080:80 & - - name: Run kserve tests - run: | - export KSERVE_INGRESS_HOST_PORT=localhost:8080 - cd ./contrib/kserve/tests && pytest . + - name: Run kserve tests + run: | + export KSERVE_INGRESS_HOST_PORT=localhost:8080 + cd ./contrib/kserve/tests && pytest . - - name: Run kserve models webapp test - run: | - kubectl wait --for=condition=Available --timeout=300s -n kubeflow deployment/kserve-models-web-app \ No newline at end of file + - name: Run kserve models webapp test + run: | + kubectl wait --for=condition=Available --timeout=300s -n kubeflow deployment/kserve-models-web-app diff --git a/.github/workflows/linting_bash_python_yaml_files.yaml b/.github/workflows/linting_bash_python_yaml_files.yaml index 3d168fde5f..bd38c1eea4 100644 --- a/.github/workflows/linting_bash_python_yaml_files.yaml +++ b/.github/workflows/linting_bash_python_yaml_files.yaml @@ -125,7 +125,7 @@ jobs: fi - name: Display changed files - if: always() # Always run this step + if: always() # Always run this step run: cat changed_files_in_PR.txt || echo "No bash files have changed in this PR." - name: Run ShellCheck on changed files diff --git a/.github/workflows/manifests_example_test.yaml b/.github/workflows/manifests_example_test.yaml index 72579e63b8..55ed3ce040 100644 --- a/.github/workflows/manifests_example_test.yaml +++ b/.github/workflows/manifests_example_test.yaml @@ -1,8 +1,8 @@ name: Unit Test on: - - push - - pull_request +- push +- pull_request jobs: build: @@ -19,4 +19,3 @@ jobs: - name: Unit Test run: | kustomize build example - diff --git a/.github/workflows/metacontroller_test.yaml b/.github/workflows/metacontroller_test.yaml index 403c126aa1..987d30cd74 100644 --- a/.github/workflows/metacontroller_test.yaml +++ b/.github/workflows/metacontroller_test.yaml @@ -2,15 +2,15 @@ name: Build & Apply contrib/metacontroller in KinD on: pull_request: paths: - - .github/workflows/metacontroller_test.yaml - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - - contrib/metacontroller/** - - tests/gh-actions/install_cert_manager.sh - - common/cert-manager/** - - tests/gh-actions/install_istio.sh - - common/istio*/** + - .github/workflows/metacontroller_test.yaml + - tests/gh-actions/kind-cluster.yaml + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh + - contrib/metacontroller/** + - tests/gh-actions/install_cert_manager.sh + - common/cert-manager/** + - tests/gh-actions/install_istio.sh + - common/istio*/** jobs: build: diff --git a/.github/workflows/model_registry_test.yaml b/.github/workflows/model_registry_test.yaml index 082d9b540b..e3c59aa6cb 100644 --- a/.github/workflows/model_registry_test.yaml +++ b/.github/workflows/model_registry_test.yaml @@ -1,14 +1,14 @@ -# If anyone changes or improve the following tests for Model Registry, please +# If anyone changes or improve the following tests for Model Registry, please # consider reflecting the same changes on https://github.com/kubeflow/model-registry name: Deploy and test Kubeflow Model Registry on: pull_request: paths: - - apps/model-registry/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kustomize.sh - - tests/gh-actions/install_istio.sh - - common/istio*/** + - apps/model-registry/upstream/** + - tests/gh-actions/kind-cluster.yaml + - tests/gh-actions/install_kustomize.sh + - tests/gh-actions/install_istio.sh + - common/istio*/** jobs: build-kfmr: @@ -61,7 +61,7 @@ jobs: 'http://localhost:8081/api/model_registry/v1alpha3/registered_models?pageSize=100&orderBy=ID&sortOrder=DESC' \ -H 'accept: application/json' - # for these steps below ensure same steps as kserve (ie: Istio with external authentication, cert-manager, knative) so to achieve same setup + # for these steps below ensure same steps as kserve (ie: Istio with external authentication, cert-manager, knative) so to achieve same setup - name: Port forward Istio gateway run: | INGRESS_GATEWAY_SERVICE=$(kubectl get svc --namespace istio-system --selector="app=istio-ingressgateway" --output jsonpath='{.items[0].metadata.name}') diff --git a/.github/workflows/notebook_controller_m2m_test.yaml b/.github/workflows/notebook_controller_m2m_test.yaml index 8c49bf40a3..fa0f70a934 100644 --- a/.github/workflows/notebook_controller_m2m_test.yaml +++ b/.github/workflows/notebook_controller_m2m_test.yaml @@ -2,15 +2,15 @@ name: Test Notebook Controller with m2m auth manifests in KinD on: pull_request: paths: - - .github/workflows/notebook_controller_m2m_test.yaml - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - - apps/jupyter/** - - common/oidc-client/oauth2-proxy/** - - common/istio*/** - - tests/gh-actions/install_istio_with_ext_auth.sh* - - tests/gh-actions/install_multi_tenancy.sh + - .github/workflows/notebook_controller_m2m_test.yaml + - tests/gh-actions/kind-cluster.yaml + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh + - apps/jupyter/** + - common/oauth2-proxy/** + - common/istio*/** + - tests/gh-actions/install_istio_with_ext_auth.sh* + - tests/gh-actions/install_multi_tenancy.sh jobs: build: diff --git a/.github/workflows/notebook_controller_test.yaml b/.github/workflows/notebook_controller_test.yaml index abb01afaf0..a423f05459 100644 --- a/.github/workflows/notebook_controller_test.yaml +++ b/.github/workflows/notebook_controller_test.yaml @@ -2,13 +2,13 @@ name: Build & Apply Notebook Controller manifests in KinD on: pull_request: paths: - - .github/workflows/notebook_controller_test.yaml - - apps/jupyter/notebook-controller/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - - tests/gh-actions/install_istio.sh - - common/istio*/** + - .github/workflows/notebook_controller_test.yaml + - apps/jupyter/notebook-controller/upstream/** + - tests/gh-actions/kind-cluster.yaml + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh + - tests/gh-actions/install_istio.sh + - common/istio*/** jobs: build: diff --git a/.github/workflows/pipeline_run_from_notebook.yaml b/.github/workflows/pipeline_run_from_notebook.yaml index 8e965bb375..7d91966adc 100644 --- a/.github/workflows/pipeline_run_from_notebook.yaml +++ b/.github/workflows/pipeline_run_from_notebook.yaml @@ -2,19 +2,19 @@ name: Create Pipeline Run from Kubeflow Notebook on: pull_request: paths: - - .github/workflows/pipeline_run_from_notebook.yaml - - apps/jupyter/notebook-controller/upstream/** - - apps/pipeline/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - - tests/gh-actions/install_istio.sh - - tests/gh-actions/install_cert_manager.sh - - common/cert-manager/** - - common/oidc-client/oauth2-proxy/** - - common/istio*/** - - common/oidc-client/** - - apps/jupyter/** + - .github/workflows/pipeline_run_from_notebook.yaml + - apps/jupyter/notebook-controller/upstream/** + - apps/pipeline/upstream/** + - tests/gh-actions/kind-cluster.yaml + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh + - tests/gh-actions/install_istio.sh + - tests/gh-actions/install_cert_manager.sh + - common/cert-manager/** + - common/oauth2-proxy/** + - common/istio*/** + - common//** + - apps/jupyter/** jobs: build: diff --git a/.github/workflows/pipeline_test.yaml b/.github/workflows/pipeline_test.yaml index 2df9ee79c0..2cfcca0cf6 100644 --- a/.github/workflows/pipeline_test.yaml +++ b/.github/workflows/pipeline_test.yaml @@ -2,125 +2,125 @@ name: Deploy and test Kubeflow Pipelines manifests with m2m auth in KinD on: pull_request: paths: - - .github/workflows/pipeline_test.yaml - - apps/pipeline/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - - tests/gh-actions/install_istio.sh - - tests/gh-actions/install_cert_manager.sh - - common/cert-manager/** - - common/oidc-client/oauth2-proxy/** - - common/istio*/** - - tests/gh-actions/install_istio_with_ext_auth.sh + - .github/workflows/pipeline_test.yaml + - apps/pipeline/upstream/** + - tests/gh-actions/kind-cluster.yaml + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh + - tests/gh-actions/install_istio.sh + - tests/gh-actions/install_cert_manager.sh + - common/cert-manager/** + - common/oauth2-proxy/** + - common/istio*/** + - tests/gh-actions/install_istio_with_ext_auth.sh jobs: build: runs-on: ubuntu-latest steps: - - name: Checkout - uses: actions/checkout@v4 - - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh - - - name: Install kubectl - run: ./tests/gh-actions/install_kubectl.sh - - - name: Install Istio with external authentication - run: ./tests/gh-actions/install_istio_with_ext_auth.sh - - - name: Install cert-manager - run: ./tests/gh-actions/install_cert_manager.sh - - - name: Create kubeflow namespace - run: kustomize build common/kubeflow-namespace/base | kubectl apply -f - - - - name: Install KF Pipelines - run: ./tests/gh-actions/install_pipelines.sh - - - name: Install KF Multi Tenancy - run: ./tests/gh-actions/install_multi_tenancy.sh - - - name: Install kubeflow-istio-resources - run: kustomize build common/istio-1-22/kubeflow-istio-resources/base | kubectl apply -f - - - - name: Create KF Profile - run: kustomize build common/user-namespace/base | kubectl apply -f - - - - name: port forward - run: | - ingress_gateway_service=$(kubectl get svc --namespace istio-system --selector="app=istio-ingressgateway" --output jsonpath='{.items[0].metadata.name}') - nohup kubectl port-forward --namespace istio-system svc/${ingress_gateway_service} 8080:80 & - while ! curl localhost:8080; do echo waiting for port-forwarding; sleep 1; done; echo port-forwarding ready - - - name: Wait for the kubeflow-m2m-oidc-configurator Job - run: | - ./tests/gh-actions/wait_for_kubeflow_m2m_oidc_configurator.sh - - - name: List and deploy test pipeline with authorized ServiceAccount Token - run: | - pip3 install kfp==2.4.0 - KF_PROFILE=kubeflow-user-example-com - TOKEN="$(kubectl -n $KF_PROFILE create token default-editor)" - - python -c ' - from time import sleep - import kfp - import sys - - token = sys.argv[1] - namespace = sys.argv[2] - client = kfp.Client(host="http://localhost:8080/pipeline", existing_token=token) - - pipeline = client.list_pipelines().pipelines[0] - pipeline_name = pipeline.display_name - pipeline_id = pipeline.pipeline_id - pipeline_version_id = client.list_pipeline_versions(pipeline_id).pipeline_versions[0].pipeline_version_id - experiment_id = client.create_experiment("m2m-test", namespace=namespace).experiment_id - - print(f"Starting pipeline {pipeline_name}.") - run_id = client.run_pipeline(experiment_id=experiment_id, job_name="m2m-test", pipeline_id=pipeline_id, version_id=pipeline_version_id).run_id - - while True: - status = client.get_run(run_id=run_id).state - if status in ["PENDING", "RUNNING"]: - print(f"Waiting for run_id: {run_id}, status: {status}.") - sleep(10) - else: - print(f"Run with id {run_id} finished with status: {status}.") - if status != "SUCCEEDED": - print("Pipeline failed") - raise SystemExit(1) - break - ' "${TOKEN}" "${KF_PROFILE}" - - - name: Fail to list pipelines with unauthorized ServiceAccount Token - run: | - pip3 install kfp==2.4.0 - KF_PROFILE=kubeflow-user-example-com - TOKEN="$(kubectl -n default create token default)" - - python -c ' - import kfp - import sys - from kfp_server_api.exceptions import ApiException - - token = sys.argv[1] - namespace = sys.argv[2] - client = kfp.Client(host="http://localhost:8080/pipeline", existing_token=token) - - try: - pipeline = client.list_runs(namespace=namespace) - except ApiException as e: - assert e.status == 403, "This API Call should return unauthorized/forbidden error." - ' "${TOKEN}" "${KF_PROFILE}" - - echo "Test succeeded. Token from unauthorized ServiceAccount cannot list \ - piplines in $KF_PROFILE namespace." + - name: Checkout + uses: actions/checkout@v4 + + - name: Install KinD + run: ./tests/gh-actions/install_kind.sh + + - name: Create KinD Cluster + run: kind create cluster --config tests/gh-actions/kind-cluster.yaml + + - name: Install kustomize + run: ./tests/gh-actions/install_kustomize.sh + + - name: Install kubectl + run: ./tests/gh-actions/install_kubectl.sh + + - name: Install Istio with external authentication + run: ./tests/gh-actions/install_istio_with_ext_auth.sh + + - name: Install cert-manager + run: ./tests/gh-actions/install_cert_manager.sh + + - name: Create kubeflow namespace + run: kustomize build common/kubeflow-namespace/base | kubectl apply -f - + + - name: Install KF Pipelines + run: ./tests/gh-actions/install_pipelines.sh + + - name: Install KF Multi Tenancy + run: ./tests/gh-actions/install_multi_tenancy.sh + + - name: Install kubeflow-istio-resources + run: kustomize build common/istio-1-22/kubeflow-istio-resources/base | kubectl apply -f - + + - name: Create KF Profile + run: kustomize build common/user-namespace/base | kubectl apply -f - + + - name: port forward + run: | + ingress_gateway_service=$(kubectl get svc --namespace istio-system --selector="app=istio-ingressgateway" --output jsonpath='{.items[0].metadata.name}') + nohup kubectl port-forward --namespace istio-system svc/${ingress_gateway_service} 8080:80 & + while ! curl localhost:8080; do echo waiting for port-forwarding; sleep 1; done; echo port-forwarding ready + + - name: Wait for the kubeflow-m2m-oidc-configurator Job + run: | + ./tests/gh-actions/wait_for_kubeflow_m2m_oidc_configurator.sh + + - name: List and deploy test pipeline with authorized ServiceAccount Token + run: | + pip3 install kfp==2.4.0 + KF_PROFILE=kubeflow-user-example-com + TOKEN="$(kubectl -n $KF_PROFILE create token default-editor)" + + python -c ' + from time import sleep + import kfp + import sys + + token = sys.argv[1] + namespace = sys.argv[2] + client = kfp.Client(host="http://localhost:8080/pipeline", existing_token=token) + + pipeline = client.list_pipelines().pipelines[0] + pipeline_name = pipeline.display_name + pipeline_id = pipeline.pipeline_id + pipeline_version_id = client.list_pipeline_versions(pipeline_id).pipeline_versions[0].pipeline_version_id + experiment_id = client.create_experiment("m2m-test", namespace=namespace).experiment_id + + print(f"Starting pipeline {pipeline_name}.") + run_id = client.run_pipeline(experiment_id=experiment_id, job_name="m2m-test", pipeline_id=pipeline_id, version_id=pipeline_version_id).run_id + + while True: + status = client.get_run(run_id=run_id).state + if status in ["PENDING", "RUNNING"]: + print(f"Waiting for run_id: {run_id}, status: {status}.") + sleep(10) + else: + print(f"Run with id {run_id} finished with status: {status}.") + if status != "SUCCEEDED": + print("Pipeline failed") + raise SystemExit(1) + break + ' "${TOKEN}" "${KF_PROFILE}" + + - name: Fail to list pipelines with unauthorized ServiceAccount Token + run: | + pip3 install kfp==2.4.0 + KF_PROFILE=kubeflow-user-example-com + TOKEN="$(kubectl -n default create token default)" + + python -c ' + import kfp + import sys + from kfp_server_api.exceptions import ApiException + + token = sys.argv[1] + namespace = sys.argv[2] + client = kfp.Client(host="http://localhost:8080/pipeline", existing_token=token) + + try: + pipeline = client.list_runs(namespace=namespace) + except ApiException as e: + assert e.status == 403, "This API Call should return unauthorized/forbidden error." + ' "${TOKEN}" "${KF_PROFILE}" + + echo "Test succeeded. Token from unauthorized ServiceAccount cannot list \ + piplines in $KF_PROFILE namespace." diff --git a/.github/workflows/profiles_test.yaml b/.github/workflows/profiles_test.yaml index 2352352bf9..d562ff7c0d 100644 --- a/.github/workflows/profiles_test.yaml +++ b/.github/workflows/profiles_test.yaml @@ -2,13 +2,13 @@ name: Build & Apply Profiles manifests in KinD on: pull_request: paths: - - .github/workflows/profiles_test.yaml - - apps/profiles/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - - tests/gh-actions/install_istio.sh - - common/istio*/** + - .github/workflows/profiles_test.yaml + - apps/profiles/upstream/** + - tests/gh-actions/kind-cluster.yaml + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh + - tests/gh-actions/install_istio.sh + - common/istio*/** jobs: build: diff --git a/.github/workflows/ray_test.yaml b/.github/workflows/ray_test.yaml index 56732ddbc6..d726105e84 100644 --- a/.github/workflows/ray_test.yaml +++ b/.github/workflows/ray_test.yaml @@ -2,10 +2,10 @@ name: Build & Apply Ray manifest in KinD on: pull_request: paths: - - .github/workflows/ray_test.yaml - - contrib/ray/** - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh + - .github/workflows/ray_test.yaml + - contrib/ray/** + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh jobs: build: diff --git a/.github/workflows/seldon_test.yaml b/.github/workflows/seldon_test.yaml index 822ba29df4..c98b8be446 100644 --- a/.github/workflows/seldon_test.yaml +++ b/.github/workflows/seldon_test.yaml @@ -2,15 +2,15 @@ name: Build & Apply Seldon manifests in KinD on: pull_request: paths: - - .github/workflows/seldon_test.yaml - - tests/gh-actions/kind-cluster.yaml - - contrib/seldon/** - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - - tests/gh-actions/install_istio.sh - - common/istio*/** - - tests/gh-actions/install_cert_manager.sh - - common/cert-manager/** + - .github/workflows/seldon_test.yaml + - tests/gh-actions/kind-cluster.yaml + - contrib/seldon/** + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh + - tests/gh-actions/install_istio.sh + - common/istio*/** + - tests/gh-actions/install_cert_manager.sh + - common/cert-manager/** jobs: build: diff --git a/.github/workflows/stale.yaml b/.github/workflows/stale.yaml index dc8c85a9da..e56901a44c 100644 --- a/.github/workflows/stale.yaml +++ b/.github/workflows/stale.yaml @@ -7,7 +7,7 @@ name: Mark stale issues and pull requests on: schedule: - - cron: '0 0 * * *' # Run every day at midnight + - cron: '0 0 * * *' # Run every day at midnight jobs: stale: @@ -17,40 +17,32 @@ jobs: pull-requests: write steps: - - uses: actions/stale@v5 - with: - repo-token: ${{ secrets.GITHUB_TOKEN }} - days-before-stale: 60 - days-before-close: 21 - # The message that will be added as a comment to the issues - # when the stale workflow marks it automatically as stale with a label. - stale-issue-message: > - This issue has been automatically marked as stale because it has not had - recent activity. It will be closed if no further activity occurs. Thank you - for your contributions. - # The message that will be added as a comment to the issues - # when the stale workflow closes it automatically after being stale for too long. - close-issue-message: > - This issue has been automatically closed because it has not had recent - activity. Please comment "/reopen" to reopen it. - stale-issue-label: lifecycle/stale - # Exclude them from being marked as stale - exempt-issue-labels: lifecycle/frozen,enhancement,good first issue - # The message that will be added as a comment to the pull requests - # when the stale workflow marks it automatically as stale with a label. - stale-pr-message: > - This pull request has been automatically marked as stale because it has not had - recent activity. It will be closed if no further activity occurs. Thank you - for your contributions. - # The message that will be added as a comment to the pull requests - # when the stale workflow closes it automatically after being stale for too long. - close-pr-message: > - This pull request has been automatically closed because it has not had recent - activity.You can reopen the PR if you want. - stale-pr-label: lifecycle/stale - # Exclude them from being marked as stale - exempt-pr-labels: lifecycle/frozen,enhancement,good first issue - # The issues or the pull requests with a milestone will not be marked as stale automatically - exempt-all-milestones: true - # Learn more about operations: https://github.com/actions/stale#operations-per-run. - operations-per-run: 250 \ No newline at end of file + - uses: actions/stale@v5 + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + days-before-stale: 60 + days-before-close: 21 + # The message that will be added as a comment to the issues + # when the stale workflow marks it automatically as stale with a label. + stale-issue-message: > + This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. + # The message that will be added as a comment to the issues + # when the stale workflow closes it automatically after being stale for too long. + close-issue-message: > + This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it. + stale-issue-label: lifecycle/stale + # Exclude them from being marked as stale + exempt-issue-labels: lifecycle/frozen,enhancement,good first issue + # The message that will be added as a comment to the pull requests + # when the stale workflow marks it automatically as stale with a label. + stale-pr-message: "This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. \n" + # The message that will be added as a comment to the pull requests + # when the stale workflow closes it automatically after being stale for too long. + close-pr-message: "This pull request has been automatically closed because it has not had recent activity.You can reopen the PR if you want. \n" + stale-pr-label: lifecycle/stale + # Exclude them from being marked as stale + exempt-pr-labels: lifecycle/frozen,enhancement,good first issue + # The issues or the pull requests with a milestone will not be marked as stale automatically + exempt-all-milestones: true + # Learn more about operations: https://github.com/actions/stale#operations-per-run. + operations-per-run: 250 diff --git a/.github/workflows/tensorboard_controller_test.yaml b/.github/workflows/tensorboard_controller_test.yaml index 9e1104fce7..69c044db9a 100644 --- a/.github/workflows/tensorboard_controller_test.yaml +++ b/.github/workflows/tensorboard_controller_test.yaml @@ -2,13 +2,13 @@ name: Build & Apply Tensorboard Controller manifests in KinD on: pull_request: paths: - - .github/workflows/tensorboard_controller_test.yaml - - apps/tensorboard/tensorboard-controller/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - - tests/gh-actions/install_istio.sh - - common/istio*/** + - .github/workflows/tensorboard_controller_test.yaml + - apps/tensorboard/tensorboard-controller/upstream/** + - tests/gh-actions/kind-cluster.yaml + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh + - tests/gh-actions/install_istio.sh + - common/istio*/** jobs: build: diff --git a/.github/workflows/tensorboards_web_application_test.yaml b/.github/workflows/tensorboards_web_application_test.yaml index b74640325b..a0072655d4 100644 --- a/.github/workflows/tensorboards_web_application_test.yaml +++ b/.github/workflows/tensorboards_web_application_test.yaml @@ -2,13 +2,13 @@ name: Build & Apply TWA manifests in KinD on: pull_request: paths: - - .github/workflows/tensorboards_web_application_test.yaml - - apps/tensorboard/tensorboards-web-app/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - - tests/gh-actions/install_istio.sh - - common/istio*/** + - .github/workflows/tensorboards_web_application_test.yaml + - apps/tensorboard/tensorboards-web-app/upstream/** + - tests/gh-actions/kind-cluster.yaml + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh + - tests/gh-actions/install_istio.sh + - common/istio*/** jobs: build: diff --git a/.github/workflows/training_operator_test.yaml b/.github/workflows/training_operator_test.yaml index 5be7d5dffe..e9b107beb1 100644 --- a/.github/workflows/training_operator_test.yaml +++ b/.github/workflows/training_operator_test.yaml @@ -2,14 +2,14 @@ name: Build & Apply Training Operator manifests in KinD on: pull_request: paths: - - .github/workflows/training_operator_test.yaml - - apps/training-operator/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - - tests/gh-actions/install_istio.sh - - common/istio*/** - - tests/gh-actions/kf-objects/tfjob.yaml + - .github/workflows/training_operator_test.yaml + - apps/training-operator/upstream/** + - tests/gh-actions/kind-cluster.yaml + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh + - tests/gh-actions/install_istio.sh + - common/istio*/** + - tests/gh-actions/kf-objects/tfjob.yaml jobs: build: diff --git a/.github/workflows/triage_issues.yaml b/.github/workflows/triage_issues.yaml index 5d4551ce97..14bcb4703b 100644 --- a/.github/workflows/triage_issues.yaml +++ b/.github/workflows/triage_issues.yaml @@ -1,4 +1,4 @@ -# Define a GitHub action workflow to determine whether issues +# Define a GitHub action workflow to determine whether issues # should be added or removed from the Needs Triage Kanban board. name: Check Triage Status of Issue on: @@ -10,10 +10,9 @@ jobs: test: runs-on: ubuntu-latest steps: - - name: Update Kanban - uses: kubeflow/code-intelligence/Issue_Triage/action@master - with: - # Letting input NEEDS_TRIAGE_PROJECT_CARD_ID use the default value - ISSUE_NUMBER: ${{ github.event.issue.number }} - GITHUB_PERSONAL_ACCESS_TOKEN: ${{ secrets.triage_projects_github_token }} - + - name: Update Kanban + uses: kubeflow/code-intelligence/Issue_Triage/action@master + with: + # Letting input NEEDS_TRIAGE_PROJECT_CARD_ID use the default value + ISSUE_NUMBER: ${{ github.event.issue.number }} + GITHUB_PERSONAL_ACCESS_TOKEN: ${{ secrets.triage_projects_github_token }} diff --git a/.github/workflows/trivy.yaml b/.github/workflows/trivy.yaml index f10d0ee828..99bfa64265 100644 --- a/.github/workflows/trivy.yaml +++ b/.github/workflows/trivy.yaml @@ -3,48 +3,46 @@ name: Image Extracting and Security Scanning on: push: branches: - - master + - master jobs: image-extraction-and-security-scan: runs-on: ubuntu-latest steps: - - name: Checkout code - uses: actions/checkout@v3 - - # Install kustomize - - name: Install kustomize - run: | - sudo apt update - sudo apt install snapd - sudo snap install kustomize - - # Install trivy - - name: Install trivy - run: | - sudo apt update - sudo snap install trivy - - # Install Python - - name: Setup Python - uses: actions/setup-python@v5 - with: - python-version: '3.12' - - # Install prettytable package - - name: Install prettytable - run: | - pip install prettytable - - - name: Run image extracting and security scanning script - run: | - cd hack - python3 trivy_scan.py - - # - name: Upload trivy scanned_results - # uses: actions/upload-artifact@v4 - # with: - # name: trivy_scanned_results - # path: image_lists/ - - + - name: Checkout code + uses: actions/checkout@v3 + + # Install kustomize + - name: Install kustomize + run: | + sudo apt update + sudo apt install snapd + sudo snap install kustomize + + # Install trivy + - name: Install trivy + run: | + sudo apt update + sudo snap install trivy + + # Install Python + - name: Setup Python + uses: actions/setup-python@v5 + with: + python-version: '3.12' + + # Install prettytable package + - name: Install prettytable + run: | + pip install prettytable + + - name: Run image extracting and security scanning script + run: | + cd hack + python3 trivy_scan.py + + # - name: Upload trivy scanned_results + # uses: actions/upload-artifact@v4 + # with: + # name: trivy_scanned_results + # path: image_lists/ diff --git a/.github/workflows/volumes_web_application_test.yaml b/.github/workflows/volumes_web_application_test.yaml index ee832b0aab..4165b90faa 100644 --- a/.github/workflows/volumes_web_application_test.yaml +++ b/.github/workflows/volumes_web_application_test.yaml @@ -2,13 +2,13 @@ name: Build & Apply VWA manifests in KinD on: pull_request: paths: - - .github/workflows/volumes_web_application_test.yaml - - apps/volumes-web-app/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - - tests/gh-actions/install_istio.sh - - common/istio*/** + - .github/workflows/volumes_web_application_test.yaml + - apps/volumes-web-app/upstream/** + - tests/gh-actions/kind-cluster.yaml + - tests/gh-actions/install_kind.sh + - tests/gh-actions/install_kustomize.sh + - tests/gh-actions/install_istio.sh + - common/istio*/** jobs: build: diff --git a/README.md b/README.md index e88f320ed7..46ba457a36 100644 --- a/README.md +++ b/README.md @@ -224,7 +224,7 @@ The oauth2-proxy extends your Istio Ingress-Gateway capabilities, to be able to ```sh echo "Installing oauth2-proxy..." -kustomize build common/oidc-client/oauth2-proxy/overlays/m2m-self-signed/ | kubectl apply -f - +kustomize build common/oauth2-proxy/overlays/m2m-self-signed/ | kubectl apply -f - kubectl wait --for=condition=ready pod -l 'app.kubernetes.io/name=oauth2-proxy' --timeout=180s -n oauth2-proxy ``` @@ -561,6 +561,6 @@ The Kubeflow security working group follows a responsible disclosure policy for ## Frequently Asked Questions - **Q:** What versions of Istio, Knative, Cert-Manager, Argo, ... are compatible with Kubeflow? \ - **A:** Please refer to each individual component's documentation for a dependency compatibility range. For Istio, Knative, Dex, Cert-Manager and OIDC-AuthService, the versions in `common` are the ones we have validated. + **A:** Please refer to each individual component's documentation for a dependency compatibility range. For Istio, Knative, Dex, Cert-Manager and OAuth2 Proxy, the versions in `common` are the ones we have validated. - **Q:** Can I use earlier version of Kustomize with Kubeflow manifests? - **A:** No, it is not supported anymore, although it might be possible with manual effort. \ No newline at end of file + **A:** No, it is not supported anymore, although it might be possible with manual effort. diff --git a/apps/centraldashboard/upstream/overlays/oauth2-proxy/kustomization.yaml b/apps/centraldashboard/overlays/oauth2-proxy/kustomization.yaml similarity index 62% rename from apps/centraldashboard/upstream/overlays/oauth2-proxy/kustomization.yaml rename to apps/centraldashboard/overlays/oauth2-proxy/kustomization.yaml index 7115c8893a..bdd73a2f6d 100644 --- a/apps/centraldashboard/upstream/overlays/oauth2-proxy/kustomization.yaml +++ b/apps/centraldashboard/overlays/oauth2-proxy/kustomization.yaml @@ -3,7 +3,7 @@ kind: Kustomization resources: # Using kserve overlay because it's also used in example installation. -- ../kserve +- ../../upstream/overlays/kserve components: -- ../../../../../common/oidc-client/oauth2-proxy/components/central-dashboard +- ../../../../common/oauth2-proxy/components/central-dashboard diff --git a/common/dex/base/config-map.yaml b/common/dex/base/config-map.yaml index 70ce4002df..a71484f3eb 100644 --- a/common/dex/base/config-map.yaml +++ b/common/dex/base/config-map.yaml @@ -25,6 +25,6 @@ data: staticClients: # https://github.com/dexidp/dex/pull/1664 - idEnv: OIDC_CLIENT_ID - redirectURIs: ["/authservice/oidc/callback"] + redirectURIs: ["/oauth2/callback"] name: 'Dex Login Application' secretEnv: OIDC_CLIENT_SECRET diff --git a/common/istio-1-22/istio-install/overlays/oauth2-proxy/kustomization.yaml b/common/istio-1-22/istio-install/overlays/oauth2-proxy/kustomization.yaml index 086e1bdb88..4f9b7365be 100644 --- a/common/istio-1-22/istio-install/overlays/oauth2-proxy/kustomization.yaml +++ b/common/istio-1-22/istio-install/overlays/oauth2-proxy/kustomization.yaml @@ -5,4 +5,4 @@ resources: - ../../base components: -- ../../../../oidc-client/oauth2-proxy/components/istio-external-auth-patches +- ../../../../oauth2-proxy/components/istio-external-auth-patches diff --git a/common/istio-cni-1-22/istio-install/overlays/oauth2-proxy/kustomization.yaml b/common/istio-cni-1-22/istio-install/overlays/oauth2-proxy/kustomization.yaml index 086e1bdb88..4f9b7365be 100644 --- a/common/istio-cni-1-22/istio-install/overlays/oauth2-proxy/kustomization.yaml +++ b/common/istio-cni-1-22/istio-install/overlays/oauth2-proxy/kustomization.yaml @@ -5,4 +5,4 @@ resources: - ../../base components: -- ../../../../oidc-client/oauth2-proxy/components/istio-external-auth-patches +- ../../../../oauth2-proxy/components/istio-external-auth-patches diff --git a/common/oidc-client/oauth2-proxy/OWNERS b/common/oauth2-proxy/OWNERS similarity index 100% rename from common/oidc-client/oauth2-proxy/OWNERS rename to common/oauth2-proxy/OWNERS diff --git a/common/oidc-client/README.md b/common/oauth2-proxy/README.md similarity index 99% rename from common/oidc-client/README.md rename to common/oauth2-proxy/README.md index 6a8986d940..12b02fd0ae 100644 --- a/common/oidc-client/README.md +++ b/common/oauth2-proxy/README.md @@ -1,4 +1,4 @@ -# Kubeflow Authentication using OIDC +# Kubeflow Authentication using Oauth2 Proxy ## Istio Envoy Filter @@ -132,4 +132,4 @@ This is based on the following: [^3]: [oauth2-proxy](https://github.com/oauth2-proxy/oauth2-proxy) [^4]: [Kubernetes TokenReview](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-review-v1/) [^5]: [Kubernetes SubjectAccessReview](https://kubernetes.io/docs/reference/kubernetes-api/authorization-resources/subject-access-review-v3/) -[^6]: [Kube RBAC Proxy](https://github.com/brancz/kube-rbac-proxy) +[^6]: [Kube RBAC Proxy](https://github.com/brancz/kube-rbac-proxy) \ No newline at end of file diff --git a/common/oidc-client/oauth2-proxy/base/README.md b/common/oauth2-proxy/base/README.md similarity index 100% rename from common/oidc-client/oauth2-proxy/base/README.md rename to common/oauth2-proxy/base/README.md diff --git a/common/oauth2-proxy/base/deployment.yaml b/common/oauth2-proxy/base/deployment.yaml new file mode 100644 index 0000000000..eba3f09429 --- /dev/null +++ b/common/oauth2-proxy/base/deployment.yaml @@ -0,0 +1,85 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: oauth2-proxy + labels: + app: oauth2-proxy +spec: + replicas: 2 + selector: + matchLabels: + app.kubernetes.io/name: oauth2-proxy + template: + metadata: + labels: + app.kubernetes.io/name: oauth2-proxy + spec: + volumes: + - name: configmain + configMap: + name: oauth2-proxy + defaultMode: 420 + containers: + - name: oauth2-proxy + image: quay.io/oauth2-proxy/oauth2-proxy:latest + args: + - --http-address=0.0.0.0:4180 + - --config=/etc/oauth2_proxy/oauth2_proxy.cfg + ports: + - name: http + containerPort: 4180 + protocol: TCP + - name: metrics + containerPort: 44180 + protocol: TCP + env: + - name: OAUTH2_PROXY_CLIENT_ID + valueFrom: + secretKeyRef: + name: oauth2-proxy + key: client-id + - name: OAUTH2_PROXY_CLIENT_SECRET + valueFrom: + secretKeyRef: + name: oauth2-proxy + key: client-secret + - name: OAUTH2_PROXY_COOKIE_SECRET + valueFrom: + secretKeyRef: + name: oauth2-proxy + key: cookie-secret + - name: OAUTH2_PROXY_COOKIE_SECURE + valueFrom: + configMapKeyRef: + name: oauth2-proxy-parameters + key: FORCE_HTTPS + - name: OAUTH2_PROXY_SSL_INSECURE_SKIP_VERIFY + valueFrom: + configMapKeyRef: + name: oauth2-proxy-parameters + key: ALLOW_SELF_SIGNED_ISSUER + - name: OAUTH2_PROXY_SKIP_JWT_BEARER_TOKENS + valueFrom: + configMapKeyRef: + name: oauth2-proxy-parameters + key: ENABLE_M2M_TOKENS + - name: OAUTH2_PROXY_EXTRA_JWT_ISSUERS + valueFrom: + configMapKeyRef: + name: oauth2-proxy-parameters + key: EXTRA_JWT_ISSUERS + volumeMounts: + - name: configmain + mountPath: /etc/oauth2_proxy/oauth2_proxy.cfg + subPath: oauth2_proxy.cfg + livenessProbe: + httpGet: + path: /ping + port: http + scheme: HTTP + readinessProbe: + httpGet: + path: /ping + port: http + scheme: HTTP + resources: {} diff --git a/common/oidc-client/oauth2-proxy/base/kustomization.yaml b/common/oauth2-proxy/base/kustomization.yaml similarity index 96% rename from common/oidc-client/oauth2-proxy/base/kustomization.yaml rename to common/oauth2-proxy/base/kustomization.yaml index 94e959bd27..0e470f0603 100644 --- a/common/oidc-client/oauth2-proxy/base/kustomization.yaml +++ b/common/oauth2-proxy/base/kustomization.yaml @@ -48,7 +48,7 @@ configMapGenerator: # If extra jwt issuers are meant to be used with m2m bearer tokens, # each issuer has to be added with RequestAuthentication so Istio can # verify, trust and use the JWT. See - # 'common/oidc-client/oauth2-proxy/components/istio-m2m' for details. + # 'common/oauth2-proxy/components/istio-m2m' for details. # Examples: # - EXTRA_JWT_ISSUERS=https://kubernetes.default.svc.cluster.local=https://kubernetes.default.svc.cluster.local # - EXTRA_JWT_ISSUERS=https://oidc.eks.region.amazonaws.com/id/1234abcd=https://kubernetes.default.svc diff --git a/common/oidc-client/oauth2-proxy/base/namespace.yaml b/common/oauth2-proxy/base/namespace.yaml similarity index 100% rename from common/oidc-client/oauth2-proxy/base/namespace.yaml rename to common/oauth2-proxy/base/namespace.yaml diff --git a/common/oidc-client/oauth2-proxy/base/oauth2-proxy-config.yaml b/common/oauth2-proxy/base/oauth2-proxy-config.yaml similarity index 100% rename from common/oidc-client/oauth2-proxy/base/oauth2-proxy-config.yaml rename to common/oauth2-proxy/base/oauth2-proxy-config.yaml diff --git a/common/oidc-client/oauth2-proxy/base/params.yaml b/common/oauth2-proxy/base/params.yaml similarity index 100% rename from common/oidc-client/oauth2-proxy/base/params.yaml rename to common/oauth2-proxy/base/params.yaml diff --git a/common/oidc-client/oauth2-proxy/base/rbac.tokenreviews.yaml b/common/oauth2-proxy/base/rbac.tokenreviews.yaml similarity index 66% rename from common/oidc-client/oauth2-proxy/base/rbac.tokenreviews.yaml rename to common/oauth2-proxy/base/rbac.tokenreviews.yaml index 4da90d4b4e..1b071279e8 100644 --- a/common/oidc-client/oauth2-proxy/base/rbac.tokenreviews.yaml +++ b/common/oauth2-proxy/base/rbac.tokenreviews.yaml @@ -3,12 +3,12 @@ kind: ClusterRole metadata: name: authn-delegator rules: - - apiGroups: - - authentication.k8s.io - resources: - - tokenreviews - verbs: - - create +- apiGroups: + - authentication.k8s.io + resources: + - tokenreviews + verbs: + - create --- apiVersion: rbac.authorization.k8s.io/v1 @@ -20,5 +20,5 @@ roleRef: kind: ClusterRole name: authn-delegator subjects: - - kind: ServiceAccount - name: oauth2-proxy +- kind: ServiceAccount + name: oauth2-proxy diff --git a/common/oidc-client/oauth2-proxy/base/service.yaml b/common/oauth2-proxy/base/service.yaml similarity index 76% rename from common/oidc-client/oauth2-proxy/base/service.yaml rename to common/oauth2-proxy/base/service.yaml index b8ee06324d..260213a63a 100644 --- a/common/oidc-client/oauth2-proxy/base/service.yaml +++ b/common/oauth2-proxy/base/service.yaml @@ -7,7 +7,7 @@ spec: selector: app.kubernetes.io/name: oauth2-proxy ports: - - port: 80 - name: http - targetPort: http + - port: 80 + name: http + targetPort: http publishNotReadyAddresses: true diff --git a/common/oidc-client/oauth2-proxy/base/serviceaccount.yaml b/common/oauth2-proxy/base/serviceaccount.yaml similarity index 100% rename from common/oidc-client/oauth2-proxy/base/serviceaccount.yaml rename to common/oauth2-proxy/base/serviceaccount.yaml diff --git a/common/oidc-client/oauth2-proxy/README.md b/common/oauth2-proxy/components/README.md similarity index 97% rename from common/oidc-client/oauth2-proxy/README.md rename to common/oauth2-proxy/components/README.md index caa0aab16f..aedbba080f 100644 --- a/common/oidc-client/oauth2-proxy/README.md +++ b/common/oauth2-proxy/components/README.md @@ -177,9 +177,9 @@ make the following changes to the `example/kustomization.yaml` file: bearer tokens with self-signed in-cluster issuer ``` # from - - ../common/oidc-client/oidc-authservice/base + - ../common//oidc-authservice/base # to - - ../common/oidc-client/oauth2-proxy/overlays/m2m-self-signed + - ../common/oauth2-proxy/overlays/m2m-self-signed ``` * change Dex overlay ``` @@ -192,7 +192,7 @@ make the following changes to the `example/kustomization.yaml` file: # from - ../apps/centraldashboard/upstream/overlays/kserve # to - - ../apps/centraldashboard/upstream/overlays/oauth2-proxy + - ../apps/centraldashboard/manuel-patches/overlays/oauth2-proxy ``` All those changes combined can be done with this single command: @@ -208,10 +208,10 @@ index c1a85789..4a50440c 100644 - ../common/istio-1-22/istio-namespace/base -- ../common/istio-1-22/istio-install/base -# OIDC Authservice --- ../common/oidc-client/oidc-authservice/base +-- ../common//oidc-authservice/base +- ../common/istio-1-22/istio-install/overlays/oauth2-proxy +# oauth2-proxy for OIDC -+- ../common/oidc-client/oauth2-proxy/overlays/m2m-self-signed ++- ../common/oauth2-proxy/overlays/m2m-self-signed # Dex -- ../common/dex/overlays/istio +- ../common/dex/overlays/oauth2-proxy @@ -223,7 +223,7 @@ index c1a85789..4a50440c 100644 - ../apps/katib/upstream/installs/katib-with-kubeflow # Central Dashboard -- ../apps/centraldashboard/upstream/overlays/kserve -+- ../apps/centraldashboard/upstream/overlays/oauth2-proxy ++- ../apps/centraldashboard/overlays # Admission Webhook - ../apps/admission-webhook/upstream/overlays/cert-manager # Jupyter Web App diff --git a/common/oidc-client/oauth2-proxy/components/allow-unauthenticated-issuer-discovery/README.md b/common/oauth2-proxy/components/allow-unauthenticated-issuer-discovery/README.md similarity index 100% rename from common/oidc-client/oauth2-proxy/components/allow-unauthenticated-issuer-discovery/README.md rename to common/oauth2-proxy/components/allow-unauthenticated-issuer-discovery/README.md diff --git a/common/oidc-client/oauth2-proxy/components/allow-unauthenticated-issuer-discovery/clusterrolebinding.unauthenticated-oidc-viewer.yaml b/common/oauth2-proxy/components/allow-unauthenticated-issuer-discovery/clusterrolebinding.unauthenticated-oidc-viewer.yaml similarity index 100% rename from common/oidc-client/oauth2-proxy/components/allow-unauthenticated-issuer-discovery/clusterrolebinding.unauthenticated-oidc-viewer.yaml rename to common/oauth2-proxy/components/allow-unauthenticated-issuer-discovery/clusterrolebinding.unauthenticated-oidc-viewer.yaml diff --git a/common/oidc-client/oauth2-proxy/components/allow-unauthenticated-issuer-discovery/kustomization.yaml b/common/oauth2-proxy/components/allow-unauthenticated-issuer-discovery/kustomization.yaml similarity index 100% rename from common/oidc-client/oauth2-proxy/components/allow-unauthenticated-issuer-discovery/kustomization.yaml rename to common/oauth2-proxy/components/allow-unauthenticated-issuer-discovery/kustomization.yaml diff --git a/common/oidc-client/oauth2-proxy/components/central-dashboard/kustomization.yaml b/common/oauth2-proxy/components/central-dashboard/kustomization.yaml similarity index 100% rename from common/oidc-client/oauth2-proxy/components/central-dashboard/kustomization.yaml rename to common/oauth2-proxy/components/central-dashboard/kustomization.yaml diff --git a/common/oidc-client/oauth2-proxy/components/central-dashboard/patches/deployment.logout-url.yaml b/common/oauth2-proxy/components/central-dashboard/patches/deployment.logout-url.yaml similarity index 68% rename from common/oidc-client/oauth2-proxy/components/central-dashboard/patches/deployment.logout-url.yaml rename to common/oauth2-proxy/components/central-dashboard/patches/deployment.logout-url.yaml index 0d6b504191..bd2605e8d2 100644 --- a/common/oidc-client/oauth2-proxy/components/central-dashboard/patches/deployment.logout-url.yaml +++ b/common/oauth2-proxy/components/central-dashboard/patches/deployment.logout-url.yaml @@ -10,9 +10,7 @@ spec: - name: centraldashboard env: # This successfully logs out the user but the user is not redirected to - # the home page. This is because of custom integration with - # oidc-authservice which will provide response with 'afterLogoutURL'. - # https://github.com/arrikto/oidc-authservice/blob/0c4ea9a/server.go#L509 + # the home page. # https://github.com/kubeflow/kubeflow/blob/c6c4492/components/centraldashboard/public/components/logout-button.js#L50 # Please refresh the page after logging out. - name: LOGOUT_URL diff --git a/common/oidc-client/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/README.md b/common/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/README.md similarity index 100% rename from common/oidc-client/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/README.md rename to common/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/README.md diff --git a/common/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/cronjob.kubeflow-m2m-oidc-configurator.yaml b/common/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/cronjob.kubeflow-m2m-oidc-configurator.yaml new file mode 100644 index 0000000000..639e96cb49 --- /dev/null +++ b/common/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/cronjob.kubeflow-m2m-oidc-configurator.yaml @@ -0,0 +1,39 @@ +apiVersion: batch/v1 +kind: CronJob +metadata: + name: kubeflow-m2m-oidc-configurator + namespace: istio-system +spec: + schedule: '*/5 * * * *' + concurrencyPolicy: Forbid + jobTemplate: + spec: + backoffLimit: 3 + ttlSecondsAfterFinished: 600 + template: + metadata: + labels: {} + spec: + restartPolicy: OnFailure + serviceAccountName: kubeflow-m2m-oidc-configurator + containers: + - image: docker.io/curlimages/curl + name: kubeflow-m2m-oidc-configurator + command: + - /script.sh + envFrom: + - configMapRef: + name: kubeflow-m2m-oidc-configurator-envs + volumeMounts: + - mountPath: /script.sh + name: script + subPath: script.sh + resources: {} + volumes: + - name: script + configMap: + name: kubeflow-m2m-oidc-configurator-script + defaultMode: 0777 + items: + - key: script.sh + path: script.sh diff --git a/common/oidc-client/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/kustomization.yaml b/common/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/kustomization.yaml similarity index 100% rename from common/oidc-client/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/kustomization.yaml rename to common/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/kustomization.yaml diff --git a/common/oidc-client/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/rbac.yaml b/common/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/rbac.yaml similarity index 69% rename from common/oidc-client/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/rbac.yaml rename to common/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/rbac.yaml index 305a8fb5c0..9bb2578515 100644 --- a/common/oidc-client/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/rbac.yaml +++ b/common/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/rbac.yaml @@ -11,13 +11,13 @@ metadata: name: kubeflow-m2m-oidc-configurator namespace: istio-system rules: - - apiGroups: - - security.istio.io - resources: - - requestauthentications - verbs: - - get - - patch +- apiGroups: + - security.istio.io + resources: + - requestauthentications + verbs: + - get + - patch --- apiVersion: rbac.authorization.k8s.io/v1 @@ -30,6 +30,6 @@ roleRef: kind: Role name: kubeflow-m2m-oidc-configurator subjects: - - kind: ServiceAccount - name: kubeflow-m2m-oidc-configurator - namespace: istio-system +- kind: ServiceAccount + name: kubeflow-m2m-oidc-configurator + namespace: istio-system diff --git a/common/oidc-client/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/script.sh b/common/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/script.sh similarity index 100% rename from common/oidc-client/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/script.sh rename to common/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/script.sh diff --git a/common/oidc-client/oauth2-proxy/components/README.md b/common/oauth2-proxy/components/istio-external-auth-patches/README.md similarity index 100% rename from common/oidc-client/oauth2-proxy/components/README.md rename to common/oauth2-proxy/components/istio-external-auth-patches/README.md diff --git a/common/oidc-client/oauth2-proxy/components/istio-external-auth-patches/kustomization.yaml b/common/oauth2-proxy/components/istio-external-auth-patches/kustomization.yaml similarity index 100% rename from common/oidc-client/oauth2-proxy/components/istio-external-auth-patches/kustomization.yaml rename to common/oauth2-proxy/components/istio-external-auth-patches/kustomization.yaml diff --git a/common/oidc-client/oauth2-proxy/components/istio-external-auth-patches/patches/cm.enable-oauth2-proxy.yaml b/common/oauth2-proxy/components/istio-external-auth-patches/patches/cm.enable-oauth2-proxy.yaml similarity index 100% rename from common/oidc-client/oauth2-proxy/components/istio-external-auth-patches/patches/cm.enable-oauth2-proxy.yaml rename to common/oauth2-proxy/components/istio-external-auth-patches/patches/cm.enable-oauth2-proxy.yaml diff --git a/common/oauth2-proxy/components/istio-external-auth-patches/patches/deployment.jwt-refresh-interval.yaml b/common/oauth2-proxy/components/istio-external-auth-patches/patches/deployment.jwt-refresh-interval.yaml new file mode 100644 index 0000000000..a11dfd1282 --- /dev/null +++ b/common/oauth2-proxy/components/istio-external-auth-patches/patches/deployment.jwt-refresh-interval.yaml @@ -0,0 +1,18 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: istiod + namespace: istio-system +spec: + template: + spec: + containers: + - name: discovery + env: + # Istio will refresh the JWT Public Keys provided with + # RequestAuthentication by default every 20 minutes. For deployment + # from scratch this is not ideal because there is a high chance that + # the istiod will be available before dex is available, triggering + # Istio mechanism to use a placeholder jwt until refreshed. + - name: PILOT_JWT_PUB_KEY_REFRESH_INTERVAL + value: "1m" diff --git a/common/oidc-client/oauth2-proxy/components/istio-external-auth-patches/README.md b/common/oauth2-proxy/components/istio-external-auth/README.md similarity index 100% rename from common/oidc-client/oauth2-proxy/components/istio-external-auth-patches/README.md rename to common/oauth2-proxy/components/istio-external-auth/README.md diff --git a/common/oidc-client/oauth2-proxy/components/istio-external-auth/authorizationpolicy.istio-ingressgateway-oauth2-proxy-cloudflare.yaml b/common/oauth2-proxy/components/istio-external-auth/authorizationpolicy.istio-ingressgateway-oauth2-proxy-cloudflare.yaml similarity index 100% rename from common/oidc-client/oauth2-proxy/components/istio-external-auth/authorizationpolicy.istio-ingressgateway-oauth2-proxy-cloudflare.yaml rename to common/oauth2-proxy/components/istio-external-auth/authorizationpolicy.istio-ingressgateway-oauth2-proxy-cloudflare.yaml diff --git a/common/oidc-client/oauth2-proxy/components/istio-external-auth/authorizationpolicy.istio-ingressgateway-oauth2-proxy.yaml b/common/oauth2-proxy/components/istio-external-auth/authorizationpolicy.istio-ingressgateway-oauth2-proxy.yaml similarity index 100% rename from common/oidc-client/oauth2-proxy/components/istio-external-auth/authorizationpolicy.istio-ingressgateway-oauth2-proxy.yaml rename to common/oauth2-proxy/components/istio-external-auth/authorizationpolicy.istio-ingressgateway-oauth2-proxy.yaml diff --git a/common/oidc-client/oauth2-proxy/components/istio-external-auth/kustomization.yaml b/common/oauth2-proxy/components/istio-external-auth/kustomization.yaml similarity index 81% rename from common/oidc-client/oauth2-proxy/components/istio-external-auth/kustomization.yaml rename to common/oauth2-proxy/components/istio-external-auth/kustomization.yaml index ec79597930..cf906f840c 100644 --- a/common/oidc-client/oauth2-proxy/components/istio-external-auth/kustomization.yaml +++ b/common/oauth2-proxy/components/istio-external-auth/kustomization.yaml @@ -5,7 +5,7 @@ resources: - authorizationpolicy.istio-ingressgateway-oauth2-proxy.yaml - requestauthentication.dex-jwt.yaml -# If you're running Kubeflow behind CloudFlare, use +# If you're running Kubeflow behind CloudFlare, use # authorizationpolicy.istio-ingressgateway-oauth2-proxy-cloudflare.yaml -# instead of +# instead of # authorizationpolicy.istio-ingressgateway-oauth2-proxy.yaml diff --git a/common/oauth2-proxy/components/istio-external-auth/requestauthentication.dex-jwt.yaml b/common/oauth2-proxy/components/istio-external-auth/requestauthentication.dex-jwt.yaml new file mode 100644 index 0000000000..55f4565281 --- /dev/null +++ b/common/oauth2-proxy/components/istio-external-auth/requestauthentication.dex-jwt.yaml @@ -0,0 +1,19 @@ +apiVersion: security.istio.io/v1beta1 +kind: RequestAuthentication +metadata: + name: dex-jwt + namespace: istio-system +spec: + jwtRules: + # forwardOriginalToken must be set to true so the authorization header will + # be passed between Kubeflow Components and Istio can configure the + # Kubeflow Auth Headers based on this request authorization header. + - forwardOriginalToken: true + issuer: http://dex.auth.svc.cluster.local:5556/dex + # These 5 lines provides integration of istio/oauth2-proxy with + # Kubeflow custom auth headers. + outputClaimToHeaders: + - header: kubeflow-userid + claim: email + - header: kubeflow-groups + claim: groups diff --git a/common/oidc-client/oauth2-proxy/components/istio-m2m/README.md b/common/oauth2-proxy/components/istio-m2m/README.md similarity index 84% rename from common/oidc-client/oauth2-proxy/components/istio-m2m/README.md rename to common/oauth2-proxy/components/istio-m2m/README.md index 40ec0619b6..750f34de8e 100644 --- a/common/oidc-client/oauth2-proxy/components/istio-m2m/README.md +++ b/common/oauth2-proxy/components/istio-m2m/README.md @@ -13,7 +13,7 @@ the platform-managed Kubernetes OIDC issuer in the `RequestAuthentication` for s and authentication compliance with the platform's security standards. For scenarios where the OIDC issuer is served behind self-signed certificates, the kustomize -overlay using this component should include the `common/oidc-client/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer` +overlay using this component should include the `common/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer` component. This additional configuration is necessary to handle the self-signed nature of the -certificates. This setup is the default in the Kustomize overlay defined in `common/oidc-client/oauth2-proxy/overlays/m2m-self-signed`, -which is tailored for environments with self-signed OIDC issuers. \ No newline at end of file +certificates. This setup is the default in the Kustomize overlay defined in `common/oauth2-proxy/overlays/m2m-self-signed`, +which is tailored for environments with self-signed OIDC issuers. diff --git a/common/oidc-client/oauth2-proxy/components/istio-m2m/kustomization.yaml b/common/oauth2-proxy/components/istio-m2m/kustomization.yaml similarity index 100% rename from common/oidc-client/oauth2-proxy/components/istio-m2m/kustomization.yaml rename to common/oauth2-proxy/components/istio-m2m/kustomization.yaml diff --git a/common/oidc-client/oauth2-proxy/components/istio-m2m/params.yaml b/common/oauth2-proxy/components/istio-m2m/params.yaml similarity index 100% rename from common/oidc-client/oauth2-proxy/components/istio-m2m/params.yaml rename to common/oauth2-proxy/components/istio-m2m/params.yaml diff --git a/common/oauth2-proxy/components/istio-m2m/requestauthentication.yaml b/common/oauth2-proxy/components/istio-m2m/requestauthentication.yaml new file mode 100644 index 0000000000..39563d5ef9 --- /dev/null +++ b/common/oauth2-proxy/components/istio-m2m/requestauthentication.yaml @@ -0,0 +1,14 @@ +apiVersion: security.istio.io/v1beta1 +kind: RequestAuthentication +metadata: + name: m2m-token-issuer + namespace: istio-system +spec: + jwtRules: + - forwardOriginalToken: true + issuer: M2M_TOKEN_ISSUER_PLACEHOLDER + outputClaimToHeaders: + - claim: sub + header: x-auth-request-user + - claim: sub + header: kubeflow-userid diff --git a/common/oidc-client/kubeflow_auth_diagram.svg b/common/oauth2-proxy/components/kubeflow_auth_diagram.svg similarity index 100% rename from common/oidc-client/kubeflow_auth_diagram.svg rename to common/oauth2-proxy/components/kubeflow_auth_diagram.svg diff --git a/common/oidc-client/oauth2-proxy/overlays/m2m-self-signed/kustomization.yaml b/common/oauth2-proxy/overlays/m2m-self-signed/kustomization.yaml similarity index 100% rename from common/oidc-client/oauth2-proxy/overlays/m2m-self-signed/kustomization.yaml rename to common/oauth2-proxy/overlays/m2m-self-signed/kustomization.yaml diff --git a/common/oidc-client/oauth2-proxy/overlays/m2m/README.md b/common/oauth2-proxy/overlays/m2m/README.md similarity index 100% rename from common/oidc-client/oauth2-proxy/overlays/m2m/README.md rename to common/oauth2-proxy/overlays/m2m/README.md diff --git a/common/oidc-client/oauth2-proxy/overlays/m2m/component-overwrite-m2m-token-issuer/kustomization.yaml b/common/oauth2-proxy/overlays/m2m/component-overwrite-m2m-token-issuer/kustomization.yaml similarity index 100% rename from common/oidc-client/oauth2-proxy/overlays/m2m/component-overwrite-m2m-token-issuer/kustomization.yaml rename to common/oauth2-proxy/overlays/m2m/component-overwrite-m2m-token-issuer/kustomization.yaml diff --git a/common/oidc-client/oauth2-proxy/overlays/m2m/kustomization.yaml b/common/oauth2-proxy/overlays/m2m/kustomization.yaml similarity index 100% rename from common/oidc-client/oauth2-proxy/overlays/m2m/kustomization.yaml rename to common/oauth2-proxy/overlays/m2m/kustomization.yaml diff --git a/common/oidc-client/oauth2-proxy/base/deployment.yaml b/common/oidc-client/oauth2-proxy/base/deployment.yaml deleted file mode 100644 index f8ef99b4b7..0000000000 --- a/common/oidc-client/oauth2-proxy/base/deployment.yaml +++ /dev/null @@ -1,85 +0,0 @@ -apiVersion: apps/v1 -kind: Deployment -metadata: - name: oauth2-proxy - labels: - app: oauth2-proxy -spec: - replicas: 2 - selector: - matchLabels: - app.kubernetes.io/name: oauth2-proxy - template: - metadata: - labels: - app.kubernetes.io/name: oauth2-proxy - spec: - volumes: - - name: configmain - configMap: - name: oauth2-proxy - defaultMode: 420 - containers: - - name: oauth2-proxy - image: quay.io/oauth2-proxy/oauth2-proxy:latest - args: - - --http-address=0.0.0.0:4180 - - --config=/etc/oauth2_proxy/oauth2_proxy.cfg - ports: - - name: http - containerPort: 4180 - protocol: TCP - - name: metrics - containerPort: 44180 - protocol: TCP - env: - - name: OAUTH2_PROXY_CLIENT_ID - valueFrom: - secretKeyRef: - name: oauth2-proxy - key: client-id - - name: OAUTH2_PROXY_CLIENT_SECRET - valueFrom: - secretKeyRef: - name: oauth2-proxy - key: client-secret - - name: OAUTH2_PROXY_COOKIE_SECRET - valueFrom: - secretKeyRef: - name: oauth2-proxy - key: cookie-secret - - name: OAUTH2_PROXY_COOKIE_SECURE - valueFrom: - configMapKeyRef: - name: oauth2-proxy-parameters - key: FORCE_HTTPS - - name: OAUTH2_PROXY_SSL_INSECURE_SKIP_VERIFY - valueFrom: - configMapKeyRef: - name: oauth2-proxy-parameters - key: ALLOW_SELF_SIGNED_ISSUER - - name: OAUTH2_PROXY_SKIP_JWT_BEARER_TOKENS - valueFrom: - configMapKeyRef: - name: oauth2-proxy-parameters - key: ENABLE_M2M_TOKENS - - name: OAUTH2_PROXY_EXTRA_JWT_ISSUERS - valueFrom: - configMapKeyRef: - name: oauth2-proxy-parameters - key: EXTRA_JWT_ISSUERS - volumeMounts: - - name: configmain - mountPath: /etc/oauth2_proxy/oauth2_proxy.cfg - subPath: oauth2_proxy.cfg - livenessProbe: - httpGet: - path: /ping - port: http - scheme: HTTP - readinessProbe: - httpGet: - path: /ping - port: http - scheme: HTTP - resources: {} diff --git a/common/oidc-client/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/cronjob.kubeflow-m2m-oidc-configurator.yaml b/common/oidc-client/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/cronjob.kubeflow-m2m-oidc-configurator.yaml deleted file mode 100644 index 3d3fc2c281..0000000000 --- a/common/oidc-client/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/cronjob.kubeflow-m2m-oidc-configurator.yaml +++ /dev/null @@ -1,39 +0,0 @@ -apiVersion: batch/v1 -kind: CronJob -metadata: - name: kubeflow-m2m-oidc-configurator - namespace: istio-system -spec: - schedule: '*/5 * * * *' - concurrencyPolicy: Forbid - jobTemplate: - spec: - backoffLimit: 3 - ttlSecondsAfterFinished: 600 - template: - metadata: - labels: {} - spec: - restartPolicy: OnFailure - serviceAccountName: kubeflow-m2m-oidc-configurator - containers: - - image: docker.io/curlimages/curl - name: kubeflow-m2m-oidc-configurator - command: - - /script.sh - envFrom: - - configMapRef: - name: kubeflow-m2m-oidc-configurator-envs - volumeMounts: - - mountPath: /script.sh - name: script - subPath: script.sh - resources: {} - volumes: - - name: script - configMap: - name: kubeflow-m2m-oidc-configurator-script - defaultMode: 0777 - items: - - key: script.sh - path: script.sh diff --git a/common/oidc-client/oauth2-proxy/components/istio-external-auth-patches/patches/deployment.jwt-refresh-interval.yaml b/common/oidc-client/oauth2-proxy/components/istio-external-auth-patches/patches/deployment.jwt-refresh-interval.yaml deleted file mode 100644 index 96987bc69c..0000000000 --- a/common/oidc-client/oauth2-proxy/components/istio-external-auth-patches/patches/deployment.jwt-refresh-interval.yaml +++ /dev/null @@ -1,18 +0,0 @@ -apiVersion: apps/v1 -kind: Deployment -metadata: - name: istiod - namespace: istio-system -spec: - template: - spec: - containers: - - name: discovery - env: - # Istio will refresh the JWT Public Keys provided with - # RequestAuthentication by default every 20 minutes. For deployment - # from scratch this is not ideal because there is a high chance that - # the istiod will be available before dex is available, triggering - # Istio mechanism to use a placeholder jwt until refreshed. - - name: PILOT_JWT_PUB_KEY_REFRESH_INTERVAL - value: "1m" diff --git a/common/oidc-client/oauth2-proxy/components/istio-external-auth/README.md b/common/oidc-client/oauth2-proxy/components/istio-external-auth/README.md deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/common/oidc-client/oauth2-proxy/components/istio-external-auth/requestauthentication.dex-jwt.yaml b/common/oidc-client/oauth2-proxy/components/istio-external-auth/requestauthentication.dex-jwt.yaml deleted file mode 100644 index 4da92ad8b5..0000000000 --- a/common/oidc-client/oauth2-proxy/components/istio-external-auth/requestauthentication.dex-jwt.yaml +++ /dev/null @@ -1,19 +0,0 @@ -apiVersion: security.istio.io/v1beta1 -kind: RequestAuthentication -metadata: - name: dex-jwt - namespace: istio-system -spec: - jwtRules: - # forwardOriginalToken must be set to true so the authorization header will - # be passed between Kubeflow Components and Istio can configure the - # Kubeflow Auth Headers based on this request authorization header. - - forwardOriginalToken: true - issuer: http://dex.auth.svc.cluster.local:5556/dex - # These 5 lines provides integration of istio/oauth2-proxy with - # Kubeflow custom auth headers. - outputClaimToHeaders: - - header: kubeflow-userid - claim: email - - header: kubeflow-groups - claim: groups diff --git a/common/oidc-client/oauth2-proxy/components/istio-m2m/requestauthentication.yaml b/common/oidc-client/oauth2-proxy/components/istio-m2m/requestauthentication.yaml deleted file mode 100644 index af0b3ecd67..0000000000 --- a/common/oidc-client/oauth2-proxy/components/istio-m2m/requestauthentication.yaml +++ /dev/null @@ -1,14 +0,0 @@ -apiVersion: security.istio.io/v1beta1 -kind: RequestAuthentication -metadata: - name: m2m-token-issuer - namespace: istio-system -spec: - jwtRules: - - forwardOriginalToken: true - issuer: M2M_TOKEN_ISSUER_PLACEHOLDER - outputClaimToHeaders: - - claim: sub - header: x-auth-request-user - - claim: sub - header: kubeflow-userid diff --git a/common/oidc-client/oidc-authservice/base/envoy-filter.yaml b/common/oidc-client/oidc-authservice/base/envoy-filter.yaml deleted file mode 100644 index 63e55a878d..0000000000 --- a/common/oidc-client/oidc-authservice/base/envoy-filter.yaml +++ /dev/null @@ -1,40 +0,0 @@ -apiVersion: networking.istio.io/v1alpha3 -kind: EnvoyFilter -metadata: - name: authn-filter -spec: - workloadSelector: - labels: - istio: ingressgateway - configPatches: - - applyTo: HTTP_FILTER - match: - context: GATEWAY - listener: - filterChain: - filter: - name: "envoy.filters.network.http_connection_manager" - patch: - # For some reason, INSERT_FIRST doesn't work - operation: INSERT_BEFORE - value: - # See: https://www.envoyproxy.io/docs/envoy/v1.17.0/configuration/http/http_filters/ext_authz_filter#config-http-filters-ext-authz - name: "envoy.filters.http.ext_authz" - typed_config: - '@type': type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz - http_service: - server_uri: - uri: http://$(AUTHSERVICE_SERVICE).$(AUTHSERVICE_NAMESPACE).svc.cluster.local - cluster: outbound|8080||$(AUTHSERVICE_SERVICE).$(AUTHSERVICE_NAMESPACE).svc.cluster.local - timeout: 10s - authorization_request: - allowed_headers: - patterns: - # XXX: MUST be lowercase! - - exact: "authorization" - - exact: "cookie" - - exact: "x-auth-token" - authorization_response: - allowed_upstream_headers: - patterns: - - exact: "kubeflow-userid" diff --git a/common/oidc-client/oidc-authservice/base/kustomization.yaml b/common/oidc-client/oidc-authservice/base/kustomization.yaml deleted file mode 100644 index 9509fb44df..0000000000 --- a/common/oidc-client/oidc-authservice/base/kustomization.yaml +++ /dev/null @@ -1,47 +0,0 @@ -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization - -resources: - - rbac.yaml - - service.yaml - - statefulset.yaml - - envoy-filter.yaml - - pvc.yaml - -namespace: istio-system - -configMapGenerator: - - name: oidc-authservice-parameters - envs: - - params.env - -secretGenerator: - - name: oidc-authservice-client - type: Opaque - envs: - - secret_params.env - -generatorOptions: - disableNameSuffixHash: true - -vars: - - name: AUTHSERVICE_NAMESPACE - objref: - kind: Service - name: authservice - apiVersion: v1 - fieldref: - fieldpath: metadata.namespace - - name: AUTHSERVICE_SERVICE - objref: - kind: Service - name: authservice - apiVersion: v1 - fieldref: - fieldpath: metadata.name -configurations: - - params.yaml -images: - - name: gcr.io/arrikto/kubeflow/oidc-authservice - newName: docker.io/kubeflowmanifestswg/oidc-authservice - newTag: e236439 diff --git a/common/oidc-client/oidc-authservice/base/params.env b/common/oidc-client/oidc-authservice/base/params.env deleted file mode 100644 index ab998157fd..0000000000 --- a/common/oidc-client/oidc-authservice/base/params.env +++ /dev/null @@ -1,11 +0,0 @@ -OIDC_PROVIDER=http://dex.auth.svc.cluster.local:5556/dex -OIDC_AUTH_URL=/dex/auth -OIDC_SCOPES=profile email groups -AUTHSERVICE_URL_PREFIX=/authservice/ -SKIP_AUTH_URLS=/dex -AFTER_LOGOUT_URL=/ -USERID_HEADER=kubeflow-userid -USERID_PREFIX= -USERID_CLAIM=email -PORT="8080" -STORE_PATH=/var/lib/authservice/data.db diff --git a/common/oidc-client/oidc-authservice/base/params.yaml b/common/oidc-client/oidc-authservice/base/params.yaml deleted file mode 100644 index fcec4c57c4..0000000000 --- a/common/oidc-client/oidc-authservice/base/params.yaml +++ /dev/null @@ -1,5 +0,0 @@ -varReference: -- path: spec/configPatches/patch/value/typed_config/http_service/server_uri/uri - kind: EnvoyFilter -- path: spec/configPatches/patch/value/typed_config/http_service/server_uri/cluster - kind: EnvoyFilter diff --git a/common/oidc-client/oidc-authservice/base/pvc.yaml b/common/oidc-client/oidc-authservice/base/pvc.yaml deleted file mode 100644 index da2f06ba3a..0000000000 --- a/common/oidc-client/oidc-authservice/base/pvc.yaml +++ /dev/null @@ -1,10 +0,0 @@ -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: authservice-pvc -spec: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 10Gi \ No newline at end of file diff --git a/common/oidc-client/oidc-authservice/base/rbac.yaml b/common/oidc-client/oidc-authservice/base/rbac.yaml deleted file mode 100644 index 8399498c1c..0000000000 --- a/common/oidc-client/oidc-authservice/base/rbac.yaml +++ /dev/null @@ -1,30 +0,0 @@ -apiVersion: v1 -kind: ServiceAccount -metadata: - name: authservice - ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: authn-delegator -rules: - - apiGroups: - - authentication.k8s.io - resources: - - tokenreviews - verbs: - - create - ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding -metadata: - name: authn-delegators -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: authn-delegator -subjects: - - kind: ServiceAccount - name: authservice diff --git a/common/oidc-client/oidc-authservice/base/secret_params.env b/common/oidc-client/oidc-authservice/base/secret_params.env deleted file mode 100644 index dc56c6c90f..0000000000 --- a/common/oidc-client/oidc-authservice/base/secret_params.env +++ /dev/null @@ -1,2 +0,0 @@ -CLIENT_ID=kubeflow-oidc-authservice -CLIENT_SECRET=pUBnBOY80SnXgjibTYM9ZWNzY2xreNGQok diff --git a/common/oidc-client/oidc-authservice/base/service.yaml b/common/oidc-client/oidc-authservice/base/service.yaml deleted file mode 100644 index 8b62830bcf..0000000000 --- a/common/oidc-client/oidc-authservice/base/service.yaml +++ /dev/null @@ -1,13 +0,0 @@ -apiVersion: v1 -kind: Service -metadata: - name: authservice -spec: - type: ClusterIP - selector: - app: authservice - ports: - - port: 8080 - name: http-authservice - targetPort: http-api - publishNotReadyAddresses: true \ No newline at end of file diff --git a/common/oidc-client/oidc-authservice/base/statefulset.yaml b/common/oidc-client/oidc-authservice/base/statefulset.yaml deleted file mode 100644 index 05fbc63c96..0000000000 --- a/common/oidc-client/oidc-authservice/base/statefulset.yaml +++ /dev/null @@ -1,43 +0,0 @@ -apiVersion: apps/v1 -kind: StatefulSet -metadata: - name: oidc-authservice -spec: - replicas: 1 - selector: - matchLabels: - app: authservice - serviceName: authservice - template: - metadata: - annotations: - sidecar.istio.io/inject: "false" - labels: - app: authservice - spec: - serviceAccountName: authservice - containers: - - name: authservice - image: gcr.io/arrikto/kubeflow/oidc-authservice:e236439 - imagePullPolicy: Always - ports: - - name: http-api - containerPort: 8080 - envFrom: - - secretRef: - name: oidc-authservice-client - - configMapRef: - name: oidc-authservice-parameters - volumeMounts: - - name: data - mountPath: /var/lib/authservice - readinessProbe: - httpGet: - path: / - port: 8081 - securityContext: - fsGroup: 111 - volumes: - - name: data - persistentVolumeClaim: - claimName: authservice-pvc diff --git a/common/oidc-client/oidc-authservice/overlays/ibm-storage-config/kustomization.yaml b/common/oidc-client/oidc-authservice/overlays/ibm-storage-config/kustomization.yaml deleted file mode 100644 index 539c28cae4..0000000000 --- a/common/oidc-client/oidc-authservice/overlays/ibm-storage-config/kustomization.yaml +++ /dev/null @@ -1,10 +0,0 @@ -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization -bases: - - ../../base -patchesStrategicMerge: - - statefulset.yaml -images: - - name: busybox - newTag: "latest" - newName: busybox \ No newline at end of file diff --git a/common/oidc-client/oidc-authservice/overlays/ibm-storage-config/statefulset.yaml b/common/oidc-client/oidc-authservice/overlays/ibm-storage-config/statefulset.yaml deleted file mode 100644 index 2e29fc100a..0000000000 --- a/common/oidc-client/oidc-authservice/overlays/ibm-storage-config/statefulset.yaml +++ /dev/null @@ -1,15 +0,0 @@ -apiVersion: apps/v1 -kind: StatefulSet -metadata: - name: oidc-authservice -spec: - template: - spec: - initContainers: - - name: fix-permission - image: busybox - command: ['sh', '-c'] - args: ['chmod -R 777 /var/lib/authservice;'] - volumeMounts: - - mountPath: /var/lib/authservice - name: data diff --git a/example/kustomization.yaml b/example/kustomization.yaml index 3e028a16df..0308d5e602 100644 --- a/example/kustomization.yaml +++ b/example/kustomization.yaml @@ -41,7 +41,7 @@ resources: - ../common/istio-1-22/istio-namespace/base - ../common/istio-1-22/istio-install/overlays/oauth2-proxy # oauth2-proxy -- ../common/oidc-client/oauth2-proxy/overlays/m2m-self-signed +- ../common/oauth2-proxy/overlays/m2m-self-signed # Dex - ../common/dex/overlays/oauth2-proxy # KNative @@ -64,7 +64,7 @@ resources: # Katib - ../apps/katib/upstream/installs/katib-with-kubeflow # Central Dashboard -- ../apps/centraldashboard/upstream/overlays/oauth2-proxy +- ../apps/centraldashboard/overlays/oauth2-proxy # Admission Webhook - ../apps/admission-webhook/upstream/overlays/cert-manager # Jupyter Web App diff --git a/hack/trivy_scan.py b/hack/trivy_scan.py index ff7706bb1b..91da390394 100644 --- a/hack/trivy_scan.py +++ b/hack/trivy_scan.py @@ -34,8 +34,8 @@ "automl": "../apps/katib/upstream/installs", "pipelines": "../apps/pipeline/upstream/env ../apps/kfp-tekton/upstream/env", "training": "../apps/training-operator/upstream/overlays", - "manifests": "../common/cert-manager/cert-manager/base ../common/cert-manager/kubeflow-issuer/base ../common/istio-1-22/istio-crds/base ../common/istio-1-22/istio-namespace/base ../common/istio-1-22/istio-install/overlays/oauth2-proxy ../common/oidc-client/oauth2-proxy/overlays/m2m-self-signed ../common/dex/overlays/oauth2-proxy ../common/knative/knative-serving/overlays/gateways ../common/knative/knative-eventing/base ../common/istio-1-22/cluster-local-gateway/base ../common/kubeflow-namespace/base ../common/kubeflow-roles/base ../common/istio-1-22/kubeflow-istio-resources/base", - "workbenches": "../apps/pvcviewer-controller/upstream/base ../apps/admission-webhook/upstream/overlays ../apps/centraldashboard/upstream/overlays/oauth2-proxy ../apps/jupyter/jupyter-web-app/upstream/overlays ../apps/volumes-web-app/upstream/overlays ../apps/tensorboard/tensorboards-web-app/upstream/overlays ../apps/profiles/upstream/overlays ../apps/jupyter/notebook-controller/upstream/overlays ../apps/tensorboard/tensorboard-controller/upstream/overlays", + "manifests": "../common/cert-manager/cert-manager/base ../common/cert-manager/kubeflow-issuer/base ../common/istio-1-22/istio-crds/base ../common/istio-1-22/istio-namespace/base ../common/istio-1-22/istio-install/overlays/oauth2-proxy ../common/oauth2-proxy/overlays/m2m-self-signed ../common/dex/overlays/oauth2-proxy ../common/knative/knative-serving/overlays/gateways ../common/knative/knative-eventing/base ../common/istio-1-22/cluster-local-gateway/base ../common/kubeflow-namespace/base ../common/kubeflow-roles/base ../common/istio-1-22/kubeflow-istio-resources/base", + "workbenches": "../apps/pvcviewer-controller/upstream/base ../apps/admission-webhook/upstream/overlays ../apps/centraldashboard/overlays ../apps/jupyter/jupyter-web-app/upstream/overlays ../apps/volumes-web-app/upstream/overlays ../apps/tensorboard/tensorboards-web-app/upstream/overlays ../apps/profiles/upstream/overlays ../apps/jupyter/notebook-controller/upstream/overlays ../apps/tensorboard/tensorboard-controller/upstream/overlays", "serving": "../contrib/kserve - ../contrib/kserve/models-web-app/overlays/kubeflow", "model-registry": "../apps/model-registry/upstream", } @@ -403,4 +403,4 @@ def extract_images(version): log("Output saved to:", output_file) log("Severity counts with images respect to WGs are saved in the",ALL_SEVERITY_COUNTS) -log("Scanned Json reports on images are saved in",SCAN_REPORTS_DIR) \ No newline at end of file +log("Scanned Json reports on images are saved in",SCAN_REPORTS_DIR) diff --git a/proposals/20240606-jwt-handling.md b/proposals/20240606-jwt-handling.md index 19db305f74..76839c4288 100644 --- a/proposals/20240606-jwt-handling.md +++ b/proposals/20240606-jwt-handling.md @@ -62,7 +62,7 @@ With the above implementation we move all the logic of handling the JWTs to the This proposal aims to put more focus on keeping and validating `id_tokens` but also bridging to the existing functionality of the backends, to avoid extensive changes. ### Implementation The technical details for the above proposal translate to the following -1. Common Kubeflow manifests, for all components, for configuring Istio for supporting multiple issuers ([Dex](https://github.com/kubeflow/manifests/blob/v1.9-branch/common/oidc-client/oauth2-proxy/components/istio-external-auth/requestauthentication.dex-jwt.yaml) and [K8s-m2m](https://github.com/kubeflow/manifests/blob/v1.9-branch/common/oidc-client/oauth2-proxy/components/istio-m2m/requestauthentication.yaml)), via `RequestAuthentication` objects +1. Common Kubeflow manifests, for all components, for configuring Istio for supporting multiple issuers ([Dex](https://github.com/kubeflow/manifests/blob/v1.9-branch/common/oauth2-proxy/components/istio-external-auth/requestauthentication.dex-jwt.yaml) and [K8s-m2m](https://github.com/kubeflow/manifests/blob/v1.9-branch/common/oauth2-proxy/components/istio-m2m/requestauthentication.yaml)), via `RequestAuthentication` objects 2. `AuthorizationPolicy` objects of components, for allowing access from Istio IngressGateway, will need to be extended for also requiring a JWT 3. Backends that need to be accessible from other user-namespaces will need to have an `AuthorizationPolicy` that allows any request, only if it has a JWT 4. Backends don't need any logic for validating the JWTs and their existence diff --git a/tests/gh-actions/install_istio_with_ext_auth.sh b/tests/gh-actions/install_istio_with_ext_auth.sh index e77839eb10..871fec394d 100755 --- a/tests/gh-actions/install_istio_with_ext_auth.sh +++ b/tests/gh-actions/install_istio_with_ext_auth.sh @@ -12,6 +12,6 @@ kubectl wait --for=condition=Ready pods --all -n istio-system --timeout=300s \ --field-selector=status.phase!=Succeeded echo "Installing oauth2-proxy..." -cd common/oidc-client +cd common/ kustomize build oauth2-proxy/overlays/m2m-self-signed/ | kubectl apply -f - kubectl wait --for=condition=ready pod -l 'app.kubernetes.io/name=oauth2-proxy' --timeout=180s -n oauth2-proxy diff --git a/tests/gh-actions/test_dex_login.py b/tests/gh-actions/test_dex_login.py new file mode 100644 index 0000000000..b027d5ba92 --- /dev/null +++ b/tests/gh-actions/test_dex_login.py @@ -0,0 +1,108 @@ +import re +import requests +import sys +from urllib.parse import urlsplit + +def get_istio_auth_session(url: str, username: str, password: str) -> dict: + """ + Determine if the specified URL is secured by Dex and try to obtain a session cookie. + WARNING: only Dex `staticPasswords` and `LDAP` authentication are currently supported + (we default default to using `staticPasswords` if both are enabled) + + :param url: Kubeflow server URL, including protocol + :param username: Dex `staticPasswords` or `LDAP` username + :param password: Dex `staticPasswords` or `LDAP` password + :return: auth session information + """ + # define the default return object + auth_session = { + "endpoint_url": url, # KF endpoint URL + "redirect_url": None, # KF redirect URL, if applicable + "dex_login_url": None, # Dex login URL (for POST of credentials) + "is_secured": None, # True if KF endpoint is secured + "session_cookie": None # Resulting session cookies in the form "key1=value1; key2=value2" + } + + # use a persistent session (for cookies) + with requests.Session() as s: + + ################ + # Determine if Endpoint is Secured + ################ + resp = s.get(url, allow_redirects=True) + if resp.status_code != 200: + raise RuntimeError( + f"HTTP status code '{resp.status_code}' for GET against: {url}" + ) + + auth_session["redirect_url"] = resp.url + + # if we were NOT redirected, then the endpoint is UNSECURED + if len(resp.history) == 0: + auth_session["is_secured"] = False + return auth_session + else: + auth_session["is_secured"] = True + + ################ + # Get Dex Login URL + ################ + redirect_url_obj = urlsplit(auth_session["redirect_url"]) + + # if we are at `/auth?=xxxx` path, we need to select an auth type + if re.search(r"/auth$", redirect_url_obj.path): + + ####### + # TIP: choose the default auth type by including ONE of the following + ####### + + # OPTION 1: set "staticPasswords" as default auth type + redirect_url_obj = redirect_url_obj._replace( + path=re.sub(r"/auth$", "/auth/local", redirect_url_obj.path) + ) + # OPTION 2: set "ldap" as default auth type + # redirect_url_obj = redirect_url_obj._replace( + # path=re.sub(r"/auth$", "/auth/ldap", redirect_url_obj.path) + # ) + + # if we are at `/auth/xxxx/login` path, then no further action is needed (we can use it for login POST) + if re.search(r"/auth/.*/login$", redirect_url_obj.path): + auth_session["dex_login_url"] = redirect_url_obj.geturl() + + # else, we need to be redirected to the actual login page + else: + # this GET should redirect us to the `/auth/xxxx/login` path + resp = s.get(redirect_url_obj.geturl(), allow_redirects=True) + if resp.status_code != 200: + raise RuntimeError( + f"HTTP status code '{resp.status_code}' for GET against: {redirect_url_obj.geturl()}" + ) + + # set the login url + auth_session["dex_login_url"] = resp.url + + ################ + # Attempt Dex Login + ################ + resp = s.post( + auth_session["dex_login_url"], + data={"login": username, "password": password}, + allow_redirects=True + ) + + return resp.status_code + +KUBEFLOW_ENDPOINT = "http://localhost:8080" +KUBEFLOW_USERNAME = "user@example.com" +KUBEFLOW_PASSWORD = "12341234" + +resp = get_istio_auth_session( + url=KUBEFLOW_ENDPOINT, + username=KUBEFLOW_USERNAME, + password=KUBEFLOW_PASSWORD +) + +if resp == 200: + sys.exit(0) +else: + sys.exit(1) From 1a3d9007041910d0bcc9612ade5b31665c165c43 Mon Sep 17 00:00:00 2001 From: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com> Date: Fri, 2 Aug 2024 13:50:06 +0200 Subject: [PATCH 17/35] disable the broken dex login test for now or fix it (#2827) Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --- .github/workflows/dex_test.yaml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/dex_test.yaml b/.github/workflows/dex_test.yaml index f06bd0e7a4..84ffecea38 100644 --- a/.github/workflows/dex_test.yaml +++ b/.github/workflows/dex_test.yaml @@ -43,4 +43,5 @@ jobs: - name: test dex login run: | pip3 install requests - ./tests/gh-actions/test_dex_login.py + # currently broken + # ./tests/gh-actions/test_dex_login.py From 6d64c3661d48a0476d97e90a35d4ea5d767d8f58 Mon Sep 17 00:00:00 2001 From: Hansini Karunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Date: Mon, 5 Aug 2024 18:47:39 +0530 Subject: [PATCH 18/35] fix issue with linting of YAML files (#2825) * fix issue with linting of YAML files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * delete a file to check the functionality Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Add the after checking the functionality Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Add folders to yaml linting Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * check linting issues Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * check linting issues Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * check linting changes Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * check linting functionality by chnaging different files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * made some fixes in linting YAML file Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * made some fixes in linting YAML file Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * made some fixes in linting YAML file and check the functionality Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * made some fixes in linting YAML file and check the functionality Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Add github, hack and tests folders also for linting Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Check functionality by deleting sh from hack and yaml file common Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Added deleted files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> --------- Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> --- .github/workflows/linting_bash_python_yaml_files.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/linting_bash_python_yaml_files.yaml b/.github/workflows/linting_bash_python_yaml_files.yaml index bd38c1eea4..90556097d0 100644 --- a/.github/workflows/linting_bash_python_yaml_files.yaml +++ b/.github/workflows/linting_bash_python_yaml_files.yaml @@ -71,7 +71,7 @@ jobs: - name: Set up changed files id: changed_files run: | - git diff --name-status origin/master...HEAD | grep -E '^[AM].*\.(yaml|yml)$' | awk '{print $2}' > changed_files_in_PR.txt || true + git diff --name-only --diff-filter=AM origin/master...HEAD | grep -E '^common/.*\.ya?ml$|^example/.*\.ya?ml$|^hack/.*\.ya?ml$|^tests/.*\.ya?ml$|^.github/.*\.ya?ml$' > changed_files_in_PR.txt || true if [ ! -s changed_files_in_PR.txt ]; then echo "No YAML files have changed in this PR." > changed_files_in_PR.txt fi From 7f0eebb10ec10ff147c3fb479312d12ea968ba2a Mon Sep 17 00:00:00 2001 From: Hansini Karunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Date: Mon, 5 Aug 2024 18:57:38 +0530 Subject: [PATCH 19/35] Expose Ray and Seldon to example kustomization.yaml file (#2834) * Expose Ray and Seldon to example kustomization.yaml file Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Add internal documentations for ray and seldon Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> --------- Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> --- example/kustomization.yaml | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/example/kustomization.yaml b/example/kustomization.yaml index 0308d5e602..88c84cb325 100644 --- a/example/kustomization.yaml +++ b/example/kustomization.yaml @@ -90,6 +90,17 @@ resources: - ../contrib/kserve/kserve - ../contrib/kserve/models-web-app/overlays/kubeflow +# Ray and Seldon resources (commented out by default) +# Ray does not support Istio yet and integration is in alpha state. +# Here is the documentation for Ray: https://docs.ray.io/en/latest/ +# Here is the internal documentation for Ray: - ../contrib/ray/README.md +# - ../contrib/ray/kuberay-operator/overlays/kubeflow +# +# Seldon integration notes: Seldon may have specific requirements and limitations. +# Documentation for Seldon: https://docs.seldon.io/projects/seldon-core/en/latest/ +# Here is the internal documentation for Seldon: - ../contrib/seldon/README.md +# - ../contrib/seldon/seldon-core-operator/overlays/application + components: # Pod Security Standards # https://kubernetes.io/docs/concepts/security/pod-security-standards/ From ef73b64e3cfc03cab03b87ab0fcfa8213f86e9d6 Mon Sep 17 00:00:00 2001 From: biswajit-9776 <115724497+biswajit-9776@users.noreply.github.com> Date: Mon, 12 Aug 2024 16:01:41 +0530 Subject: [PATCH 20/35] Added tests to tests/gh-actions to enable baseline and restricted PSS (#2819) * Patched PSS labels to multi_tenancy Signed-off-by: biswajit-9776 * Added script in gh-actions to patch PSS/static/baseline/pacthes Signed-off-by: biswajit-9776 * Added PSS scripts for both baseline and restricted labels of static namespaces and renamed directories Signed-off-by: biswajit-9776 * Added tests to enable PSS in gh-actions Signed-off-by: biswajit-9776 * Added workflow test for PSS labels Signed-off-by: biswajit-9776 * Fixed indentation Signed-off-by: biswajit-9776 --------- Signed-off-by: biswajit-9776 --- .github/workflows/pss_test.yaml | 35 +++++++++++++++++++ .../PSS/static/baseline/kustomization.yaml | 6 ++-- .../{dex-labels.yaml => auth-labels.yaml} | 0 ...o-labels.yaml => istio-system-labels.yaml} | 0 .../PSS/static/restricted/kustomization.yaml | 5 ++- .../{dex-labels.yaml => auth-labels.yaml} | 0 ...o-labels.yaml => istio-system-labels.yaml} | 0 tests/gh-actions/enable_baseline_PSS.sh | 10 ++++++ tests/gh-actions/enable_restricted_PSS.sh | 10 ++++++ 9 files changed, 60 insertions(+), 6 deletions(-) create mode 100644 .github/workflows/pss_test.yaml rename contrib/security/PSS/static/baseline/patches/{dex-labels.yaml => auth-labels.yaml} (100%) rename contrib/security/PSS/static/baseline/patches/{istio-labels.yaml => istio-system-labels.yaml} (100%) rename contrib/security/PSS/static/restricted/patches/{dex-labels.yaml => auth-labels.yaml} (100%) rename contrib/security/PSS/static/restricted/patches/{istio-labels.yaml => istio-system-labels.yaml} (100%) create mode 100755 tests/gh-actions/enable_baseline_PSS.sh create mode 100755 tests/gh-actions/enable_restricted_PSS.sh diff --git a/.github/workflows/pss_test.yaml b/.github/workflows/pss_test.yaml new file mode 100644 index 0000000000..9aded5a7f5 --- /dev/null +++ b/.github/workflows/pss_test.yaml @@ -0,0 +1,35 @@ +name: Appy PSS labels to namespaces +on: + pull_request: + paths: + - .github/workflows/* + - tests/gh-actions/kind-cluster.yaml + - apps/profiles/upstream/** + - common/dex/** + - common/cert-manager/** + - common/oidc-client/oauth2-proxy/** + - common/istio*/** + - tests/gh-actions/install_istio_with_ext_auth.sh + - tests/gh-actions/install_multitenancy.sh + +jobs: + build: + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Install KinD + run: ./tests/gh-actions/install_kind.sh + + - name: Create KinD Cluster + run: kind create cluster --config tests/gh-actions/kind-cluster.yaml + + - name: Install kustomize + run: ./tests/gh-actions/install_kustomize.sh + + - name: Install kubectl + run: ./tests/gh-actions/install_kubectl.sh + + - name: Applying Pod Security Standards baseline levels for static namespaces + run: ./tests/gh-actions/enable_baseline_PSS.sh diff --git a/contrib/security/PSS/static/baseline/kustomization.yaml b/contrib/security/PSS/static/baseline/kustomization.yaml index ec133f809c..4550b10fcb 100644 --- a/contrib/security/PSS/static/baseline/kustomization.yaml +++ b/contrib/security/PSS/static/baseline/kustomization.yaml @@ -3,7 +3,7 @@ kind: Component patches: - path: patches/kubeflow-labels.yaml -- path: patches/istio-labels.yaml +- path: patches/istio-system-labels.yaml - path: patches/cert-manager-labels.yaml -- path: patches/dex-labels.yaml -- path: patches/oauth2-proxy-labels.yaml \ No newline at end of file +- path: patches/auth-labels.yaml +- path: patches/oauth2-proxy-labels.yaml diff --git a/contrib/security/PSS/static/baseline/patches/dex-labels.yaml b/contrib/security/PSS/static/baseline/patches/auth-labels.yaml similarity index 100% rename from contrib/security/PSS/static/baseline/patches/dex-labels.yaml rename to contrib/security/PSS/static/baseline/patches/auth-labels.yaml diff --git a/contrib/security/PSS/static/baseline/patches/istio-labels.yaml b/contrib/security/PSS/static/baseline/patches/istio-system-labels.yaml similarity index 100% rename from contrib/security/PSS/static/baseline/patches/istio-labels.yaml rename to contrib/security/PSS/static/baseline/patches/istio-system-labels.yaml diff --git a/contrib/security/PSS/static/restricted/kustomization.yaml b/contrib/security/PSS/static/restricted/kustomization.yaml index f42ff9746b..4550b10fcb 100644 --- a/contrib/security/PSS/static/restricted/kustomization.yaml +++ b/contrib/security/PSS/static/restricted/kustomization.yaml @@ -3,8 +3,7 @@ kind: Component patches: - path: patches/kubeflow-labels.yaml -- path: patches/istio-labels.yaml +- path: patches/istio-system-labels.yaml - path: patches/cert-manager-labels.yaml -- path: patches/dex-labels.yaml +- path: patches/auth-labels.yaml - path: patches/oauth2-proxy-labels.yaml -- path: patches/istio-labels.yaml diff --git a/contrib/security/PSS/static/restricted/patches/dex-labels.yaml b/contrib/security/PSS/static/restricted/patches/auth-labels.yaml similarity index 100% rename from contrib/security/PSS/static/restricted/patches/dex-labels.yaml rename to contrib/security/PSS/static/restricted/patches/auth-labels.yaml diff --git a/contrib/security/PSS/static/restricted/patches/istio-labels.yaml b/contrib/security/PSS/static/restricted/patches/istio-system-labels.yaml similarity index 100% rename from contrib/security/PSS/static/restricted/patches/istio-labels.yaml rename to contrib/security/PSS/static/restricted/patches/istio-system-labels.yaml diff --git a/tests/gh-actions/enable_baseline_PSS.sh b/tests/gh-actions/enable_baseline_PSS.sh new file mode 100755 index 0000000000..94bbcbad1c --- /dev/null +++ b/tests/gh-actions/enable_baseline_PSS.sh @@ -0,0 +1,10 @@ +#!/bin/bash + +NAMESPACES=("istio-system" "auth" "cert-manager" "oauth2-proxy" "kubeflow") + +for NAMESPACE in "${NAMESPACES[@]}"; do + if kubectl get namespace "$NAMESPACE" >/dev/null 2>&1; then + echo "Patching the PSS-baseline labels for namespace $NAMESPACE..." + kubectl patch namespace $NAMESPACE --patch-file ./contrib/security/PSS/static/baseline/patches/${NAMESPACE}-labels.yaml + fi +done diff --git a/tests/gh-actions/enable_restricted_PSS.sh b/tests/gh-actions/enable_restricted_PSS.sh new file mode 100755 index 0000000000..5c17ce2ea9 --- /dev/null +++ b/tests/gh-actions/enable_restricted_PSS.sh @@ -0,0 +1,10 @@ +#!/bin/bash + +NAMESPACES=("istio-system" "auth" "cert-manager" "oauth2-proxy" "kubeflow") + +for NAMESPACE in "${NAMESPACES[@]}"; do + if kubectl get namespace "$NAMESPACE" >/dev/null 2>&1; then + echo "Patching the PSS-restricted labels for namespace $NAMESPACE..." + kubectl patch namespace $NAMESPACE --patch-file ./contrib/security/PSS/static/restricted/patches/${NAMESPACE}-labels.yaml + fi +done From fcf861f5c150aa10e7298d55f54c5c6b45833bd3 Mon Sep 17 00:00:00 2001 From: biswajit-9776 <115724497+biswajit-9776@users.noreply.github.com> Date: Fri, 16 Aug 2024 18:00:28 +0530 Subject: [PATCH 21/35] Added changes to pss_test.yaml workflow (#2838) * Added changes to pss_test.yaml workflow Signed-off-by: biswajit-9776 * Fixed the directory names Signed-off-by: biswajit-9776 * Fixed typo and added wait commands Signed-off-by: biswajit-9776 * Fixed typos Signed-off-by: biswajit-9776 --------- Signed-off-by: biswajit-9776 --- .github/workflows/pss_test.yaml | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/.github/workflows/pss_test.yaml b/.github/workflows/pss_test.yaml index 9aded5a7f5..2a43707dfb 100644 --- a/.github/workflows/pss_test.yaml +++ b/.github/workflows/pss_test.yaml @@ -1,4 +1,4 @@ -name: Appy PSS labels to namespaces +name: Apply PSS labels to namespaces on: pull_request: paths: @@ -31,5 +31,31 @@ jobs: - name: Install kubectl run: ./tests/gh-actions/install_kubectl.sh + - name: Install all deployments from static namespaces + run: | + kustomize build common/kubeflow-namespace/base | kubectl apply -f - + ./tests/gh-actions/install_cert_manager.sh + ./tests/gh-actions/install_istio_with_ext_auth.sh + kustomize build common/istio-1-22/kubeflow-istio-resources/base | kubectl apply -f - + ./tests/gh-actions/install_multi_tenancy.sh + kustomize build ./common/oauth2-proxy/overlays/m2m-self-signed | kubectl apply -f - + echo "Waiting for all oauth2-proxy pods to become ready..." + kubectl wait --for=condition=ready pod -l 'app.kubernetes.io/name=oauth2-proxy' --timeout=180s -n oauth2-proxy + kustomize build ./common/dex/overlays/oauth2-proxy | kubectl apply -f - + echo "Waiting for pods in auth namespace to become ready..." + kubectl wait --for=condition=Ready pods --all --timeout=180s -n auth + - name: Applying Pod Security Standards baseline levels for static namespaces run: ./tests/gh-actions/enable_baseline_PSS.sh + + - name: Unapplying applied baseline values + run: | + NAMESPACES=("istio-system" "auth" "cert-manager" "oauth2-proxy" "kubeflow") + for NAMESPACE in "${NAMESPACES[@]}"; do + if kubectl get namespace "$NAMESPACE" >/dev/null 2>&1; then + kubectl label namespace $NAMESPACE pod-security.kubernetes.io/enforce- + fi + done + + - name: Applying Pod Security Standards restricted levels for static namespaces + run: ./tests/gh-actions/enable_restricted_PSS.sh From 49d2c9cc1b82971927b73b5b62e8a47ae5f310f6 Mon Sep 17 00:00:00 2001 From: biswajit-9776 <115724497+biswajit-9776@users.noreply.github.com> Date: Fri, 16 Aug 2024 18:02:28 +0530 Subject: [PATCH 22/35] Added securityContext profiles for deployments failing PSS restricted level (#2836) * Added securityContext for profile-controller deployments Signed-off-by: biswajit-9776 * Fixed PSS warnings for containers kfam and manager Signed-off-by: biswajit-9776 * Added securitycontext profiles to dex and oauth2-proxy deployments Signed-off-by: biswajit-9776 * Added seccompProfile for cluster-local-gateway Signed-off-by: biswajit-9776 * Added securityContext to cronjob Signed-off-by: biswajit-9776 * Added securityContext to pipelines pods Signed-off-by: biswajit-9776 * trigger GitHub actions Signed-off-by: biswajit-9776 * trigger GitHub actions Signed-off-by: biswajit-9776 * Undoing changes to cronjob Signed-off-by: biswajit-9776 --------- Signed-off-by: biswajit-9776 --- contrib/security/PSS/patches/cache-server.yaml | 17 +++++++++++++++++ .../PSS/patches/cluster-local-gateway.yaml | 12 ++++++++++++ contrib/security/PSS/patches/dex.yaml | 17 +++++++++++++++++ contrib/security/PSS/patches/kfam.yaml | 17 +++++++++++++++++ .../kubeflow-pipelines-profile-controller.yaml | 17 +++++++++++++++++ contrib/security/PSS/patches/manager.yaml | 17 +++++++++++++++++ .../PSS/patches/metadata-envoy-deployment.yaml | 17 +++++++++++++++++ .../PSS/patches/metadata-grpc-deployment.yaml | 17 +++++++++++++++++ .../security/PSS/patches/metadata-writer.yaml | 17 +++++++++++++++++ contrib/security/PSS/patches/minio.yaml | 17 +++++++++++++++++ .../patches/ml-pipeline-persistenceagent.yaml | 17 +++++++++++++++++ .../patches/ml-pipeline-scheduledworkflow.yaml | 17 +++++++++++++++++ .../security/PSS/patches/ml-pipeline-ui.yaml | 17 +++++++++++++++++ .../PSS/patches/ml-pipeline-viewer-crd.yaml | 17 +++++++++++++++++ .../ml-pipeline-visualizationserver.yaml | 17 +++++++++++++++++ contrib/security/PSS/patches/ml-pipeline.yaml | 17 +++++++++++++++++ contrib/security/PSS/patches/mysql.yaml | 17 +++++++++++++++++ contrib/security/PSS/patches/oauth2-proxy.yaml | 18 ++++++++++++++++++ 18 files changed, 302 insertions(+) create mode 100644 contrib/security/PSS/patches/cache-server.yaml create mode 100644 contrib/security/PSS/patches/cluster-local-gateway.yaml create mode 100644 contrib/security/PSS/patches/dex.yaml create mode 100644 contrib/security/PSS/patches/kfam.yaml create mode 100644 contrib/security/PSS/patches/kubeflow-pipelines-profile-controller.yaml create mode 100644 contrib/security/PSS/patches/manager.yaml create mode 100644 contrib/security/PSS/patches/metadata-envoy-deployment.yaml create mode 100644 contrib/security/PSS/patches/metadata-grpc-deployment.yaml create mode 100644 contrib/security/PSS/patches/metadata-writer.yaml create mode 100644 contrib/security/PSS/patches/minio.yaml create mode 100644 contrib/security/PSS/patches/ml-pipeline-persistenceagent.yaml create mode 100644 contrib/security/PSS/patches/ml-pipeline-scheduledworkflow.yaml create mode 100644 contrib/security/PSS/patches/ml-pipeline-ui.yaml create mode 100644 contrib/security/PSS/patches/ml-pipeline-viewer-crd.yaml create mode 100644 contrib/security/PSS/patches/ml-pipeline-visualizationserver.yaml create mode 100644 contrib/security/PSS/patches/ml-pipeline.yaml create mode 100644 contrib/security/PSS/patches/mysql.yaml create mode 100644 contrib/security/PSS/patches/oauth2-proxy.yaml diff --git a/contrib/security/PSS/patches/cache-server.yaml b/contrib/security/PSS/patches/cache-server.yaml new file mode 100644 index 0000000000..e62a8bcc49 --- /dev/null +++ b/contrib/security/PSS/patches/cache-server.yaml @@ -0,0 +1,17 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: cache-server +spec: + template: + spec: + containers: + - name: server + securityContext: + allowPrivilegeEscalation: false + seccompProfile: + type: RuntimeDefault + runAsNonRoot: true + capabilities: + drop: + - ALL diff --git a/contrib/security/PSS/patches/cluster-local-gateway.yaml b/contrib/security/PSS/patches/cluster-local-gateway.yaml new file mode 100644 index 0000000000..09b890872f --- /dev/null +++ b/contrib/security/PSS/patches/cluster-local-gateway.yaml @@ -0,0 +1,12 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: cluster-local-gateway +spec: + template: + spec: + containers: + - name: istio-proxy + securityContext: + seccompProfile: + type: RuntimeDefault diff --git a/contrib/security/PSS/patches/dex.yaml b/contrib/security/PSS/patches/dex.yaml new file mode 100644 index 0000000000..d82ecba673 --- /dev/null +++ b/contrib/security/PSS/patches/dex.yaml @@ -0,0 +1,17 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: dex +spec: + template: + spec: + containers: + - name: dex + securityContext: + allowPrivilegeEscalation: false + seccompProfile: + type: RuntimeDefault + runAsNonRoot: true + capabilities: + drop: + - ALL diff --git a/contrib/security/PSS/patches/kfam.yaml b/contrib/security/PSS/patches/kfam.yaml new file mode 100644 index 0000000000..cfd999f56a --- /dev/null +++ b/contrib/security/PSS/patches/kfam.yaml @@ -0,0 +1,17 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: profiles-deployment +spec: + template: + spec: + containers: + - name: kfam + securityContext: + allowPrivilegeEscalation: false + seccompProfile: + type: RuntimeDefault + runAsNonRoot: true + capabilities: + drop: + - ALL diff --git a/contrib/security/PSS/patches/kubeflow-pipelines-profile-controller.yaml b/contrib/security/PSS/patches/kubeflow-pipelines-profile-controller.yaml new file mode 100644 index 0000000000..9c29bfb9ab --- /dev/null +++ b/contrib/security/PSS/patches/kubeflow-pipelines-profile-controller.yaml @@ -0,0 +1,17 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: kubeflow-pipelines-profile-controller +spec: + template: + spec: + containers: + - name: profile-controller + securityContext: + allowPrivilegeEscalation: false + seccompProfile: + type: RuntimeDefault + runAsNonRoot: true + capabilities: + drop: + - ALL diff --git a/contrib/security/PSS/patches/manager.yaml b/contrib/security/PSS/patches/manager.yaml new file mode 100644 index 0000000000..cbf3f3579d --- /dev/null +++ b/contrib/security/PSS/patches/manager.yaml @@ -0,0 +1,17 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: profiles-deployment +spec: + template: + spec: + containers: + - name: manager + securityContext: + allowPrivilegeEscalation: false + seccompProfile: + type: RuntimeDefault + runAsNonRoot: true + capabilities: + drop: + - ALL diff --git a/contrib/security/PSS/patches/metadata-envoy-deployment.yaml b/contrib/security/PSS/patches/metadata-envoy-deployment.yaml new file mode 100644 index 0000000000..ff0370c0f5 --- /dev/null +++ b/contrib/security/PSS/patches/metadata-envoy-deployment.yaml @@ -0,0 +1,17 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: metadata-envoy-deployment +spec: + template: + spec: + containers: + - name: container + securityContext: + allowPrivilegeEscalation: false + seccompProfile: + type: RuntimeDefault + runAsNonRoot: true + capabilities: + drop: + - ALL diff --git a/contrib/security/PSS/patches/metadata-grpc-deployment.yaml b/contrib/security/PSS/patches/metadata-grpc-deployment.yaml new file mode 100644 index 0000000000..778f849a08 --- /dev/null +++ b/contrib/security/PSS/patches/metadata-grpc-deployment.yaml @@ -0,0 +1,17 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: metadata-grpc-deployment +spec: + template: + spec: + containers: + - name: container + securityContext: + allowPrivilegeEscalation: false + seccompProfile: + type: RuntimeDefault + runAsNonRoot: true + capabilities: + drop: + - ALL diff --git a/contrib/security/PSS/patches/metadata-writer.yaml b/contrib/security/PSS/patches/metadata-writer.yaml new file mode 100644 index 0000000000..ada39a1bc1 --- /dev/null +++ b/contrib/security/PSS/patches/metadata-writer.yaml @@ -0,0 +1,17 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: metadata-writer +spec: + template: + spec: + containers: + - name: main + securityContext: + allowPrivilegeEscalation: false + seccompProfile: + type: RuntimeDefault + runAsNonRoot: true + capabilities: + drop: + - ALL diff --git a/contrib/security/PSS/patches/minio.yaml b/contrib/security/PSS/patches/minio.yaml new file mode 100644 index 0000000000..0dc2a0e5fb --- /dev/null +++ b/contrib/security/PSS/patches/minio.yaml @@ -0,0 +1,17 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: minio +spec: + template: + spec: + containers: + - name: minio + securityContext: + allowPrivilegeEscalation: false + seccompProfile: + type: RuntimeDefault + runAsNonRoot: true + capabilities: + drop: + - ALL diff --git a/contrib/security/PSS/patches/ml-pipeline-persistenceagent.yaml b/contrib/security/PSS/patches/ml-pipeline-persistenceagent.yaml new file mode 100644 index 0000000000..61aa1f2a69 --- /dev/null +++ b/contrib/security/PSS/patches/ml-pipeline-persistenceagent.yaml @@ -0,0 +1,17 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ml-pipeline-persistenceagent +spec: + template: + spec: + containers: + - name: ml-pipeline-persistenceagent + securityContext: + allowPrivilegeEscalation: false + seccompProfile: + type: RuntimeDefault + runAsNonRoot: true + capabilities: + drop: + - ALL diff --git a/contrib/security/PSS/patches/ml-pipeline-scheduledworkflow.yaml b/contrib/security/PSS/patches/ml-pipeline-scheduledworkflow.yaml new file mode 100644 index 0000000000..72e8773638 --- /dev/null +++ b/contrib/security/PSS/patches/ml-pipeline-scheduledworkflow.yaml @@ -0,0 +1,17 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ml-pipeline-scheduledworkflow +spec: + template: + spec: + containers: + - name: ml-pipeline-scheduledworkflow + securityContext: + allowPrivilegeEscalation: false + seccompProfile: + type: RuntimeDefault + runAsNonRoot: true + capabilities: + drop: + - ALL diff --git a/contrib/security/PSS/patches/ml-pipeline-ui.yaml b/contrib/security/PSS/patches/ml-pipeline-ui.yaml new file mode 100644 index 0000000000..c3ce8228d5 --- /dev/null +++ b/contrib/security/PSS/patches/ml-pipeline-ui.yaml @@ -0,0 +1,17 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ml-pipeline-ui +spec: + template: + spec: + containers: + - name: ml-pipeline-ui + securityContext: + allowPrivilegeEscalation: false + seccompProfile: + type: RuntimeDefault + runAsNonRoot: true + capabilities: + drop: + - ALL diff --git a/contrib/security/PSS/patches/ml-pipeline-viewer-crd.yaml b/contrib/security/PSS/patches/ml-pipeline-viewer-crd.yaml new file mode 100644 index 0000000000..cf52aef9e7 --- /dev/null +++ b/contrib/security/PSS/patches/ml-pipeline-viewer-crd.yaml @@ -0,0 +1,17 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ml-pipeline-viewer-crd +spec: + template: + spec: + containers: + - name: ml-pipeline-viewer-crd + securityContext: + allowPrivilegeEscalation: false + seccompProfile: + type: RuntimeDefault + runAsNonRoot: true + capabilities: + drop: + - ALL diff --git a/contrib/security/PSS/patches/ml-pipeline-visualizationserver.yaml b/contrib/security/PSS/patches/ml-pipeline-visualizationserver.yaml new file mode 100644 index 0000000000..1673b60c66 --- /dev/null +++ b/contrib/security/PSS/patches/ml-pipeline-visualizationserver.yaml @@ -0,0 +1,17 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ml-pipeline-visualizationserver +spec: + template: + spec: + containers: + - name: ml-pipeline-visualizationserver + securityContext: + allowPrivilegeEscalation: false + seccompProfile: + type: RuntimeDefault + runAsNonRoot: true + capabilities: + drop: + - ALL diff --git a/contrib/security/PSS/patches/ml-pipeline.yaml b/contrib/security/PSS/patches/ml-pipeline.yaml new file mode 100644 index 0000000000..67fc9a41f8 --- /dev/null +++ b/contrib/security/PSS/patches/ml-pipeline.yaml @@ -0,0 +1,17 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ml-pipeline +spec: + template: + spec: + containers: + - name: ml-pipeline-api-server + securityContext: + allowPrivilegeEscalation: false + seccompProfile: + type: RuntimeDefault + runAsNonRoot: true + capabilities: + drop: + - ALL diff --git a/contrib/security/PSS/patches/mysql.yaml b/contrib/security/PSS/patches/mysql.yaml new file mode 100644 index 0000000000..257264d6a3 --- /dev/null +++ b/contrib/security/PSS/patches/mysql.yaml @@ -0,0 +1,17 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: mysql +spec: + template: + spec: + containers: + - name: mysql + securityContext: + allowPrivilegeEscalation: false + seccompProfile: + type: RuntimeDefault + runAsNonRoot: true + capabilities: + drop: + - ALL diff --git a/contrib/security/PSS/patches/oauth2-proxy.yaml b/contrib/security/PSS/patches/oauth2-proxy.yaml new file mode 100644 index 0000000000..007e4d8bdc --- /dev/null +++ b/contrib/security/PSS/patches/oauth2-proxy.yaml @@ -0,0 +1,18 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: oauth2-proxy +spec: + replicas: 2 + template: + spec: + containers: + - name: oauth2-proxy + securityContext: + allowPrivilegeEscalation: false + seccompProfile: + type: RuntimeDefault + runAsNonRoot: true + capabilities: + drop: + - ALL From bf3dd8878af2a52480e60ee90d650a098eacd113 Mon Sep 17 00:00:00 2001 From: biswajit-9776 <115724497+biswajit-9776@users.noreply.github.com> Date: Mon, 19 Aug 2024 13:03:55 +0530 Subject: [PATCH 23/35] Refactor .github/workflows/pss_test.yaml (#2845) * Added pipelines to PSS test and fixed typo Signed-off-by: biswajit-9776 * Added dynamic namespaces to PSS test Signed-off-by: biswajit-9776 --------- Signed-off-by: biswajit-9776 --- .github/workflows/pss_test.yaml | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/.github/workflows/pss_test.yaml b/.github/workflows/pss_test.yaml index 2a43707dfb..984df974b1 100644 --- a/.github/workflows/pss_test.yaml +++ b/.github/workflows/pss_test.yaml @@ -5,9 +5,10 @@ on: - .github/workflows/* - tests/gh-actions/kind-cluster.yaml - apps/profiles/upstream/** + - apps/pipeline/upstream/** - common/dex/** - common/cert-manager/** - - common/oidc-client/oauth2-proxy/** + - common/oauth2-proxy/** - common/istio*/** - tests/gh-actions/install_istio_with_ext_auth.sh - tests/gh-actions/install_multitenancy.sh @@ -45,9 +46,26 @@ jobs: echo "Waiting for pods in auth namespace to become ready..." kubectl wait --for=condition=Ready pods --all --timeout=180s -n auth + - name: Install KF Pipelines + run: ./tests/gh-actions/install_pipelines.sh + - name: Applying Pod Security Standards baseline levels for static namespaces run: ./tests/gh-actions/enable_baseline_PSS.sh + - name: Applying Pod Security Standards baseline levels for dynamic namespaces + run: | + cat << EOF > ./kustomization.yaml + apiVersion: kustomize.config.k8s.io/v1beta1 + kind: Kustomization + resources: + - apps/profiles/upstream/overlays/kubeflow + components: + - contrib/security/PSS/dynamic/baseline + EOF + kubectl apply -k . + rm ./kustomization.yaml + kubectl -n kubeflow wait --for=condition=Ready pods -l kustomize.component=profiles --timeout 180s + - name: Unapplying applied baseline values run: | NAMESPACES=("istio-system" "auth" "cert-manager" "oauth2-proxy" "kubeflow") From e04bd397a466ed474d8a3074b03a24a3755fddcb Mon Sep 17 00:00:00 2001 From: Hansini Karunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Date: Thu, 22 Aug 2024 03:48:57 +0530 Subject: [PATCH 24/35] Consolidate Kubeflow tests (#2805) * Training operator CICD improvements (#2779) * Add the networkpolicies Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * rework the training operator tests Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * fix the comments Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * fix filename Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * try to fix the permissions Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * try to fix the permissions Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * change to the user namespace Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * update the image to rc.1 Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * fixes Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * fixes Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * fixes Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * fixes Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * fixes Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --------- Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Signed-off-by: hansinikarunarathne * create seperate file for install_KinD_create_KinD_cluster_install_kustomize.sh to avoid code repitition Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Signed-off-by: hansinikarunarathne * make sh file executable Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Signed-off-by: hansinikarunarathne * remove reduntant codes Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Signed-off-by: hansinikarunarathne * combine kind, kuztomize and kind-cluster create sh files into one file Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Signed-off-by: hansinikarunarathne * fix issues with combined file Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Signed-off-by: hansinikarunarathne * fix configurations issues Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Signed-off-by: hansinikarunarathne * fix linting issues in workflow files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Signed-off-by: hansinikarunarathne * fix trailing issues in workflow files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Signed-off-by: hansinikarunarathne * fix trailing issues in workflow file bentoml Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Signed-off-by: hansinikarunarathne * made tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh executable and removed chmod command from the workflow files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Signed-off-by: hansinikarunarathne * made modifications in workflow files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Signed-off-by: hansinikarunarathne * deleted redundant files from tests folder Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Signed-off-by: hansinikarunarathne * Add kustomize installation file Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Signed-off-by: hansinikarunarathne * made install_kustomize.sh file executable Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Signed-off-by: hansinikarunarathne * made a fix Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Signed-off-by: hansinikarunarathne * fix issue with linting of YAML files (#2825) * fix issue with linting of YAML files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * delete a file to check the functionality Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Add the after checking the functionality Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Add folders to yaml linting Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * check linting issues Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * check linting issues Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * check linting changes Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * check linting functionality by chnaging different files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * made some fixes in linting YAML file Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * made some fixes in linting YAML file Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * made some fixes in linting YAML file and check the functionality Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * made some fixes in linting YAML file and check the functionality Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Add github, hack and tests folders also for linting Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Check functionality by deleting sh from hack and yaml file common Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Added deleted files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> --------- Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Signed-off-by: hansinikarunarathne * Expose Ray and Seldon to example kustomization.yaml file (#2834) * Expose Ray and Seldon to example kustomization.yaml file Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Add internal documentations for ray and seldon Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> --------- Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Signed-off-by: hansinikarunarathne * Increase the time out of notebook and katib test Signed-off-by: hansinikarunarathne * increase the timout time Signed-off-by: hansinikarunarathne * increase timeout of the create pipeline Run from Kubeflow Notebook to 2400s Signed-off-by: hansinikarunarathne * Add code to calculate time taken to pull a docker image Signed-off-by: hansinikarunarathne * fix syntax error in yaml file Signed-off-by: hansinikarunarathne * increase pipeline time Signed-off-by: hansinikarunarathne * remove one step in action Signed-off-by: hansinikarunarathne * Added tests to tests/gh-actions to enable baseline and restricted PSS (#2819) * Patched PSS labels to multi_tenancy Signed-off-by: biswajit-9776 * Added script in gh-actions to patch PSS/static/baseline/pacthes Signed-off-by: biswajit-9776 * Added PSS scripts for both baseline and restricted labels of static namespaces and renamed directories Signed-off-by: biswajit-9776 * Added tests to enable PSS in gh-actions Signed-off-by: biswajit-9776 * Added workflow test for PSS labels Signed-off-by: biswajit-9776 * Fixed indentation Signed-off-by: biswajit-9776 --------- Signed-off-by: biswajit-9776 Signed-off-by: hansinikarunarathne * add kind cluster step into pss test yaml file Signed-off-by: hansinikarunarathne * change time pot time Signed-off-by: hansinikarunarathne --------- Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Signed-off-by: hansinikarunarathne Signed-off-by: biswajit-9776 Co-authored-by: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com> Co-authored-by: biswajit-9776 <115724497+biswajit-9776@users.noreply.github.com> --- .github/workflows/admission_webhook_test.yaml | 14 +--- .github/workflows/bentoml_test.yaml | 14 +--- .github/workflows/centraldashboard_test.yaml | 14 +--- .github/workflows/dex_test.yaml | 14 +--- .../jupyter_web_application_test.yaml | 14 +--- .github/workflows/katib_test.yaml | 16 ++--- .github/workflows/kserve_cni_test.yaml | 14 +--- .github/workflows/kserve_m2m_test.yaml | 16 ++--- .github/workflows/kserve_test.yaml | 14 +--- .github/workflows/metacontroller_test.yaml | 14 +--- .github/workflows/model_registry_test.yaml | 13 +--- .../notebook_controller_m2m_test.yaml | 16 ++--- .../workflows/notebook_controller_test.yaml | 14 +--- .../workflows/pipeline_run_from_notebook.yaml | 20 ++---- .github/workflows/pipeline_test.yaml | 16 ++--- .github/workflows/profiles_test.yaml | 14 +--- .github/workflows/pss_test.yaml | 11 +--- .github/workflows/ray_test.yaml | 13 +--- .github/workflows/seldon_test.yaml | 14 +--- .../tensorboard_controller_test.yaml | 14 +--- .../tensorboards_web_application_test.yaml | 14 +--- .github/workflows/training_operator_test.yaml | 14 +--- .../volumes_web_application_test.yaml | 14 +--- ...D_create_KinD_cluster_install_kustomize.sh | 66 +++++++++++++++++++ .../gh-actions/install_istio_with_ext_auth.sh | 2 +- tests/gh-actions/install_kind.sh | 15 ----- tests/gh-actions/kind-cluster.yaml | 26 -------- 27 files changed, 143 insertions(+), 297 deletions(-) create mode 100755 tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh delete mode 100755 tests/gh-actions/install_kind.sh delete mode 100644 tests/gh-actions/kind-cluster.yaml diff --git a/.github/workflows/admission_webhook_test.yaml b/.github/workflows/admission_webhook_test.yaml index 2660a70e7a..fecfb8b5e5 100644 --- a/.github/workflows/admission_webhook_test.yaml +++ b/.github/workflows/admission_webhook_test.yaml @@ -2,11 +2,9 @@ name: Build & Apply PodDefaults manifests in KinD on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/admission_webhook_test.yaml - apps/admission-webhook/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - tests/gh-actions/install_istio.sh - tests/gh-actions/install_cert_manager.sh - common/cert-manager/** @@ -18,14 +16,8 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Install Istio run: ./tests/gh-actions/install_istio.sh diff --git a/.github/workflows/bentoml_test.yaml b/.github/workflows/bentoml_test.yaml index 41a8fc276b..ad234ba55b 100644 --- a/.github/workflows/bentoml_test.yaml +++ b/.github/workflows/bentoml_test.yaml @@ -2,10 +2,8 @@ name: Build & Apply BentoML Yatai Stack manifests in KinD on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/bentoml_test.yaml - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - tests/gh-actions/install_cert_manager.sh - common/cert-manager/** - contrib/bentoml/** @@ -17,14 +15,8 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Install cert-manager run: ./tests/gh-actions/install_cert_manager.sh diff --git a/.github/workflows/centraldashboard_test.yaml b/.github/workflows/centraldashboard_test.yaml index f5373d1277..4ec4c0baf7 100644 --- a/.github/workflows/centraldashboard_test.yaml +++ b/.github/workflows/centraldashboard_test.yaml @@ -2,11 +2,9 @@ name: Build & Apply CentralDashboard manifests in KinD on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/centraldashboard_test.yaml - apps/centraldashboard/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - tests/gh-actions/install_istio.sh jobs: @@ -16,14 +14,8 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Install Istio run: ./tests/gh-actions/install_istio.sh diff --git a/.github/workflows/dex_test.yaml b/.github/workflows/dex_test.yaml index 84ffecea38..a2f51aced8 100644 --- a/.github/workflows/dex_test.yaml +++ b/.github/workflows/dex_test.yaml @@ -2,11 +2,9 @@ name: Build & Apply Dex manifests in KinD on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/dex_test.yaml - common/dex/base/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - tests/gh-actions/install_istio.sh jobs: @@ -16,14 +14,8 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Install Istio run: ./tests/gh-actions/install_istio.sh diff --git a/.github/workflows/jupyter_web_application_test.yaml b/.github/workflows/jupyter_web_application_test.yaml index 13715e1740..f1054d15bb 100644 --- a/.github/workflows/jupyter_web_application_test.yaml +++ b/.github/workflows/jupyter_web_application_test.yaml @@ -2,11 +2,9 @@ name: Build & Apply JWA manifests in KinD on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/jupyter_web_application_test.yaml - apps/jupyter/jupyter-web-app/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - tests/gh-actions/install_istio.sh jobs: @@ -16,14 +14,8 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Install Istio run: ./tests/gh-actions/install_istio.sh diff --git a/.github/workflows/katib_test.yaml b/.github/workflows/katib_test.yaml index 9edace0aa7..841da24e6c 100644 --- a/.github/workflows/katib_test.yaml +++ b/.github/workflows/katib_test.yaml @@ -2,11 +2,9 @@ name: Build & Apply Katib manifests in KinD on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/katib_test.yaml - apps/katib/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - tests/gh-actions/install_istio.sh - tests/gh-actions/install_cert_manager.sh - common/cert-manager/** @@ -18,14 +16,8 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Install Istio run: ./tests/gh-actions/install_istio.sh @@ -57,7 +49,7 @@ jobs: kubectl wait --for=condition=Running experiments.kubeflow.org -n kubeflow-user --all --timeout 300s echo "Waiting for all Trials to become Succeeded..." - kubectl wait --for=condition=Succeeded trials.kubeflow.org -n kubeflow-user --all --timeout 300s + kubectl wait --for=condition=Succeeded trials.kubeflow.org -n kubeflow-user --all --timeout 1200s echo "Waiting for the Experiment to become Succeeded..." kubectl wait --for=condition=Succeeded experiments.kubeflow.org -n kubeflow-user --all --timeout 300s diff --git a/.github/workflows/kserve_cni_test.yaml b/.github/workflows/kserve_cni_test.yaml index c9ac27229e..f8c8c5cd53 100644 --- a/.github/workflows/kserve_cni_test.yaml +++ b/.github/workflows/kserve_cni_test.yaml @@ -2,10 +2,8 @@ name: Build & Apply KServe manifests in KinD, using istio CNI on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/kserve_cni_test.yaml - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - common/istio-cni-1-22/** - tests/gh-actions/install_cert_manager.sh - common/cert-manager/** @@ -20,14 +18,8 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Create kubeflow namespace run: kustomize build common/kubeflow-namespace/base | kubectl apply -f - diff --git a/.github/workflows/kserve_m2m_test.yaml b/.github/workflows/kserve_m2m_test.yaml index b468702f1a..8a057accc1 100644 --- a/.github/workflows/kserve_m2m_test.yaml +++ b/.github/workflows/kserve_m2m_test.yaml @@ -2,12 +2,10 @@ name: Deploy and test KServe with m2m auth in KinD on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/kserve_m2m_test.yaml - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - contrib/kserve/** - - common/oauth2-proxy/** + - common/oidc-client/oauth2-proxy/** - common/istio*/** - tests/gh-actions/install_istio_with_ext_auth.sh* - tests/gh-actions/install_cert_manager.sh @@ -23,18 +21,12 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Install kubectl run: ./tests/gh-actions/install_kubectl.sh - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh - - name: Create kubeflow namespace run: kustomize build common/kubeflow-namespace/base | kubectl apply -f - diff --git a/.github/workflows/kserve_test.yaml b/.github/workflows/kserve_test.yaml index 81b9825115..0a01979c74 100644 --- a/.github/workflows/kserve_test.yaml +++ b/.github/workflows/kserve_test.yaml @@ -2,10 +2,8 @@ name: Build & Apply KServe manifests in KinD on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/kserve_test.yaml - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - contrib/kserve/** - tests/gh-actions/install_cert_manager.sh - common/cert-manager/** @@ -20,14 +18,8 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Create kubeflow namespace run: kustomize build common/kubeflow-namespace/base | kubectl apply -f - diff --git a/.github/workflows/metacontroller_test.yaml b/.github/workflows/metacontroller_test.yaml index 987d30cd74..3774304fa4 100644 --- a/.github/workflows/metacontroller_test.yaml +++ b/.github/workflows/metacontroller_test.yaml @@ -2,10 +2,8 @@ name: Build & Apply contrib/metacontroller in KinD on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/metacontroller_test.yaml - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - contrib/metacontroller/** - tests/gh-actions/install_cert_manager.sh - common/cert-manager/** @@ -19,14 +17,8 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config ./tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Install Istio run: ./tests/gh-actions/install_istio.sh diff --git a/.github/workflows/model_registry_test.yaml b/.github/workflows/model_registry_test.yaml index e3c59aa6cb..2b18e27bfa 100644 --- a/.github/workflows/model_registry_test.yaml +++ b/.github/workflows/model_registry_test.yaml @@ -4,9 +4,8 @@ name: Deploy and test Kubeflow Model Registry on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - apps/model-registry/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kustomize.sh - tests/gh-actions/install_istio.sh - common/istio*/** @@ -17,18 +16,12 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Install kubectl run: ./tests/gh-actions/install_kubectl.sh - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh - - name: Create kubeflow namespace run: kustomize build common/kubeflow-namespace/base | kubectl apply -f - diff --git a/.github/workflows/notebook_controller_m2m_test.yaml b/.github/workflows/notebook_controller_m2m_test.yaml index fa0f70a934..54a53e5f19 100644 --- a/.github/workflows/notebook_controller_m2m_test.yaml +++ b/.github/workflows/notebook_controller_m2m_test.yaml @@ -2,12 +2,10 @@ name: Test Notebook Controller with m2m auth manifests in KinD on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/notebook_controller_m2m_test.yaml - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - apps/jupyter/** - - common/oauth2-proxy/** + - common/oidc-client/oauth2-proxy/** - common/istio*/** - tests/gh-actions/install_istio_with_ext_auth.sh* - tests/gh-actions/install_multi_tenancy.sh @@ -19,18 +17,12 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Install kubectl run: ./tests/gh-actions/install_kubectl.sh - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh - - name: Create kubeflow namespace run: kustomize build common/kubeflow-namespace/base | kubectl apply -f - diff --git a/.github/workflows/notebook_controller_test.yaml b/.github/workflows/notebook_controller_test.yaml index a423f05459..31bba95793 100644 --- a/.github/workflows/notebook_controller_test.yaml +++ b/.github/workflows/notebook_controller_test.yaml @@ -2,11 +2,9 @@ name: Build & Apply Notebook Controller manifests in KinD on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/notebook_controller_test.yaml - apps/jupyter/notebook-controller/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - tests/gh-actions/install_istio.sh - common/istio*/** @@ -17,14 +15,8 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Install Istio run: ./tests/gh-actions/install_istio.sh diff --git a/.github/workflows/pipeline_run_from_notebook.yaml b/.github/workflows/pipeline_run_from_notebook.yaml index 7d91966adc..d11c6879f5 100644 --- a/.github/workflows/pipeline_run_from_notebook.yaml +++ b/.github/workflows/pipeline_run_from_notebook.yaml @@ -2,18 +2,16 @@ name: Create Pipeline Run from Kubeflow Notebook on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/pipeline_run_from_notebook.yaml - apps/jupyter/notebook-controller/upstream/** - apps/pipeline/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - tests/gh-actions/install_istio.sh - tests/gh-actions/install_cert_manager.sh - common/cert-manager/** - - common/oauth2-proxy/** + - common/oidc-client/oauth2-proxy/** - common/istio*/** - - common//** + - common/oidc-client/** - apps/jupyter/** jobs: @@ -23,14 +21,8 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Install Istio with external authentication run: ./tests/gh-actions/install_istio_with_ext_auth.sh @@ -69,7 +61,7 @@ jobs: kubectl apply -f tests/gh-actions/kf-objects/notebook.test.kubeflow-user-example.com.yaml kubectl wait --for=jsonpath='{.status.readyReplicas}'=1 \ -f tests/gh-actions/kf-objects/notebook.test.kubeflow-user-example.com.yaml \ - --timeout 300s + --timeout 2400s - name: Wait for the kubeflow-m2m-oidc-configurator Job run: | diff --git a/.github/workflows/pipeline_test.yaml b/.github/workflows/pipeline_test.yaml index 2cfcca0cf6..61b05258ca 100644 --- a/.github/workflows/pipeline_test.yaml +++ b/.github/workflows/pipeline_test.yaml @@ -2,15 +2,13 @@ name: Deploy and test Kubeflow Pipelines manifests with m2m auth in KinD on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/pipeline_test.yaml - apps/pipeline/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - tests/gh-actions/install_istio.sh - tests/gh-actions/install_cert_manager.sh - common/cert-manager/** - - common/oauth2-proxy/** + - common/oidc-client/oauth2-proxy/** - common/istio*/** - tests/gh-actions/install_istio_with_ext_auth.sh @@ -21,14 +19,8 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Install kubectl run: ./tests/gh-actions/install_kubectl.sh diff --git a/.github/workflows/profiles_test.yaml b/.github/workflows/profiles_test.yaml index d562ff7c0d..c1ad802a82 100644 --- a/.github/workflows/profiles_test.yaml +++ b/.github/workflows/profiles_test.yaml @@ -2,11 +2,9 @@ name: Build & Apply Profiles manifests in KinD on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/profiles_test.yaml - apps/profiles/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - tests/gh-actions/install_istio.sh - common/istio*/** @@ -17,14 +15,8 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Install Istio run: ./tests/gh-actions/install_istio.sh diff --git a/.github/workflows/pss_test.yaml b/.github/workflows/pss_test.yaml index 984df974b1..9922aa2ef3 100644 --- a/.github/workflows/pss_test.yaml +++ b/.github/workflows/pss_test.yaml @@ -2,6 +2,7 @@ name: Apply PSS labels to namespaces on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/* - tests/gh-actions/kind-cluster.yaml - apps/profiles/upstream/** @@ -20,14 +21,8 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Install kubectl run: ./tests/gh-actions/install_kubectl.sh diff --git a/.github/workflows/ray_test.yaml b/.github/workflows/ray_test.yaml index d726105e84..20e3870153 100644 --- a/.github/workflows/ray_test.yaml +++ b/.github/workflows/ray_test.yaml @@ -2,10 +2,9 @@ name: Build & Apply Ray manifest in KinD on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/ray_test.yaml - contrib/ray/** - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh jobs: build: @@ -14,14 +13,8 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Build & Apply manifests run: | diff --git a/.github/workflows/seldon_test.yaml b/.github/workflows/seldon_test.yaml index c98b8be446..5653358bc4 100644 --- a/.github/workflows/seldon_test.yaml +++ b/.github/workflows/seldon_test.yaml @@ -2,11 +2,9 @@ name: Build & Apply Seldon manifests in KinD on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/seldon_test.yaml - - tests/gh-actions/kind-cluster.yaml - contrib/seldon/** - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - tests/gh-actions/install_istio.sh - common/istio*/** - tests/gh-actions/install_cert_manager.sh @@ -19,14 +17,8 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Install Istio run: ./tests/gh-actions/install_istio.sh diff --git a/.github/workflows/tensorboard_controller_test.yaml b/.github/workflows/tensorboard_controller_test.yaml index 69c044db9a..f6b248fde8 100644 --- a/.github/workflows/tensorboard_controller_test.yaml +++ b/.github/workflows/tensorboard_controller_test.yaml @@ -2,11 +2,9 @@ name: Build & Apply Tensorboard Controller manifests in KinD on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/tensorboard_controller_test.yaml - apps/tensorboard/tensorboard-controller/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - tests/gh-actions/install_istio.sh - common/istio*/** @@ -17,14 +15,8 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Install Istio run: ./tests/gh-actions/install_istio.sh diff --git a/.github/workflows/tensorboards_web_application_test.yaml b/.github/workflows/tensorboards_web_application_test.yaml index a0072655d4..377f83c47c 100644 --- a/.github/workflows/tensorboards_web_application_test.yaml +++ b/.github/workflows/tensorboards_web_application_test.yaml @@ -2,11 +2,9 @@ name: Build & Apply TWA manifests in KinD on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/tensorboards_web_application_test.yaml - apps/tensorboard/tensorboards-web-app/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - tests/gh-actions/install_istio.sh - common/istio*/** @@ -17,14 +15,8 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Install Istio run: ./tests/gh-actions/install_istio.sh diff --git a/.github/workflows/training_operator_test.yaml b/.github/workflows/training_operator_test.yaml index e9b107beb1..8d469e6c71 100644 --- a/.github/workflows/training_operator_test.yaml +++ b/.github/workflows/training_operator_test.yaml @@ -2,11 +2,9 @@ name: Build & Apply Training Operator manifests in KinD on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/training_operator_test.yaml - apps/training-operator/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - tests/gh-actions/install_istio.sh - common/istio*/** - tests/gh-actions/kf-objects/tfjob.yaml @@ -18,14 +16,8 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Install kubectl run: ./tests/gh-actions/install_kubectl.sh diff --git a/.github/workflows/volumes_web_application_test.yaml b/.github/workflows/volumes_web_application_test.yaml index 4165b90faa..7305d3bde2 100644 --- a/.github/workflows/volumes_web_application_test.yaml +++ b/.github/workflows/volumes_web_application_test.yaml @@ -2,11 +2,9 @@ name: Build & Apply VWA manifests in KinD on: pull_request: paths: + - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/volumes_web_application_test.yaml - apps/volumes-web-app/upstream/** - - tests/gh-actions/kind-cluster.yaml - - tests/gh-actions/install_kind.sh - - tests/gh-actions/install_kustomize.sh - tests/gh-actions/install_istio.sh - common/istio*/** @@ -17,14 +15,8 @@ jobs: - name: Checkout uses: actions/checkout@v4 - - name: Install KinD - run: ./tests/gh-actions/install_kind.sh - - - name: Create KinD Cluster - run: kind create cluster --config tests/gh-actions/kind-cluster.yaml - - - name: Install kustomize - run: ./tests/gh-actions/install_kustomize.sh + - name: Install KinD, Create KinD cluster and Install kustomize + run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - name: Install Istio run: ./tests/gh-actions/install_istio.sh diff --git a/tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh b/tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh new file mode 100755 index 0000000000..59e01d2445 --- /dev/null +++ b/tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh @@ -0,0 +1,66 @@ +#!/bin/bash +set -e + +error_exit() { + echo "Error occurred in script at line: ${1}." + exit 1 +} + +trap 'error_exit $LINENO' ERR + +echo "Install KinD..." +sudo swapoff -a + +# This conditional helps running GH Workflows through +# [act](https://github.com/nektos/act) +if [ -e /swapfile ]; then + sudo rm -f /swapfile + sudo mkdir -p /tmp/etcd + sudo mount -t tmpfs tmpfs /tmp/etcd +fi + +{ + curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.20.0/kind-linux-amd64 + chmod +x ./kind + sudo mv kind /usr/local/bin +} || { echo "Failed to install KinD"; exit 1; } + + +echo "Creating KinD cluster ..." +echo " +apiVersion: kind.x-k8s.io/v1alpha4 +kind: Cluster +# Configure registry for KinD. +containerdConfigPatches: +- |- + [plugins.\"io.containerd.grpc.v1.cri\".registry.mirrors.\"REGISTRY_NAME:REGISTRY_PORT\"] + endpoint = [\"http://REGISTRY_NAME:REGISTRY_PORT\"] +# This is needed in order to support projected volumes with service account tokens. +# See: https://kubernetes.slack.com/archives/CEKK1KTN2/p1600268272383600 +kubeadmConfigPatches: + - | + apiVersion: kubeadm.k8s.io/v1beta2 + kind: ClusterConfiguration + metadata: + name: config + apiServer: + extraArgs: + \"service-account-issuer\": \"kubernetes.default.svc\" + \"service-account-signing-key-file\": \"/etc/kubernetes/pki/sa.key\" +nodes: +- role: control-plane + image: kindest/node:v1.29.4@sha256:3abb816a5b1061fb15c6e9e60856ec40d56b7b52bcea5f5f1350bc6e2320b6f8 +- role: worker + image: kindest/node:v1.29.4@sha256:3abb816a5b1061fb15c6e9e60856ec40d56b7b52bcea5f5f1350bc6e2320b6f8 +- role: worker + image: kindest/node:v1.29.4@sha256:3abb816a5b1061fb15c6e9e60856ec40d56b7b52bcea5f5f1350bc6e2320b6f8 +" | kind create cluster --config - + + +echo "Install Kustomize ..." +{ + curl --silent --location --remote-name "https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv5.2.1/kustomize_v5.2.1_linux_amd64.tar.gz" + tar -xzvf kustomize_v5.2.1_linux_amd64.tar.gz + chmod a+x kustomize + sudo mv kustomize /usr/local/bin/kustomize +} || { echo "Failed to install Kustomize"; exit 1; } diff --git a/tests/gh-actions/install_istio_with_ext_auth.sh b/tests/gh-actions/install_istio_with_ext_auth.sh index 871fec394d..6b56131ea3 100755 --- a/tests/gh-actions/install_istio_with_ext_auth.sh +++ b/tests/gh-actions/install_istio_with_ext_auth.sh @@ -14,4 +14,4 @@ kubectl wait --for=condition=Ready pods --all -n istio-system --timeout=300s \ echo "Installing oauth2-proxy..." cd common/ kustomize build oauth2-proxy/overlays/m2m-self-signed/ | kubectl apply -f - -kubectl wait --for=condition=ready pod -l 'app.kubernetes.io/name=oauth2-proxy' --timeout=180s -n oauth2-proxy +kubectl wait --for=condition=ready pod -l 'app.kubernetes.io/name=oauth2-proxy' --timeout=180s -n oauth2-proxy \ No newline at end of file diff --git a/tests/gh-actions/install_kind.sh b/tests/gh-actions/install_kind.sh deleted file mode 100755 index 37f89ad92f..0000000000 --- a/tests/gh-actions/install_kind.sh +++ /dev/null @@ -1,15 +0,0 @@ -#!/bin/bash -set -e -echo "Fetching KinD executable ..." -sudo swapoff -a - -# This conditional helps running GH Workflows through -# [act](https://github.com/nektos/act) -if [ -e /swapfile ]; then - sudo rm -f /swapfile - sudo mkdir -p /tmp/etcd - sudo mount -t tmpfs tmpfs /tmp/etcd -fi -curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.20.0/kind-linux-amd64 -chmod +x ./kind -sudo mv kind /usr/local/bin \ No newline at end of file diff --git a/tests/gh-actions/kind-cluster.yaml b/tests/gh-actions/kind-cluster.yaml deleted file mode 100644 index 83dd8b3325..0000000000 --- a/tests/gh-actions/kind-cluster.yaml +++ /dev/null @@ -1,26 +0,0 @@ -apiVersion: kind.x-k8s.io/v1alpha4 -kind: Cluster -# Configure registry for KinD. -containerdConfigPatches: -- |- - [plugins."io.containerd.grpc.v1.cri".registry.mirrors."$REGISTRY_NAME:$REGISTRY_PORT"] - endpoint = ["http://$REGISTRY_NAME:$REGISTRY_PORT"] -# This is needed in order to support projected volumes with service account tokens. -# See: https://kubernetes.slack.com/archives/CEKK1KTN2/p1600268272383600 -kubeadmConfigPatches: - - | - apiVersion: kubeadm.k8s.io/v1beta2 - kind: ClusterConfiguration - metadata: - name: config - apiServer: - extraArgs: - "service-account-issuer": "kubernetes.default.svc" - "service-account-signing-key-file": "/etc/kubernetes/pki/sa.key" -nodes: -- role: control-plane - image: kindest/node:v1.29.4@sha256:3abb816a5b1061fb15c6e9e60856ec40d56b7b52bcea5f5f1350bc6e2320b6f8 -- role: worker - image: kindest/node:v1.29.4@sha256:3abb816a5b1061fb15c6e9e60856ec40d56b7b52bcea5f5f1350bc6e2320b6f8 -- role: worker - image: kindest/node:v1.29.4@sha256:3abb816a5b1061fb15c6e9e60856ec40d56b7b52bcea5f5f1350bc6e2320b6f8 \ No newline at end of file From d7834bc9b64e7211abaaf6c98b73d709558037cf Mon Sep 17 00:00:00 2001 From: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com> Date: Sun, 25 Aug 2024 19:50:01 +0200 Subject: [PATCH 25/35] re-enable more parts of the dex test (#2854) * renable more parts of the dex test Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * fix dex-login-test (#2830) * fix dex-login-test: - set executable permission for python script - set shebang line in python script Signed-off-by: Tom Zaspel <40226087+tzabbi@users.noreply.github.com> * Add all required componentes which have to be installed to test dex login Signed-off-by: Tom Zaspel * Add kustomizeation file to deploy all required stuff to test dex login Signed-off-by: Tom Zaspel * Add statuscode response in dex login Signed-off-by: Tom Zaspel <40226087+tzabbi@users.noreply.github.com> --------- Signed-off-by: Tom Zaspel <40226087+tzabbi@users.noreply.github.com> Signed-off-by: Tom Zaspel --------- Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> Signed-off-by: Tom Zaspel <40226087+tzabbi@users.noreply.github.com> Signed-off-by: Tom Zaspel Co-authored-by: Tom Zaspel <40226087+tzabbi@users.noreply.github.com> --- .github/workflows/dex_test.yaml | 9 +-- .../kustomization.yaml | 58 +++++++++++++++++++ tests/gh-actions/test_dex_login.py | 4 +- 3 files changed, 63 insertions(+), 8 deletions(-) create mode 100644 tests/gh-actions/deploy-dex-login-environment/kustomization.yaml mode change 100644 => 100755 tests/gh-actions/test_dex_login.py diff --git a/.github/workflows/dex_test.yaml b/.github/workflows/dex_test.yaml index a2f51aced8..8a9be13475 100644 --- a/.github/workflows/dex_test.yaml +++ b/.github/workflows/dex_test.yaml @@ -17,13 +17,9 @@ jobs: - name: Install KinD, Create KinD cluster and Install kustomize run: ./tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - - name: Install Istio - run: ./tests/gh-actions/install_istio.sh - - name: Build & Apply manifests run: | - cd common/dex - kustomize build overlays/istio | kubectl apply -f - + while ! kustomize build ./tests/gh-actions/deploy-dex-login-environment | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 20; done kubectl wait --for=condition=Ready pods --all --all-namespaces --timeout 180s - name: port forward @@ -35,5 +31,4 @@ jobs: - name: test dex login run: | pip3 install requests - # currently broken - # ./tests/gh-actions/test_dex_login.py + ./tests/gh-actions/test_dex_login.py diff --git a/tests/gh-actions/deploy-dex-login-environment/kustomization.yaml b/tests/gh-actions/deploy-dex-login-environment/kustomization.yaml new file mode 100644 index 0000000000..2fbf5ee86d --- /dev/null +++ b/tests/gh-actions/deploy-dex-login-environment/kustomization.yaml @@ -0,0 +1,58 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +sortOptions: + order: legacy + legacySortOptions: + orderFirst: + - Namespace + - ResourceQuota + - StorageClass + - CustomResourceDefinition + - MutatingWebhookConfiguration + - ServiceAccount + - PodSecurityPolicy + - NetworkPolicy + - Role + - ClusterRole + - RoleBinding + - ClusterRoleBinding + - ConfigMap + - Secret + - Endpoints + - Service + - LimitRange + - PriorityClass + - PersistentVolume + - PersistentVolumeClaim + - Deployment + - StatefulSet + - CronJob + - PodDisruptionBudget + orderLast: + - ValidatingWebhookConfiguration + +resources: +# Istio +- ../../../common/istio-1-22/istio-crds/base +- ../../../common/istio-1-22/istio-namespace/base +- ../../../common/istio-1-22/istio-install/overlays/oauth2-proxy +# oauth2-proxy +- ../../../common/oauth2-proxy/overlays/m2m-self-signed +# Dex +- ../../../common/dex/overlays/oauth2-proxy +- ../../../common/istio-1-22/cluster-local-gateway/base +# Kubeflow namespace +- ../../../common/kubeflow-namespace/base +# NetworkPolicies +- ../../../common/networkpolicies/base +# Kubeflow Roles +- ../../../common/kubeflow-roles/base +# Kubeflow Istio Resources +- ../../../common/istio-1-22/kubeflow-istio-resources/base +# Central Dashboard +- ../../../apps/centraldashboard/overlays/oauth2-proxy +# Profiles + KFAM +- ../../../apps/profiles/upstream/overlays/kubeflow +# User namespace +- ../../../common/user-namespace/base diff --git a/tests/gh-actions/test_dex_login.py b/tests/gh-actions/test_dex_login.py old mode 100644 new mode 100755 index b027d5ba92..b93c6ed5a9 --- a/tests/gh-actions/test_dex_login.py +++ b/tests/gh-actions/test_dex_login.py @@ -1,3 +1,5 @@ +#!/usr/bin/env python3 + import re import requests import sys @@ -101,7 +103,7 @@ def get_istio_auth_session(url: str, username: str, password: str) -> dict: username=KUBEFLOW_USERNAME, password=KUBEFLOW_PASSWORD ) - +print(f"{resp}") if resp == 200: sys.exit(0) else: From 39b4c63e1183a9c1b1a9af91a3857a8ec5c5bf01 Mon Sep 17 00:00:00 2001 From: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com> Date: Sun, 25 Aug 2024 19:52:00 +0200 Subject: [PATCH 26/35] Synchronize kubeflow kubeflow manifests v1.9.0 (#2853) * Update kubeflow/kubeflow manifests from v1.9.0 Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * update synchronization script Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --------- Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --- .../base/configs/spawner_ui_config.yaml | 30 +++++++++---------- hack/synchronize-kubeflow-manifests.sh | 4 +-- 2 files changed, 17 insertions(+), 17 deletions(-) diff --git a/apps/jupyter/jupyter-web-app/upstream/base/configs/spawner_ui_config.yaml b/apps/jupyter/jupyter-web-app/upstream/base/configs/spawner_ui_config.yaml index 6851aa3242..90af4d0753 100644 --- a/apps/jupyter/jupyter-web-app/upstream/base/configs/spawner_ui_config.yaml +++ b/apps/jupyter/jupyter-web-app/upstream/base/configs/spawner_ui_config.yaml @@ -37,15 +37,15 @@ spawnerFormDefaults: ################################################################ image: # the default container image - value: kubeflownotebookswg/jupyter-scipy:latest + value: kubeflownotebookswg/jupyter-scipy:v1.9.0 # the list of available container images in the dropdown options: - - kubeflownotebookswg/jupyter-scipy:latest - - kubeflownotebookswg/jupyter-pytorch-full:latest - - kubeflownotebookswg/jupyter-pytorch-cuda-full:latest - - kubeflownotebookswg/jupyter-tensorflow-full:latest - - kubeflownotebookswg/jupyter-tensorflow-cuda-full:latest + - kubeflownotebookswg/jupyter-scipy:v1.9.0 + - kubeflownotebookswg/jupyter-pytorch-full:v1.9.0 + - kubeflownotebookswg/jupyter-pytorch-cuda-full:v1.9.0 + - kubeflownotebookswg/jupyter-tensorflow-full:v1.9.0 + - kubeflownotebookswg/jupyter-tensorflow-cuda-full:v1.9.0 ################################################################ # VSCode-like Container Images (Group 1) @@ -60,11 +60,11 @@ spawnerFormDefaults: ################################################################ imageGroupOne: # the default container image - value: kubeflownotebookswg/codeserver-python:latest + value: kubeflownotebookswg/codeserver-python:v1.9.0 # the list of available container images in the dropdown options: - - kubeflownotebookswg/codeserver-python:latest + - kubeflownotebookswg/codeserver-python:v1.9.0 ################################################################ # RStudio-like Container Images (Group 2) @@ -81,11 +81,11 @@ spawnerFormDefaults: ################################################################ imageGroupTwo: # the default container image - value: kubeflownotebookswg/rstudio-tidyverse:latest + value: kubeflownotebookswg/rstudio-tidyverse:v1.9.0 # the list of available container images in the dropdown options: - - kubeflownotebookswg/rstudio-tidyverse:latest + - kubeflownotebookswg/rstudio-tidyverse:v1.9.0 ################################################################ # CPU Resources @@ -130,10 +130,10 @@ spawnerFormDefaults: # `limitsKey` - what will be set as the actual limit # `uiName` - what will be displayed in the dropdown UI vendors: - - limitsKey: "nvidia.com/gpu" - uiName: "NVIDIA" - - limitsKey: "amd.com/gpu" - uiName: "AMD" + - limitsKey: "nvidia.com/gpu" + uiName: "NVIDIA" + - limitsKey: "amd.com/gpu" + uiName: "AMD" # the default value of the limit # (possible values: "none", "1", "2", "4", "8") @@ -162,7 +162,7 @@ spawnerFormDefaults: requests: storage: 5Gi accessModes: - - ReadWriteOnce + - ReadWriteOnce ################################################################ # Data Volumes diff --git a/hack/synchronize-kubeflow-manifests.sh b/hack/synchronize-kubeflow-manifests.sh index 71bbe3e4a8..d83462a6c2 100644 --- a/hack/synchronize-kubeflow-manifests.sh +++ b/hack/synchronize-kubeflow-manifests.sh @@ -40,13 +40,13 @@ else fi echo "Checking out in $SRC_DIR to $COMMIT..." -# Checkout the Model Registry repository +# Checkout the upstream repository mkdir -p $SRC_DIR cd $SRC_DIR if [ ! -d "kubeflow/.git" ]; then git clone https://github.com/kubeflow/kubeflow.git fi -cd $SRC_DIR/models-web-app +cd $SRC_DIR/kubeflow if ! git rev-parse --verify --quiet $COMMIT; then git checkout -b $COMMIT else From b5c58343a1ecb0613a06c19158121921328ec5c5 Mon Sep 17 00:00:00 2001 From: Hansini Karunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Date: Mon, 26 Aug 2024 13:31:01 +0530 Subject: [PATCH 27/35] =?UTF-8?q?Change=20kind=20version=20and=20kubernete?= =?UTF-8?q?s=20version=20to=20support=20networkpolicies=E2=80=A6=20(#2843)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: hansinikarunarathne --- README.md | 2 +- .../install_KinD_create_KinD_cluster_install_kustomize.sh | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 46ba457a36..0d7656cba8 100644 --- a/README.md +++ b/README.md @@ -117,7 +117,7 @@ kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes: - role: control-plane - image: kindest/node:v1.29.4 + image: kindest/node:v1.31.0@sha256:53df588e04085fd41ae12de0c3fe4c72f7013bba32a20e7325357a1ac94ba865 kubeadmConfigPatches: - | kind: ClusterConfiguration diff --git a/tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh b/tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh index 59e01d2445..b38efa33a4 100755 --- a/tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh +++ b/tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh @@ -20,7 +20,7 @@ if [ -e /swapfile ]; then fi { - curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.20.0/kind-linux-amd64 + curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.24.0/kind-linux-amd64 chmod +x ./kind sudo mv kind /usr/local/bin } || { echo "Failed to install KinD"; exit 1; } @@ -49,11 +49,11 @@ kubeadmConfigPatches: \"service-account-signing-key-file\": \"/etc/kubernetes/pki/sa.key\" nodes: - role: control-plane - image: kindest/node:v1.29.4@sha256:3abb816a5b1061fb15c6e9e60856ec40d56b7b52bcea5f5f1350bc6e2320b6f8 + image: kindest/node:v1.31.0@sha256:53df588e04085fd41ae12de0c3fe4c72f7013bba32a20e7325357a1ac94ba865 - role: worker - image: kindest/node:v1.29.4@sha256:3abb816a5b1061fb15c6e9e60856ec40d56b7b52bcea5f5f1350bc6e2320b6f8 + image: kindest/node:v1.31.0@sha256:53df588e04085fd41ae12de0c3fe4c72f7013bba32a20e7325357a1ac94ba865 - role: worker - image: kindest/node:v1.29.4@sha256:3abb816a5b1061fb15c6e9e60856ec40d56b7b52bcea5f5f1350bc6e2320b6f8 + image: kindest/node:v1.31.0@sha256:53df588e04085fd41ae12de0c3fe4c72f7013bba32a20e7325357a1ac94ba865 " | kind create cluster --config - From c6c9ed0768809404f245b47d808c7f21ca1d2fa9 Mon Sep 17 00:00:00 2001 From: Hansini Karunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Date: Mon, 26 Aug 2024 22:09:01 +0530 Subject: [PATCH 28/35] add kubeflow installation guide to issue report (#2859) * add kubeflow installation guide to issue report Signed-off-by: hansinikarunarathne * fix a linting issue Signed-off-by: hansinikarunarathne --------- Signed-off-by: hansinikarunarathne --- .github/ISSUE_TEMPLATE/issue-report.yml | 117 ++++++++++++------------ 1 file changed, 60 insertions(+), 57 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/issue-report.yml b/.github/ISSUE_TEMPLATE/issue-report.yml index 7b9a95c71c..9f442c91e2 100644 --- a/.github/ISSUE_TEMPLATE/issue-report.yml +++ b/.github/ISSUE_TEMPLATE/issue-report.yml @@ -1,64 +1,67 @@ name: Issue Report description: Report an Issue body: - - type: markdown - attributes: - value: | - Hello, Please fill out the sections below to help everyone identify and fix the bug - - type: checkboxes - id: Vaildation - attributes: - label: Validation Checklist - options: - - label: Is this a Kubeflow issue? - required: true - - label: Are you posting in the right repository ? - required: true - - label: Did you follow the installation guide https://github.com/kubeflow/manifests?tab=readme-ov-file ? - required: true - - label: Is the issue report properly structured and detailed with version numbers? - required: true - - label: Is this for Kubeflow development ? - required: false - - label: Would you like to work on this issue? - required: false - - label: You can join the CNCF Slack and access our meetings at the [Kubeflow Community](https://www.kubeflow.org/docs/about/community/) website. Our channel on the CNCF Slack is here [**#kubeflow-platform**](https://app.slack.com/client/T08PSQ7BQ/C073W572LA2). - required: false - - type: dropdown - id: version - attributes: - label: Version - description: What version of our software are you running? - options: - - master - - 1.9 - - 1.8 - validations: +- type: markdown + attributes: + value: | + Fill out the sections below to help everyone identify and fix the bug +- type: markdown + attributes: + value: | + Follow the [Kubeflow installation guidelines](https://github.com/kubeflow/manifests/blob/master/README.md). +- type: checkboxes + id: Vaildation + attributes: + label: Validation Checklist + options: + - label: Is this a Kubeflow issue? required: true - - type: textarea - id: description - attributes: - label: Describe your issue - placeholder: When installing Kubeflow I got the following error message from ... - validations: + - label: Are you posting in the right repository ? required: true - - type: textarea - id: steps - attributes: - label: Steps to reproduce the issue - placeholder: | - 1. Try this ... - 2. Then do this ... - validations: + - label: Did you follow the [Kubeflow installation guideline](https://github.com/kubeflow/manifests/blob/master/README.md) ? required: true - - type: textarea - id: screenshots - attributes: - label: Put here any screenshots or videos (optional) - - type: markdown - attributes: - value: | - > **Note:** Please note that you have to fill required fields to post an Issue. - If not please close or redo the issue and you can join the CNCF Slack and access our meetings at the [Kubeflow Community](https://www.kubeflow.org/docs/about/community/) website. Our channel on the CNCF Slack is here [**#kubeflow-platform**](https://app.slack.com/client/T08PSQ7BQ/C073W572LA2). + - label: Is the issue report properly structured and detailed with version numbers? + required: true + - label: Is this for Kubeflow development ? + required: false + - label: Would you like to work on this issue? + required: false + - label: You can join the CNCF Slack and access our meetings at the [Kubeflow Community](https://www.kubeflow.org/docs/about/community/) website. Our channel on the CNCF Slack is here [**#kubeflow-platform**](https://app.slack.com/client/T08PSQ7BQ/C073W572LA2). + required: false +- type: dropdown + id: version + attributes: + label: Version + description: Which version of the Kubeflow platform are you using? + options: + - master + - 1.9 + validations: + required: true +- type: textarea + id: description + attributes: + label: Describe your issue + placeholder: When installing Kubeflow I got the following error message from ... + validations: + required: true +- type: textarea + id: steps + attributes: + label: Steps to reproduce the issue + placeholder: | + 1. Try this ... + 2. Then do this ... + validations: + required: true +- type: textarea + id: screenshots + attributes: + label: Put here any screenshots or videos (optional) +- type: markdown + attributes: + value: | + > **Note:** Please note that you have to fill required fields to post an issue. + If not please close or redo the issue and you can join the CNCF slack and access our meetings at the [Kubeflow Community](https://www.kubeflow.org/docs/about/community/) website. Our channel on the CNCF Slack is here [**#kubeflow-platform**](https://app.slack.com/client/T08PSQ7BQ/C073W572LA2). - **Thanks for reporting this issue! We will get back to you as soon as possible.** \ No newline at end of file + **Thanks for reporting this issue! We will get back to you as soon as possible.** From 271dfeebd36a6c9dcb5dd083f23ce193c3cd555a Mon Sep 17 00:00:00 2001 From: Hansini Karunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Date: Tue, 27 Aug 2024 20:56:02 +0530 Subject: [PATCH 29/35] Adjust the Trivy scanning workflow to upload artifacts (#2860) * add step Upload trivy scanned_results to trivy.yaml Signed-off-by: hansinikarunarathne * fix linting issues Signed-off-by: hansinikarunarathne --------- Signed-off-by: hansinikarunarathne --- .github/workflows/trivy.yaml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/trivy.yaml b/.github/workflows/trivy.yaml index 99bfa64265..3144f102dc 100644 --- a/.github/workflows/trivy.yaml +++ b/.github/workflows/trivy.yaml @@ -41,8 +41,8 @@ jobs: cd hack python3 trivy_scan.py - # - name: Upload trivy scanned_results - # uses: actions/upload-artifact@v4 - # with: - # name: trivy_scanned_results - # path: image_lists/ + - name: Upload trivy scanned_results + uses: actions/upload-artifact@v4 + with: + name: trivy_scanned_results + path: image_lists/ From df2c0a669c48928de5deeb10aa761e3feccf444b Mon Sep 17 00:00:00 2001 From: Matteo Mortari Date: Wed, 28 Aug 2024 08:52:02 +0200 Subject: [PATCH 30/35] update PR template (#2863) * update PR template follow-up to #2793 - add link to sign-off/DCO - CLA no longer required Signed-off-by: tarilabs * update DCO link to one in KF community Signed-off-by: tarilabs --------- Signed-off-by: tarilabs --- .github/PULL_REQUEST_TEMPLATE.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 2de7cae887..2a58b1dc4e 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -11,8 +11,7 @@ ## ✅ Contributor checklist - Make sure you have tested with kustomize. See [Installation Prerequisites](https://github.com/kubeflow/manifests#prerequisites) - - All the commits have been _signed-off_ (To pass the `DCO` check) - - Submit the [Contributor License Agreements](https://cla.developers.google.com/clas) (To pass the `cla/google` check) + - All the commits have been [_signed-off_](https://github.com/kubeflow/community/tree/master/dco-signoff-hook#signing-off-commits) (To pass the `DCO` check) --- From 3c342f23e824a9543025c28fc3e1c892cc7bd3d6 Mon Sep 17 00:00:00 2001 From: Matteo Mortari Date: Wed, 28 Aug 2024 10:23:02 +0200 Subject: [PATCH 31/35] cve: fix Double-count in "All Images" for CVE image scan report (#2856) * WIP: CVE scan double-counting CVEs total baseline Signed-off-by: tarilabs * fixed to avoid double-counting with demo Signed-off-by: tarilabs * remove demo files Signed-off-by: tarilabs --------- Signed-off-by: tarilabs --- hack/trivy_scan.py | 34 ++++++++++++++++++++-------------- 1 file changed, 20 insertions(+), 14 deletions(-) diff --git a/hack/trivy_scan.py b/hack/trivy_scan.py index 91da390394..3e2213eaa5 100644 --- a/hack/trivy_scan.py +++ b/hack/trivy_scan.py @@ -275,6 +275,7 @@ def extract_images(version): ) # Initialize counters +unique_images = {} # unique set of images across all WGs total_images = 0 total_low = 0 total_medium = 0 @@ -309,12 +310,9 @@ def extract_images(version): high = sum(entry["severity_counts"]["HIGH"] for entry in data) critical = sum(entry["severity_counts"]["CRITICAL"] for entry in data) - # Update the total counts - total_images += image_count - total_low += low - total_medium += medium - total_high += high - total_critical += critical + # Update unique_images for the total counts later + for d in data: + unique_images[d["image"]] = d # Create the output for this file file_data = { @@ -328,15 +326,23 @@ def extract_images(version): # Update merged_data with filename as key merged_data[filename] = file_data - # Add total counts to merged_data - merged_data["total"] = { - "images": total_images, - "LOW": total_low, - "MEDIUM": total_medium, - "HIGH": total_high, - "CRITICAL": total_critical, - } +# Update the total counts +unique_images = unique_images.values() # keep the set of values +total_images += len(unique_images) +total_low += sum(entry["severity_counts"]["LOW"] for entry in unique_images) +total_medium += sum(entry["severity_counts"]["MEDIUM"] for entry in unique_images) +total_high += sum(entry["severity_counts"]["HIGH"] for entry in unique_images) +total_critical += sum(entry["severity_counts"]["CRITICAL"] for entry in unique_images) + +# Add total counts to merged_data +merged_data["total"] = { + "images": total_images, + "LOW": total_low, + "MEDIUM": total_medium, + "HIGH": total_high, + "CRITICAL": total_critical, +} log("Summary in Json Format:") log(json.dumps(merged_data, indent=4)) From 9b915bb98e1c2b7c296a9281522f73e9e6e17647 Mon Sep 17 00:00:00 2001 From: Cyber Nagle Date: Wed, 28 Aug 2024 17:04:03 +0800 Subject: [PATCH 32/35] fix: wrong diagram path for kubeflow auth . (#2862) Signed-off-by: Cyber Nagle --- common/oauth2-proxy/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/common/oauth2-proxy/README.md b/common/oauth2-proxy/README.md index 12b02fd0ae..2110e0b7a0 100644 --- a/common/oauth2-proxy/README.md +++ b/common/oauth2-proxy/README.md @@ -78,7 +78,7 @@ when client calls API to list the KF Pipeline Runs: ### Auth analysis diagram for Kubeflow Pipelines -![Kubeflow Auth Diagram](./kubeflow_auth_diagram.svg) +![Kubeflow Auth Diagram](./components/kubeflow_auth_diagram.svg) ## Kubeflow Notebooks User and M2M Authentication and Authorization @@ -132,4 +132,4 @@ This is based on the following: [^3]: [oauth2-proxy](https://github.com/oauth2-proxy/oauth2-proxy) [^4]: [Kubernetes TokenReview](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-review-v1/) [^5]: [Kubernetes SubjectAccessReview](https://kubernetes.io/docs/reference/kubernetes-api/authorization-resources/subject-access-review-v3/) -[^6]: [Kube RBAC Proxy](https://github.com/brancz/kube-rbac-proxy) \ No newline at end of file +[^6]: [Kube RBAC Proxy](https://github.com/brancz/kube-rbac-proxy) From afc358d6d473a24029149f2a0ca21671af4aca6d Mon Sep 17 00:00:00 2001 From: Hansini Karunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Date: Wed, 28 Aug 2024 14:55:03 +0530 Subject: [PATCH 33/35] Consolidate Kubeflow Manifest tests (#2861) Signed-off-by: hansinikarunarathne --- .github/workflows/katib_test.yaml | 2 +- .github/workflows/pipeline_run_from_notebook.yaml | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/katib_test.yaml b/.github/workflows/katib_test.yaml index 841da24e6c..973fa15588 100644 --- a/.github/workflows/katib_test.yaml +++ b/.github/workflows/katib_test.yaml @@ -49,7 +49,7 @@ jobs: kubectl wait --for=condition=Running experiments.kubeflow.org -n kubeflow-user --all --timeout 300s echo "Waiting for all Trials to become Succeeded..." - kubectl wait --for=condition=Succeeded trials.kubeflow.org -n kubeflow-user --all --timeout 1200s + kubectl wait --for=condition=Succeeded trials.kubeflow.org -n kubeflow-user --all --timeout 600s echo "Waiting for the Experiment to become Succeeded..." kubectl wait --for=condition=Succeeded experiments.kubeflow.org -n kubeflow-user --all --timeout 300s diff --git a/.github/workflows/pipeline_run_from_notebook.yaml b/.github/workflows/pipeline_run_from_notebook.yaml index d11c6879f5..2e9c07e9cc 100644 --- a/.github/workflows/pipeline_run_from_notebook.yaml +++ b/.github/workflows/pipeline_run_from_notebook.yaml @@ -9,7 +9,7 @@ on: - tests/gh-actions/install_istio.sh - tests/gh-actions/install_cert_manager.sh - common/cert-manager/** - - common/oidc-client/oauth2-proxy/** + - common/oauth2-proxy/** - common/istio*/** - common/oidc-client/** - apps/jupyter/** @@ -61,7 +61,7 @@ jobs: kubectl apply -f tests/gh-actions/kf-objects/notebook.test.kubeflow-user-example.com.yaml kubectl wait --for=jsonpath='{.status.readyReplicas}'=1 \ -f tests/gh-actions/kf-objects/notebook.test.kubeflow-user-example.com.yaml \ - --timeout 2400s + --timeout 600s - name: Wait for the kubeflow-m2m-oidc-configurator Job run: | From b91cbc4980c9728c83ce4302c8a47403847e9ac0 Mon Sep 17 00:00:00 2001 From: biswajit-9776 <115724497+biswajit-9776@users.noreply.github.com> Date: Fri, 30 Aug 2024 01:00:04 +0530 Subject: [PATCH 34/35] Added securityContext profile to cronjob in istio-system and remove PSS workflow warnings (#2848) * Added securityContext profile to cronjob in istio-system Signed-off-by: biswajit-9776 * Limited script to run for a minute Signed-off-by: biswajit-9776 * Undo change to script Signed-off-by: biswajit-9776 * Added time constraint to waiting for job loop Signed-off-by: biswajit-9776 * Added if condition to script Signed-off-by: biswajit-9776 * Added version to curl image Signed-off-by: biswajit-9776 * Undo change to curl image Signed-off-by: biswajit-9776 * Fixed failing cronjob Signed-off-by: biswajit-9776 * Refactored the script Signed-off-by: biswajit-9776 * Added workflow job to clear PSS warnings Signed-off-by: biswajit-9776 * Removed cluster-local-gateway PSS patch Signed-off-by: biswajit-9776 * Fixed typo in patches Signed-off-by: biswajit-9776 * Debugging failing warnings Signed-off-by: biswajit-9776 * Empty commit Signed-off-by: biswajit-9776 * Fixed trigger paths for m2m tests Signed-off-by: biswajit-9776 * Remove debug commands Signed-off-by: biswajit-9776 * Fixed typo Signed-off-by: biswajit-9776 * Increases cronjob time from 60s to 100s Signed-off-by: biswajit-9776 * Change UID to debug Signed-off-by: biswajit-9776 --------- Signed-off-by: biswajit-9776 --- .github/workflows/kserve_m2m_test.yaml | 2 +- .../notebook_controller_m2m_test.yaml | 2 +- .../workflows/pipeline_run_from_notebook.yaml | 2 +- .github/workflows/pipeline_test.yaml | 2 +- .github/workflows/pss_test.yaml | 24 ++++++++++++++++--- ...ronjob.kubeflow-m2m-oidc-configurator.yaml | 9 +++++++ .../security/PSS/patches/cache-server.yaml | 1 + .../PSS/patches/cluster-local-gateway.yaml | 12 ---------- contrib/security/PSS/patches/dex.yaml | 1 + contrib/security/PSS/patches/kfam.yaml | 1 + ...kubeflow-pipelines-profile-controller.yaml | 1 + contrib/security/PSS/patches/manager.yaml | 1 + .../patches/metadata-envoy-deployment.yaml | 1 + .../PSS/patches/metadata-grpc-deployment.yaml | 1 + .../security/PSS/patches/metadata-writer.yaml | 1 + contrib/security/PSS/patches/minio.yaml | 1 + .../patches/ml-pipeline-persistenceagent.yaml | 1 + .../ml-pipeline-scheduledworkflow.yaml | 1 + .../security/PSS/patches/ml-pipeline-ui.yaml | 1 + .../PSS/patches/ml-pipeline-viewer-crd.yaml | 1 + .../ml-pipeline-visualizationserver.yaml | 1 + contrib/security/PSS/patches/ml-pipeline.yaml | 1 + contrib/security/PSS/patches/mysql.yaml | 1 + .../security/PSS/patches/oauth2-proxy.yaml | 1 + ...wait_for_kubeflow_m2m_oidc_configurator.sh | 18 ++++++++++---- 25 files changed, 65 insertions(+), 23 deletions(-) delete mode 100644 contrib/security/PSS/patches/cluster-local-gateway.yaml diff --git a/.github/workflows/kserve_m2m_test.yaml b/.github/workflows/kserve_m2m_test.yaml index 8a057accc1..2f0b679c9f 100644 --- a/.github/workflows/kserve_m2m_test.yaml +++ b/.github/workflows/kserve_m2m_test.yaml @@ -5,7 +5,7 @@ on: - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/kserve_m2m_test.yaml - contrib/kserve/** - - common/oidc-client/oauth2-proxy/** + - common/oauth2-proxy/** - common/istio*/** - tests/gh-actions/install_istio_with_ext_auth.sh* - tests/gh-actions/install_cert_manager.sh diff --git a/.github/workflows/notebook_controller_m2m_test.yaml b/.github/workflows/notebook_controller_m2m_test.yaml index 54a53e5f19..4fddc3c326 100644 --- a/.github/workflows/notebook_controller_m2m_test.yaml +++ b/.github/workflows/notebook_controller_m2m_test.yaml @@ -5,7 +5,7 @@ on: - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh - .github/workflows/notebook_controller_m2m_test.yaml - apps/jupyter/** - - common/oidc-client/oauth2-proxy/** + - common/oauth2-proxy/** - common/istio*/** - tests/gh-actions/install_istio_with_ext_auth.sh* - tests/gh-actions/install_multi_tenancy.sh diff --git a/.github/workflows/pipeline_run_from_notebook.yaml b/.github/workflows/pipeline_run_from_notebook.yaml index 2e9c07e9cc..f83b729a06 100644 --- a/.github/workflows/pipeline_run_from_notebook.yaml +++ b/.github/workflows/pipeline_run_from_notebook.yaml @@ -11,7 +11,7 @@ on: - common/cert-manager/** - common/oauth2-proxy/** - common/istio*/** - - common/oidc-client/** + - common/** - apps/jupyter/** jobs: diff --git a/.github/workflows/pipeline_test.yaml b/.github/workflows/pipeline_test.yaml index 61b05258ca..fec9c9eccd 100644 --- a/.github/workflows/pipeline_test.yaml +++ b/.github/workflows/pipeline_test.yaml @@ -8,7 +8,7 @@ on: - tests/gh-actions/install_istio.sh - tests/gh-actions/install_cert_manager.sh - common/cert-manager/** - - common/oidc-client/oauth2-proxy/** + - common/oauth2-proxy/** - common/istio*/** - tests/gh-actions/install_istio_with_ext_auth.sh diff --git a/.github/workflows/pss_test.yaml b/.github/workflows/pss_test.yaml index 9922aa2ef3..86c89fde2b 100644 --- a/.github/workflows/pss_test.yaml +++ b/.github/workflows/pss_test.yaml @@ -44,10 +44,10 @@ jobs: - name: Install KF Pipelines run: ./tests/gh-actions/install_pipelines.sh - - name: Applying Pod Security Standards baseline levels for static namespaces + - name: Apply Pod Security Standards baseline levels for static namespaces run: ./tests/gh-actions/enable_baseline_PSS.sh - - name: Applying Pod Security Standards baseline levels for dynamic namespaces + - name: Apply Pod Security Standards baseline levels for dynamic namespaces run: | cat << EOF > ./kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 @@ -61,7 +61,7 @@ jobs: rm ./kustomization.yaml kubectl -n kubeflow wait --for=condition=Ready pods -l kustomize.component=profiles --timeout 180s - - name: Unapplying applied baseline values + - name: Unapply applied baseline values run: | NAMESPACES=("istio-system" "auth" "cert-manager" "oauth2-proxy" "kubeflow") for NAMESPACE in "${NAMESPACES[@]}"; do @@ -69,6 +69,24 @@ jobs: kubectl label namespace $NAMESPACE pod-security.kubernetes.io/enforce- fi done + sleep 10 + + - name: Apply patches to clear warnings + run: | + DIRECTORY="contrib/security/PSS/patches" + for file in "$DIRECTORY"/*.yaml; do + echo "Patching file: $file" + + KIND=$(kubectl get -f "$file" -o jsonpath='{.kind}') + NAME=$(kubectl get -f "$file" -o jsonpath='{.metadata.name}') + NAMESPACE=$(kubectl get -f "$file" -o jsonpath='{.metadata.namespace}') + + # Apply the patch + kubectl get "$KIND" "$NAME" -n "$NAMESPACE" &> /dev/null + if [ $? -eq 0 ]; then + kubectl patch "$KIND" "$NAME" -n "$NAMESPACE" --patch-file "$file" + fi + done - name: Applying Pod Security Standards restricted levels for static namespaces run: ./tests/gh-actions/enable_restricted_PSS.sh diff --git a/common/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/cronjob.kubeflow-m2m-oidc-configurator.yaml b/common/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/cronjob.kubeflow-m2m-oidc-configurator.yaml index 639e96cb49..5c3e305b5e 100644 --- a/common/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/cronjob.kubeflow-m2m-oidc-configurator.yaml +++ b/common/oauth2-proxy/components/configure-self-signed-kubernetes-oidc-issuer/cronjob.kubeflow-m2m-oidc-configurator.yaml @@ -29,6 +29,15 @@ spec: name: script subPath: script.sh resources: {} + securityContext: + allowPrivilegeEscalation: false + seccompProfile: + type: RuntimeDefault + runAsNonRoot: true + runAsUser: 1000 + capabilities: + drop: + - ALL volumes: - name: script configMap: diff --git a/contrib/security/PSS/patches/cache-server.yaml b/contrib/security/PSS/patches/cache-server.yaml index e62a8bcc49..51c1123020 100644 --- a/contrib/security/PSS/patches/cache-server.yaml +++ b/contrib/security/PSS/patches/cache-server.yaml @@ -2,6 +2,7 @@ apiVersion: apps/v1 kind: Deployment metadata: name: cache-server + namespace: kubeflow spec: template: spec: diff --git a/contrib/security/PSS/patches/cluster-local-gateway.yaml b/contrib/security/PSS/patches/cluster-local-gateway.yaml deleted file mode 100644 index 09b890872f..0000000000 --- a/contrib/security/PSS/patches/cluster-local-gateway.yaml +++ /dev/null @@ -1,12 +0,0 @@ -apiVersion: apps/v1 -kind: Deployment -metadata: - name: cluster-local-gateway -spec: - template: - spec: - containers: - - name: istio-proxy - securityContext: - seccompProfile: - type: RuntimeDefault diff --git a/contrib/security/PSS/patches/dex.yaml b/contrib/security/PSS/patches/dex.yaml index d82ecba673..edde313997 100644 --- a/contrib/security/PSS/patches/dex.yaml +++ b/contrib/security/PSS/patches/dex.yaml @@ -2,6 +2,7 @@ apiVersion: apps/v1 kind: Deployment metadata: name: dex + namespace: auth spec: template: spec: diff --git a/contrib/security/PSS/patches/kfam.yaml b/contrib/security/PSS/patches/kfam.yaml index cfd999f56a..0445d53764 100644 --- a/contrib/security/PSS/patches/kfam.yaml +++ b/contrib/security/PSS/patches/kfam.yaml @@ -2,6 +2,7 @@ apiVersion: apps/v1 kind: Deployment metadata: name: profiles-deployment + namespace: kubeflow spec: template: spec: diff --git a/contrib/security/PSS/patches/kubeflow-pipelines-profile-controller.yaml b/contrib/security/PSS/patches/kubeflow-pipelines-profile-controller.yaml index 9c29bfb9ab..342ca70f6e 100644 --- a/contrib/security/PSS/patches/kubeflow-pipelines-profile-controller.yaml +++ b/contrib/security/PSS/patches/kubeflow-pipelines-profile-controller.yaml @@ -2,6 +2,7 @@ apiVersion: apps/v1 kind: Deployment metadata: name: kubeflow-pipelines-profile-controller + namespace: kubeflow spec: template: spec: diff --git a/contrib/security/PSS/patches/manager.yaml b/contrib/security/PSS/patches/manager.yaml index cbf3f3579d..b4080721fa 100644 --- a/contrib/security/PSS/patches/manager.yaml +++ b/contrib/security/PSS/patches/manager.yaml @@ -2,6 +2,7 @@ apiVersion: apps/v1 kind: Deployment metadata: name: profiles-deployment + namespace: kubeflow spec: template: spec: diff --git a/contrib/security/PSS/patches/metadata-envoy-deployment.yaml b/contrib/security/PSS/patches/metadata-envoy-deployment.yaml index ff0370c0f5..717bd851c1 100644 --- a/contrib/security/PSS/patches/metadata-envoy-deployment.yaml +++ b/contrib/security/PSS/patches/metadata-envoy-deployment.yaml @@ -2,6 +2,7 @@ apiVersion: apps/v1 kind: Deployment metadata: name: metadata-envoy-deployment + namespace: kubeflow spec: template: spec: diff --git a/contrib/security/PSS/patches/metadata-grpc-deployment.yaml b/contrib/security/PSS/patches/metadata-grpc-deployment.yaml index 778f849a08..af6971827b 100644 --- a/contrib/security/PSS/patches/metadata-grpc-deployment.yaml +++ b/contrib/security/PSS/patches/metadata-grpc-deployment.yaml @@ -2,6 +2,7 @@ apiVersion: apps/v1 kind: Deployment metadata: name: metadata-grpc-deployment + namespace: kubeflow spec: template: spec: diff --git a/contrib/security/PSS/patches/metadata-writer.yaml b/contrib/security/PSS/patches/metadata-writer.yaml index ada39a1bc1..49f10ab0c9 100644 --- a/contrib/security/PSS/patches/metadata-writer.yaml +++ b/contrib/security/PSS/patches/metadata-writer.yaml @@ -2,6 +2,7 @@ apiVersion: apps/v1 kind: Deployment metadata: name: metadata-writer + namespace: kubeflow spec: template: spec: diff --git a/contrib/security/PSS/patches/minio.yaml b/contrib/security/PSS/patches/minio.yaml index 0dc2a0e5fb..896233f298 100644 --- a/contrib/security/PSS/patches/minio.yaml +++ b/contrib/security/PSS/patches/minio.yaml @@ -2,6 +2,7 @@ apiVersion: apps/v1 kind: Deployment metadata: name: minio + namespace: kubeflow spec: template: spec: diff --git a/contrib/security/PSS/patches/ml-pipeline-persistenceagent.yaml b/contrib/security/PSS/patches/ml-pipeline-persistenceagent.yaml index 61aa1f2a69..dfa05641ca 100644 --- a/contrib/security/PSS/patches/ml-pipeline-persistenceagent.yaml +++ b/contrib/security/PSS/patches/ml-pipeline-persistenceagent.yaml @@ -2,6 +2,7 @@ apiVersion: apps/v1 kind: Deployment metadata: name: ml-pipeline-persistenceagent + namespace: kubeflow spec: template: spec: diff --git a/contrib/security/PSS/patches/ml-pipeline-scheduledworkflow.yaml b/contrib/security/PSS/patches/ml-pipeline-scheduledworkflow.yaml index 72e8773638..bf9fc18e33 100644 --- a/contrib/security/PSS/patches/ml-pipeline-scheduledworkflow.yaml +++ b/contrib/security/PSS/patches/ml-pipeline-scheduledworkflow.yaml @@ -2,6 +2,7 @@ apiVersion: apps/v1 kind: Deployment metadata: name: ml-pipeline-scheduledworkflow + namespace: kubeflow spec: template: spec: diff --git a/contrib/security/PSS/patches/ml-pipeline-ui.yaml b/contrib/security/PSS/patches/ml-pipeline-ui.yaml index c3ce8228d5..b5c35fcb42 100644 --- a/contrib/security/PSS/patches/ml-pipeline-ui.yaml +++ b/contrib/security/PSS/patches/ml-pipeline-ui.yaml @@ -2,6 +2,7 @@ apiVersion: apps/v1 kind: Deployment metadata: name: ml-pipeline-ui + namespace: kubeflow spec: template: spec: diff --git a/contrib/security/PSS/patches/ml-pipeline-viewer-crd.yaml b/contrib/security/PSS/patches/ml-pipeline-viewer-crd.yaml index cf52aef9e7..a52470533b 100644 --- a/contrib/security/PSS/patches/ml-pipeline-viewer-crd.yaml +++ b/contrib/security/PSS/patches/ml-pipeline-viewer-crd.yaml @@ -2,6 +2,7 @@ apiVersion: apps/v1 kind: Deployment metadata: name: ml-pipeline-viewer-crd + namespace: kubeflow spec: template: spec: diff --git a/contrib/security/PSS/patches/ml-pipeline-visualizationserver.yaml b/contrib/security/PSS/patches/ml-pipeline-visualizationserver.yaml index 1673b60c66..a2bc3450ba 100644 --- a/contrib/security/PSS/patches/ml-pipeline-visualizationserver.yaml +++ b/contrib/security/PSS/patches/ml-pipeline-visualizationserver.yaml @@ -2,6 +2,7 @@ apiVersion: apps/v1 kind: Deployment metadata: name: ml-pipeline-visualizationserver + namespace: kubeflow spec: template: spec: diff --git a/contrib/security/PSS/patches/ml-pipeline.yaml b/contrib/security/PSS/patches/ml-pipeline.yaml index 67fc9a41f8..097ad1b8f2 100644 --- a/contrib/security/PSS/patches/ml-pipeline.yaml +++ b/contrib/security/PSS/patches/ml-pipeline.yaml @@ -2,6 +2,7 @@ apiVersion: apps/v1 kind: Deployment metadata: name: ml-pipeline + namespace: kubeflow spec: template: spec: diff --git a/contrib/security/PSS/patches/mysql.yaml b/contrib/security/PSS/patches/mysql.yaml index 257264d6a3..b2663f5b55 100644 --- a/contrib/security/PSS/patches/mysql.yaml +++ b/contrib/security/PSS/patches/mysql.yaml @@ -2,6 +2,7 @@ apiVersion: apps/v1 kind: Deployment metadata: name: mysql + namespace: kubeflow spec: template: spec: diff --git a/contrib/security/PSS/patches/oauth2-proxy.yaml b/contrib/security/PSS/patches/oauth2-proxy.yaml index 007e4d8bdc..4fc94e93af 100644 --- a/contrib/security/PSS/patches/oauth2-proxy.yaml +++ b/contrib/security/PSS/patches/oauth2-proxy.yaml @@ -2,6 +2,7 @@ apiVersion: apps/v1 kind: Deployment metadata: name: oauth2-proxy + namespace: oauth2-proxy spec: replicas: 2 template: diff --git a/tests/gh-actions/wait_for_kubeflow_m2m_oidc_configurator.sh b/tests/gh-actions/wait_for_kubeflow_m2m_oidc_configurator.sh index 726ae62fca..2968885a72 100755 --- a/tests/gh-actions/wait_for_kubeflow_m2m_oidc_configurator.sh +++ b/tests/gh-actions/wait_for_kubeflow_m2m_oidc_configurator.sh @@ -2,6 +2,8 @@ CRONJOB_NAME=kubeflow-m2m-oidc-configurator NAMESPACE=istio-system +RETRY_INTERVAL=5 # Each loop iterates after 'RETRY_INTERVAL' seconds +MAX_RETRIES=20 # Each loop iterates for a total number of 'MAX_RETRIES' # Function to get the latest Job created by the CronJob get_latest_job() { @@ -13,19 +15,23 @@ get_latest_job() { # Wait until a Job is created echo "Waiting for a Job to be created by the ${CRONJOB_NAME} CronJob..." -while true; do +for ((i=1; i<=MAX_RETRIES; i++)); do JOB_NAME=$(get_latest_job) if [[ -n "${JOB_NAME}" ]]; then echo "Job ${JOB_NAME} created." break fi - sleep 5 + if [[ $i -eq $MAX_RETRIES ]]; then + echo "Job creation timed out." + exit 1 + fi + sleep "${RETRY_INTERVAL}" echo "Waiting..." done # Wait for the Job to complete successfully echo "Waiting for the Job ${JOB_NAME} to complete..." -while true; do +for ((i=1; i<=MAX_RETRIES; i++)); do STATUS=$(kubectl get job "${JOB_NAME}" -n "${NAMESPACE}" -o jsonpath='{.status.conditions[?(@.type=="Complete")].status}') if [[ "${STATUS}" == "True" ]]; then echo "Job ${JOB_NAME} completed successfully." @@ -37,5 +43,9 @@ while true; do echo "Job ${JOB_NAME} failed." exit 1 fi - sleep 5 + if [[ $i -eq $MAX_RETRIES ]]; then + echo "Job completion timed out." + exit 1 + fi + sleep "${RETRY_INTERVAL}" done From da0255f10d875040c2d845cd61b7938236c0dfaa Mon Sep 17 00:00:00 2001 From: Hansini Karunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Date: Tue, 3 Sep 2024 21:41:14 +0530 Subject: [PATCH 35/35] update manifest test README (#2867) Signed-off-by: hansinikarunarathne --- tests/README.md | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/tests/README.md b/tests/README.md index e0804b6ee8..0cb989c055 100644 --- a/tests/README.md +++ b/tests/README.md @@ -17,9 +17,4 @@ included in the tests. The e2e tests are completely independent of the underlying K8s cluster, as well as the platform of the cluster. These tests should be able to run in real -world clusters, as well as ephemeral ones like KinD. - -The tests are also explicitly bypassing any authentication system of the -installation. The goal of these tests are to ensure that a common use case that -deploys all of Kubeflow's components can succeed. Thus we only want to test -that the core CRDs can be successfully applied and complete. +world clusters, as well as ephemeral ones like KinD. \ No newline at end of file