diff --git a/samples/test/README.md b/samples/test/README.md index 728f36f36ad..220a6f54477 100644 --- a/samples/test/README.md +++ b/samples/test/README.md @@ -1,6 +1,6 @@ # Sample Test -Refer to [V2 samples test documentation](https://github.com/kubeflow/pipelines/tree/master/v2/test) for more details. +Refer to [V2 samples test documentation](https://github.com/kubeflow/pipelines/tree/master/backend/src/v2/test) for more details. ## Test Samples diff --git a/sdk/python/test_data/README.md b/sdk/python/test_data/README.md index 9b8557f0cf0..35f0a24f5c0 100644 --- a/sdk/python/test_data/README.md +++ b/sdk/python/test_data/README.md @@ -21,6 +21,6 @@ for f in sdk/python/test_data/components/*.py ; do echo "$f" && python3 "$f" ; d 2. Pipeline execution tests -> Location: `test/sdk-execution-tests.py` +> Location: `test/sdk-execution-tests/sdk_execution_tests.py` These tests ensure that the KFP OSS BE can execute the pipelines. `execute`: may be `false` in the `test_data_config.yaml` for a given `test_case` if the test case (a) isn't a complete example (e.g., a dependency doesn't exist in the image, etc.) or (b) the KFP OSS BE cannot execute the pipeline. diff --git a/test/README.md b/test/README.md index c1216884f32..ec9cc7dd49f 100644 --- a/test/README.md +++ b/test/README.md @@ -1,93 +1,100 @@ -# ML pipeline test infrastructure +# ML Pipeline Test Infrastructure -This folder contains the integration/e2e tests for ML pipeline. We use Argo workflow to run the tests. +This folder contains the integration and end-to-end (E2E) tests for the ML pipeline. Tests are executed using Kind (Kubernetes IN Docker) to simulate a Kubernetes cluster locally. GitHub Actions (GHAs) handle automated testing on pull requests. -At a high level, a typical test workflow will +At a high level, a typical test workflow will: +- Build images for all components. +- Create a dedicated test namespace in the cluster. +- Deploy the ML pipeline using the newly built components. +- Run the tests. +- Clean up the namespace and temporary resources. -- build docker images for all components -- create a dedicate test namespace in the cluster -- deploy ml pipeline using the newly built components -- run the test -- delete the namespace -- delete the images +These steps are performed in the same Kubernetes cluster. -All these steps will be taking place in the same Kubernetes cluster. -You can use GKE to test against the code in a Github Branch. The image will be temporarily stored in the GCR repository in the same project. +--- -Tests are run automatically on each commit in a Kubernetes cluster using -[Prow](https://github.com/kubernetes/test-infra/tree/master/prow). -Tests can also be run manually, see the next section. +## Running Tests Locally with Kind -## Run tests using GKE - -You could run the tests against a specific commit. +To run tests locally, set up a Kind cluster and follow the same steps as the GitHub Actions workflows. This section details the process. ### Setup -Here are the one-time steps to prepare for your GKE testing cluster: -- Follow the [deployment guide](https://www.kubeflow.org/docs/components/pipelines/installation/standalone-deployment/#set-up-your-cluster) to -create a GKE cluster. -- Install [Argo](https://github.com/argoproj/argo-workflows/) -in the cluster. -- Create cluster role binding. - ``` - kubectl create clusterrolebinding default-as-admin --clusterrole=cluster-admin --serviceaccount=default:default - ``` -- Follow the -[guideline](https://developer.github.com/v3/guides/managing-deploy-keys/) to -create a -[ssh](https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/) -deploy key, and store as Kubernetes secret in your cluster, so the job can -later access the code. Note it requires admin permission to add a deploy key -to github repo. This step is not needed when the project is public. - ``` - kubectl create secret generic ssh-key-secret - --from-file=id_rsa=/path/to/your/id_rsa - --from-file=id_rsa.pub=/path/to/your/id_rsa.pub - ``` - -### Run tests -Simply submit the test workflow to the GKE cluster, with a parameter -specifying the commit you want to test (master HEAD by default): -``` -argo submit integration_test_gke.yaml -p commit-sha= -``` -You can check the result by doing: -``` -argo list -``` -The workflow will create a temporary namespace with the same name as the Argo -workflow. All the images will be stored in -**gcr.io/project_id/workflow_name/branch_name/***. By default when the test is -*finished, the namespace and images will be deleted. -However you can keep them by providing additional parameter. -``` -argo submit integration_test_gke.yaml -p branch="my-branch" -p cleanup="false" -``` - -### Run presubmit-tests-with-pipeline-deployment.sh locally - -Run the following commands from root of kubeflow/pipelines repo. -``` -# $WORKSPACE are env variables set by Prow -export WORKSPACE=$(pwd) # root of kubeflow/pipelines git repo -export SA_KEY_FILE=PATH/TO/YOUR/GCP/PROJECT/SERVICE/ACCOUNT/KEY -# (optional) uncomment the following to keep reusing the same cluster -# export TEST_CLUSTER=YOUR_PRECONFIGURED_CLUSTER_NAME -# (optional) uncomment the following to disable built image caching -# export DISABLE_IMAGE_CACHING=true - -./test/presubmit-tests-with-pipeline-deployment.sh \ - --workflow_file e2e_test_gke_v2.yaml \ # You can specify other workflows you want to test too. - --test_result_folder ${FOLDER_NAME_TO_HOLD_TEST_RESULT} \ - --test_result_bucket ${YOUR_GCS_TEST_RESULT_BUCKET} \ - --project ${YOUR_GCS_PROJECT} -``` - -## Troubleshooting - -**Q: Why is my test taking so long on GKE?** - -The cluster downloads a bunch of images during the first time the test runs. It will be faster the second time since the images are cached. -The image building steps are running in parallel and usually takes 2~3 minutes in total. If you are experiencing high latency, it might due to the resource constrains -on your GKE cluster. In that case you need to deploy a larger cluster. +1. **Install Prerequisites**: + - A container engine like [Podman](https://podman.io) or [Docker](https://docs.docker.com/get-docker/) + - [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation) + - [kubectl](https://kubernetes.io/docs/tasks/tools/) + +2. **Set Up a Kind Cluster**: + Create a configuration file for your Kind cluster (optional): + ```yaml + kind: Cluster + apiVersion: kind.x-k8s.io/v1alpha4 + nodes: + - role: control-plane + - role: worker + ``` + Create the cluster: + + `kind create cluster --name kfp-test-cluster --config kind-config.yaml` + + Verify the cluster: + + `kubectl cluster-info --context kind-kfp-test-cluster` + +3. **Prepare the Test Environment**: + + - Install Python test dependencies: + + `pip install -r test/requirements.txt` + + - Deploy Kubeflow Pipelines to the Kind cluster: + + `kubectl apply -k manifests/` + +4. **Run the Tests**: +Execute the desired test suite: + + `pytest test/kfp-functional-test/` + +For additional guidance on deploying Kubeflow Pipelines in Kind, refer to: + +- [Kind Local Cluster Deployment Guide](https://www.kubeflow.org/docs/components/pipelines/legacy-v1/installation/localcluster-deployment/#kind) +- [Operator Deployment Guide](https://www.kubeflow.org/docs/components/pipelines/operator-guides/installation/#deploying-kubeflow-pipelines) + + +## Automated Testing with GitHub Actions + + +Tests are automatically triggered on GitHub when: + +- A pull request is opened or updated. + +GitHub Actions workflows are defined in the `.github/workflows/` directory. + +### Reproducing CI Steps Locally + +To replicate the steps locally: + +1. Clone the Kubeflow Pipelines repository: + + `git clone https://github.com/kubeflow/pipelines.git + cd pipelines` + +2. Follow the steps outlined in the **Running Tests Locally with Kind** section. + +3. To mimic the GitHub Actions environment, export any required environment variables found in the workflow files. + +* * * * * + +Troubleshooting +--------------- + +**Q: Why is my test taking so long?** + +- The first run downloads many container images. Subsequent runs will be faster due to caching. +- If you experience high latency, ensure the local system running Kind has sufficient resources (CPU, memory). + +**Q: How do I clean up the Kind cluster?** + +- Delete the Kind cluster: + `kind delete cluster --name kfp-test-cluster` \ No newline at end of file diff --git a/test/kfp-functional-test/README.md b/test/kfp-functional-test/README.md index c08795d322a..2387cc7d775 100644 --- a/test/kfp-functional-test/README.md +++ b/test/kfp-functional-test/README.md @@ -6,6 +6,7 @@ dependencies. To update dependencies: 1. edit [requirements.in](requirements.in) 1. run + ```bash pip-compile requirements.in ``` @@ -13,45 +14,71 @@ dependencies. To update dependencies: ## Run kfp-functional-test in local -### Via python +### Via python (Using Kind) -1. run - ```bash - gcloud auth application-default login - ``` - acquire new user credentials to use for Application Default Credentials. +1. Set up a Kind cluster: + + ```bash + kind create cluster --name kfp-functional-test-cluster + ``` + +2. Deploy Kubeflow Pipelines to the Kind cluster: + + ```bash + kubectl apply -k manifests/ + ``` + +3. Ensure the cluster is ready: -1. go to the root directory of kubeflow pipelines project, run ```bash - cd {YOUR_ROOT_DIRECTORY_OF_KUBEFLOW_PIPELINES} - python3 ./test/kfp-functional-test/run_kfp_functional_test.py --host "https://$(curl https://raw.githubusercontent.com/kubeflow/testing/master/test-infra/kfp/endpoint)" + kubectl cluster-info --context kind-kfp-functional-test-cluster ``` -### Via docker -1. run +4. Run the functional test: + ```bash - gcloud auth application-default login + cd {YOUR_ROOT_DIRECTORY_OF_KUBEFLOW_PIPELINES} + python3 ./test/kfp-functional-test/run_kfp_functional_test.py --host "http://localhost:8080" ``` - acquire new user credentials to use for Application Default Credentials. - Credentials saved to file with {CREDENTIALS_PATH} similar to: [$HOME/.config/gcloud/application_default_credentials.json] -1. copy the Credentials to the temp folder +### Via Kind - ````bash - cp {CREDENTIALS_PATH} /tmp/keys/{FILENAME}.json - ``` -1. Provide authentication credentials by setting the environment variable GOOGLE_APPLICATION_CREDENTIALS. - Replace [PATH] with the file path of the JSON file that contains your credentials. - run +1. Set up a Kind cluster: - ```bash - export GOOGLE_APPLICATION_CREDENTIALS="/tmp/keys/{FILENAME}.json" - ``` + ```bash + kind create cluster --name kfp-functional-test-cluster + ``` + +2. Deploy Kubeflow Pipelines to the Kind cluster: + + ```bash + kubectl apply -k manifests/ + ``` + +3. Ensure the cluster is ready: + + ```bash + kubectl cluster-info --context kind-kfp-functional-test-cluster + ``` + +4. Start a container and run the functional test: + + Using Docker: + ```bash + docker run -it -v $(pwd):/tmp/src -w /tmp/src python:3.9-slim\ + /tmp/src/test/kfp-functional-test/kfp-functional-test.sh --host "http://localhost:8080" + ``` + + Using Podman: + ```bash + podman run -it -v $(pwd):/tmp/src:Z -w /tmp/src python:3.9-slim \ + /tmp/src/test/kfp-functional-test/kfp-functional-test.sh --host "http://localhost:8080" + ``` + + +## Periodic Functional Tests with GitHub Actions + +A periodic GitHub Actions workflow is configured to automatically run functional tests daily. The workflow ensures consistent validation of the Kubeflow Pipelines functionality. + +For more details, see the [Periodic Functional Tests GitHub Actions workflow](https://github.com/kubeflow/pipelines/blob/master/.github/workflows/periodic.yml) -1. go to the root directory of kubeflow pipelines project and run - ```bash - cd {YOUR_ROOT_DIRECTORY_OF_KUBEFLOW_PIPELINES} - docker run -it -v $(pwd):/tmp/src -w /tmp/src -e GOOGLE_APPLICATION_CREDENTIALS=/tmp/keys/{FILENAME}.json \ - -v $GOOGLE_APPLICATION_CREDENTIALS:/tmp/keys/{FILENAME}.json:ro \ - python:3.9-slim /tmp/src/test/kfp-functional-test/kfp-functional-test.sh - ```