Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test: Kubeflow Pipelines V2 integration Tests #11125

Merged
merged 1 commit into from
Aug 23, 2024
Merged

test: Kubeflow Pipelines V2 integration Tests #11125

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions .github/workflows/kubeflow-pipelines-integration-v2.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
name: Kubeflow Pipelines V2 integration Tests

on:
push:
branches:
- master
pull_request:
paths:
- '.github/workflows/kubeflow-pipelines-integration-v2.yml'
- 'samples'
- 'core'
- 'backend'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

backend is not included in https://github.com/GoogleCloudPlatform/oss-test-infra/blob/68230effbdc9d59429b205475325c3212ac887d7/prow/prowjobs/kubeflow/pipelines/kubeflow-pipelines-presubmits.yaml#L61.
Do you really think it should be included here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure!


jobs:
integration-tests:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.8]

steps:
- name: Checkout code
uses: actions/checkout@v2

- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python }}

- name: Create KFP cluster
uses: ./.github/actions/kfp-cluster

- name: Forward API port
run: ./scripts/deploy/github/forward-port.sh "kubeflow" "ml-pipeline" 8888 8888

- name: Run the Integration Tests
run: |
./backend/src/v2/test/integration-test.sh
14 changes: 2 additions & 12 deletions backend/src/v2/test/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,19 +9,9 @@ include $(ENV_PATH)
SHELL = /bin/bash

.PHONY: integration-test
integration-test: upload
integration-test:
export KF_PIPELINES_ENDPOINT=$(HOST) \
&& python -u sample_test.py \
--samples_config samples/test/config-integration.yaml \
--timeout_mins 60 \
--context $(GCS_ROOT)/src/context.tar.gz \
--gcs_root $(GCS_ROOT)/data \
--gcr_root $(GCR_ROOT) \
--kfp_package_path "$(KFP_PACKAGE_PATH)"

.PHONY: upload
upload: context
python -u scripts/upload_gcs_blob.py tmp/context.tar.gz $(GCS_ROOT)/src/context.tar.gz
&& python -u ../../../../samples/v2/sample_test.py
diegolovison marked this conversation as resolved.
Show resolved Hide resolved

.PHONY: context
context:
Comment on lines 9 to 17
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file adds nothing to the logic and can be completely removed. You can invoke sample_test.py from backend/src/v2/test/integration-test.sh.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, I decided not to change the scope of things. Those files probably are going to change again once we starting moving to our CI. For now, I kept as closer to the original. Any problem?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would remove it anyway. But feel free to decide it.

Expand Down
6 changes: 0 additions & 6 deletions backend/src/v2/test/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,3 @@
# install kfp sdk from local path
-e ../../../../sdk/python
# TODO(chensun): remove the deprecated dependencies once migrated tests.
-r ../../../../sdk/python/requirements-deprecated.txt
ml-metadata==1.14.0
minio==7.0.4
google-cloud-storage
fire
-e ../../../../samples/test/utils
20 changes: 0 additions & 20 deletions backend/src/v2/test/scripts/upload_gcs_blob.py
View file
Open in desktop

This file was deleted.

Loading