[bug] kfp-kubernetes-execution-tests failing

[hbelmiro]

============================= test session starts ==============================
platform linux -- Python 3.9.19, pytest-7.1.3, pluggy-1.5.0
rootdir: /home/runner/work/data-science-pipelines/data-science-pipelines, configfile: pytest.ini
plugins: xdist-2.5.0, asyncio-cooperative-0.28.0, forked-1.6.0
collected 6 items
Experiment details: http://localhost:8888/#/experiments/details/4a528a7e-59e4-4baf-aed3-ab951f27225b
Run details: http://localhost:8888/#/runs/details/4e538fb6-5227-4364-90bb-44cba02cb765
- Created run my_pipeline-create_mount_delete_dynamic_pvc
	Module: /home/runner/work/data-science-pipelines/data-science-pipelines/kubernetes_platform/python/test/snapshot/data/create_mount_delete_dynamic_pvc.py
	URL: http://localhost:8888/#/runs/details/4e538fb6-5227-4364-90bb-44cba02cb765
Experiment details: http://localhost:8888/#/experiments/details/4a528a7e-59e4-4baf-aed3-ab951f27225b
Run details: http://localhost:8888/#/runs/details/2f4dafd5-62e0-4258-bbf4-031bee9701f6
- Created run my_pipeline-create_mount_delete_existing_pvc
	Module: /home/runner/work/data-science-pipelines/data-science-pipelines/kubernetes_platform/python/test/snapshot/data/create_mount_delete_existing_pvc.py
	URL: http://localhost:8888/#/runs/details/2f4dafd5-62e0-4258-bbf4-031bee9701f6
Experiment details: http://localhost:8888/#/experiments/details/4a528a7e-59e4-4baf-aed3-ab951f27225b
Run details: http://localhost:8888/#/runs/details/cf20fbe6-1743-48df-b2b2-79fb382b22ad
- Created run my_pipeline-create_mount_delete_existing_pvc_from_task_output
	Module: /home/runner/work/data-science-pipelines/data-science-pipelines/kubernetes_platform/python/test/snapshot/data/create_mount_delete_existing_pvc_from_task_output.py
	URL: http://localhost:8888/#/runs/details/cf20fbe6-1743-48df-b2b2-79fb382b22ad
Experiment details: http://localhost:8888/#/experiments/details/4a528a7e-59e4-4baf-aed3-ab951f27225b
Run details: http://localhost:8888/#/runs/details/f095468e-c7e8-4b09-bc89-bcfd2d955bb1
- Created run my_pipeline-secret_as_env
	Module: /home/runner/work/data-science-pipelines/data-science-pipelines/kubernetes_platform/python/test/snapshot/data/secret_as_env.py
	URL: http://localhost:8888/#/runs/details/f095468e-c7e8-4b09-bc89-bcfd2d955bb1
Experiment details: http://localhost:8888/#/experiments/details/4a528a7e-59e4-4baf-aed3-ab951f27225b
Run details: http://localhost:8888/#/runs/details/597fa077-2863-40c4-a683-ddd6bbc6781b
- Created run my_pipeline-secret_as_vol
	Module: /home/runner/work/data-science-pipelines/data-science-pipelines/kubernetes_platform/python/test/snapshot/data/secret_as_vol.py
	URL: http://localhost:8888/#/runs/details/597fa077-2863-40c4-a683-ddd6bbc6781b
Experiment details: http://localhost:8888/#/experiments/details/4a528a7e-59e4-4baf-aed3-ab951f27225b
Run details: http://localhost:8888/#/runs/details/6002a883-1c0a-44f7-9d2b-7364d0ccecdf
- Created run my_pipeline-node_selector
	Module: /home/runner/work/data-science-pipelines/data-science-pipelines/kubernetes_platform/python/test/snapshot/data/node_selector.py
	URL: http://localhost:8888/#/runs/details/6002a883-1c0a-44f7-9d2b-7364d0ccecdf
test/kfp-kubernetes-execution-tests/sdk_execution_tests.py FF....        [100%]
=================================== FAILURES ===================================
_______________________________ test[test_case3] _______________________________
test_case = TestCase(name='my_pipeline-secret_as_env', module_path='/home/runner/work/data-science-pipelines/data-science-pipeline...pelines/kubernetes_platform/python/test/snapshot/data/secret_as_env.yaml', function_name='my_pipeline', arguments=None)
    @pytest.mark.asyncio_cooperative
    @pytest.mark.parametrize('test_case', create_test_case_parameters())
    async def test(test_case: TestCase) -> None:
        """Asynchronously runs all samples and test that they succeed."""
        event_loop = asyncio.get_running_loop()
        try:
            run_url, run_result = run(test_case)
        except Exception as e:
            raise RuntimeError(
                f'Error triggering pipeline {test_case.name}.') from e
    
        api_run = await event_loop.run_in_executor(None, wait, run_result)
>       assert api_run.state == 'SUCCEEDED', f'Pipeline {test_case.name} ended with incorrect status: {api_run.state}. More info: {run_url}'
E       AssertionError: Pipeline my_pipeline-secret_as_env ended with incorrect status: FAILED. More info: http://localhost:8888/#/runs/details/f095468e-c7e8-4b09-bc89-bcfd2d955bb1
E       assert 'FAILED' == 'SUCCEEDED'
E         - SUCCEEDED
E         + FAILED
test/kfp-kubernetes-execution-tests/sdk_execution_tests.py:137: AssertionError
_______________________________ test[test_case4] _______________________________
test_case = TestCase(name='my_pipeline-secret_as_vol', module_path='/home/runner/work/data-science-pipelines/data-science-pipeline...pelines/kubernetes_platform/python/test/snapshot/data/secret_as_vol.yaml', function_name='my_pipeline', arguments=None)
    @pytest.mark.asyncio_cooperative
    @pytest.mark.parametrize('test_case', create_test_case_parameters())
    async def test(test_case: TestCase) -> None:
        """Asynchronously runs all samples and test that they succeed."""
        event_loop = asyncio.get_running_loop()
        try:
            run_url, run_result = run(test_case)
        except Exception as e:
            raise RuntimeError(
                f'Error triggering pipeline {test_case.name}.') from e
    
        api_run = await event_loop.run_in_executor(None, wait, run_result)
>       assert api_run.state == 'SUCCEEDED', f'Pipeline {test_case.name} ended with incorrect status: {api_run.state}. More info: {run_url}'
E       AssertionError: Pipeline my_pipeline-secret_as_vol ended with incorrect status: FAILED. More info: http://localhost:8888/#/runs/details/597fa077-2863-40c4-a683-ddd6bbc6781b
E       assert 'FAILED' == 'SUCCEEDED'
E         - SUCCEEDED
E         + FAILED
test/kfp-kubernetes-execution-tests/sdk_execution_tests.py:137: AssertionError
=============================== warnings summary ===============================
../../../../../opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/kfp/client/client.py:159
  /opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/kfp/client/client.py:159: FutureWarning: This client only works with Kubeflow Pipeline v2.0.0-beta.2 and later versions.
    warnings.warn(
../../../../../opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/kfp_server_api/rest.py:47
test/kfp-kubernetes-execution-tests/sdk_execution_tests.py::test[test_case3]
  /opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/kfp_server_api/rest.py:47: DeprecationWarning: HTTPResponse.getheader() is deprecated and will be removed in urllib3 v2.1.0. Instead use HTTPResponse.headers.get(name, default).
    return self.urllib3_response.getheader(name, default)
test/kfp-kubernetes-execution-tests/sdk_execution_tests.py:44
  /home/runner/work/data-science-pipelines/data-science-pipelines/test/kfp-kubernetes-execution-tests/sdk_execution_tests.py:44: PytestCollectionWarning: cannot collect test class 'TestCase' because it has a __init__ constructor (from: test/kfp-kubernetes-execution-tests/sdk_execution_tests.py)
    @dataclasses.dataclass
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED test/kfp-kubernetes-execution-tests/sdk_execution_tests.py::test[test_case3]
FAILED test/kfp-kubernetes-execution-tests/sdk_execution_tests.py::test[test_case4]
============= 2 failed, 4 passed, 4 warnings in 112.17s (0:01:52) ==============

/area testing

---

Impacted by this bug? Give it a 👍.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[hbelmiro]

/remove-lifecycle stale
/lifecycle frozen

[liavweiss]

/assign