[backend] <Cannot get MLMD objects from Metadata store. Cannot find context>(version 1.8.0 and 1.10.0)

[ZDowney926]

Environment

• How did you deploy Kubeflow Pipelines (KFP)?
  Following the guide of https://github.com/kubeflow/manifests/tree/v1.10-branch
• KFP version:
  v2.2.0
• KFP SDK version:
  (kfp: 2.4.0
  kfp-pipeline-spec: 0.2.2
  kfp-server-api: 2.0.3)

Steps to reproduce

Here is a pipeline issue, I try in KubeFlow both v1.8.0 and v1.10.0 ,it both have same issue. I have read relevant issue, and try lots of method, it doesn't work. It seems like MLMD problem?( I don't know is it about database?)
I followed the example to build a simple pipeline, here is the code.

The pipeline.yaml show:

Then, I put the yaml file to pipleline

I create run of pipeline, then I get the result of this "Cannot get MLMD objects from Metadata store.", if click Details, it shows

---

Impacted by this bug? Give it a 👍.

[rimolive]

https://github.com/kubeflow/manifests/tree/v1.10-branch is still under development. I recommend use https://github.com/kubeflow/manifests/tree/v1.9-branch or the v1.9.0 tag.

[ESKYoung]

I'm encountering the same issue, but from installing only Kubeflow Pipelines as detailed here. Will investigate the full installation.

[nparkstar]

I have the same problem.

[nparkstar]

I solved my issue at last.
I did fresh install on the new machine, and the problem has not appear anymore.

But I think the following instruction is wrong.
"Install individual components" of https://github.com/kubeflow/manifests/tree/v1.9.0.

I cannot connect to the central dashboard after installing kubeflow according to the above instruction.

I succeeded after installing using bellow command.

while ! kustomize build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done

Thanks,

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[shivanibhargove]

Hi,
I am also facing the same issue with kfp upgrade. Could anyone help?
Details:
kfp-pipeline: 2.3.0
kfp-server-api: 2.0.3 )
kubeflow: I have tried with 1.8, 1.9
Error: Cannot find context with {"typeName":"system.PipelineRun","contextName":"9d29166b-68e1-433f-b506-62a70f0e1a13"}: Cannot find specified context

It is a blocker for us to upgrade to kfp v2.

[juliusvonkohout]

@rimolive i can confirm the issue.

[922tech]

I think if you guys send pods' status in kubeflow and related namespaces (auth, admission-controllee, etc.) and also k8s system pods would help a lot. Because these pods are almost like a chain and each pod depends on another one to work and you must find the broken link.

[shivanibhargove]

Any update on the issue fix?
Not able to run pipelines because of this issue.

[juliusvonkohout]

I got a pipeline to finish, but there are still errors with ml-metadata
Also no clear errors in the ml-metadata deployments. I think i have to examine the database directly Maybe I know more in a few weeks @rimolive

CC @kubeflow/release-team