[backend] metadata-writer cannot save metadata of S3 artifacts with argo v3.1+

[tktest1234]

Environment

• How did you deploy Kubeflow Pipelines (KFP)?
  kubeflow 1.4 manifest (git::https://github.com/kubeflow/manifests.git?ref=v1.4.0 )
  and manually change config for S3
• KFP version:
  1.7.0 (kubeflow 1.4)
• KFP SDK version:
  1.7.1

Steps to reproduce

1. setting to save artifacts to S3 (change configmap for workflow-controller)
2. Run pipeline: [Tutorial] Data passing in python components

metadata-writer output log

Kubernetes Pod event:  ADDED file-passing-pipelines-txmqj-2254123803 14598940
Traceback (most recent call last):
  File "/kfp/metadata_writer/metadata_writer.py", line 238, in <module>
    artifact_uri = argo_artifact_to_uri(argo_artifact)
  File "/kfp/metadata_writer/metadata_writer.py", line 106, in argo_artifact_to_uri
    provider=get_object_store_provider(s3_artifact['endpoint']),
KeyError: 'endpoint'

result:

• Artifact in metadb is empty

Expected result

metadata for the artifacts is logged into metadb

Materials and Reference

I think it is caused by changes for argo Key-Only Artifacts.

In https://github.com/kubeflow/pipelines/blob/1.7.0/backend/metadata_writer/src/metadata_writer.py#L318, metadata-writer read "workflows.argoproj.io/outputs", but endpoint and bucket are not passed by argo3.1+. So https://github.com/kubeflow/pipelines/blob/1.7.0/backend/metadata_writer/src/metadata_writer.py#L105 cannot set the correct URL.

Related frontend issue: #5930

---

Impacted by this bug? Give it a 👍.

[surajkota]

@tktest1234 Are you running on AWS and want to use KFP with S3 as storage backend?

[tktest1234]

Yes, I install kubeflow1.4 on AWS(EKS) and want to use KFP with S3 as storage backend.
Isn't it expected?

[surajkota]

Hey @tktest1234 please follow the instructions on AWS distribution of Kubeflow to install KFP with S3 as artifact storage https://awslabs.github.io/kubeflow-manifests/docs/deployment/rds-s3/

Please create an issue on the awslabs repository if you face any issues

[tktest1234]

Thank you, but awslabs's installation doesn't solve the problem.

#5829 is exactly the PR for this issue. I hope it to progress.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[rimolive]

Closing this issue. No activity for more than a year.

/close

[google-oss-prow]

@rimolive: Closing this issue.

In response to this:

Closing this issue. No activity for more than a year.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.