feat(sdk): add PipelineConfig to DSL to re-implement pipeline-level config

[gregsheremeta]

Description of your changes:

KFP v1 supported setting pipeline-level configuration via a PipelineConf class. This class was deprecated and no replacement was added to KFP v2.

Add new PipelineConfig class to support setting pipeline-level configuration in KFP v2.

Note: I discussed this approach with @chensun in the KFP Community meeting on August 14 (agenda, recording). He gave preliminary approval for this approach.

Here's an example of how it would be used.

Checklist:

• The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign connor-mccarthy for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• api/OWNERS
• sdk/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[DharmitD]

/rerun-all

[gregsheremeta]

@chensun bump to the top of your inbox :) thanks!

[gregsheremeta]

@chensun bump, looking for an ack on the approach and the proto fields :)

[gregsheremeta]

@chensun bumping, can you take a look?

[gregsheremeta]

Sharing the document that @chensun provided in today's KFP Community meeting (agenda, recording) :

[gregsheremeta]

Thanks for discussing this internally. Here are my responses.

Actions items:
Ask community contributors to provide real world use cases and override designs for further evaluation.

Some examples:

• Allowing users to do synchronization at the pipeline level (issue, Argo Workflows doc)
• Setting TTL at the pipeline level (issue, example from WIP: feat(api/sdk/backend): implement PipelineConfig and setting ttl on pipelines. Fixes: #10899 #10942)
• Allowing users to control caching behavior at the pipeline level instead of on every task (issue)

Ask community contributors to evaluate PlatformSpec as an alternative place to add PipelineConf

I'm fine to put it wherever you all think is best. To clarify, is PlatformSpec the second document at the bottom?

root:
  dag:
    tasks:
      comp:
        cachingOptions:
          enableCache: true
        componentRef:
          name: comp-comp
        taskInfo:
          name: comp
schemaVersion: 2.1.0
sdkVersion: kfp-2.7.0
---
platforms:
  kubernetes:
    deploymentSpec:
      executors:
        exec-comp:
          emptyDirMounts:
          - medium: Memory
            mountPath: /mnt/my_vol_1
            sizeLimit: 1Gi
            volumeName: emptydir-vol-1

And if so, about where should pipeline-level stuff go? Should it be a sibling of kubernetes? Note that this was my approach in #10942 -- quoting from that

# PIPELINE DEFINITION
# Name: test-ttl-pipeline
components:
  comp-empty-component:
    executorLabel: exec-empty-component
deploymentSpec:
  completedPipelineTtl: 120
SNIP

but I (perhaps mistakenly) thought that that was rejected.

[gregsheremeta]

@chensun bumping to the top of your inbox. Can you provide guidance on the spec questions in my previous comment?