feat(sdk+api) Implement Semaphores and Mutexes Pipeline Configuration

[DharmitD]

Description of your changes:
This PR introduces semaphoreKey and mutexName fields in PipelineConfig to support pipeline-level concurrency controls in KFP SDK.

Testing instructions

Create a Python virtualenv and install the SDK and IR YAML API packages locally:

$ python -m venv .venv
$ source .venv/bin/activate
$ pip install wheel setuptools protobuf grpcio grpcio-tools
$ pip install -r sdk/python/requirements-dev.txt
$ pip install -e api/v2alpha1/python
$ pip install -e sdk/python

Use the example code to compile

$ kfp dsl compile --py main.py --output main.yaml

You should be able to compile and find the following snippet in the main.yaml file:

---
platforms:
  kubernetes:
    pipelineConfig:
      mutexName: mutex
      semaphoreKey: semaphore

Checklist:

• You have signed off your commits
• The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chensun for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• api/OWNERS
• sdk/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[gregsheremeta]

couple nitpicks but overall lgtm.

I don't want to merge it until the backend PR is posted and close to shipping.

[gregsheremeta]

/hold

lgtm but let's wait until the backend PR is ready to go too
edit: need to use the right protoc version when generating the pipeline_spec files

[HumairAK]

can we add a test where we verify the semaphore setting compiles to the expected IR?

[DharmitD]

Edit: Updated to have the DSL generated SemaphoreKey instead of SemaphoreName field, since having a fixed, hardcoded semaphore name and letting users define the semaphore key is the format prescribed in Argo Workflows' synchronization docs here.

[gregsheremeta]

can we add a test where we verify the semaphore setting compiles to the expected IR?

still need to do this. Here's an example:
https://github.com/kubeflow/pipelines/pull/10913/files#diff-cac76397fcf6a375f3865a864a3f6725b023c69b5216fa1ed71a86b334641399

[DharmitD]

can we add a test where we verify the semaphore setting compiles to the expected IR?

still need to do this. Here's an example: https://github.com/kubeflow/pipelines/pull/10913/files#diff-cac76397fcf6a375f3865a864a3f6725b023c69b5216fa1ed71a86b334641399

Yes I'm working on this ATM. I'm adding a unit test to the compiler_test.py similar to the platform tests here:

pipelines/sdk/python/kfp/compiler/compiler_test.py

Line 3489 in 8a7b0e5

def test_one_task_one_platform(self):

Have a draft ready, testing locally and pushing it in a bit.

[DharmitD]

can we add a test where we verify the semaphore setting compiles to the expected IR?

still need to do this. Here's an example: https://github.com/kubeflow/pipelines/pull/10913/files#diff-cac76397fcf6a375f3865a864a3f6725b023c69b5216fa1ed71a86b334641399

Yes I'm working on this ATM. I'm adding a unit test to the compiler_test.py similar to the platform tests here:

pipelines/sdk/python/kfp/compiler/compiler_test.py

Line 3489 in 8a7b0e5

def test_one_task_one_platform(self):

Have a draft ready, testing locally and pushing it in a bit.

Added a unit test to verify semaphore and mutex setting, the test works as expected and passes on my local venv, but fails in this PR's CI.

Wondering if it's because the .proto changes haven't been merged into the code base yet. @gregsheremeta could you take a look and confirm?
And if it is so, then should the test be added in a follow up PR?

[gregsheremeta]

ml_pipelines.PipelineConfig" has no field named "semaphoreKey" at "SinglePlatformSpec.pipelineConfig".

So yes. The test can stay here. Tests just won't pass until the proto is merged and this is rebased. It's the normal/usual/expected flow.