Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add dev docs for kfp/sdk #11046

Merged
merged 1 commit into from
Jul 30, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion api/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,16 @@ clean-go:
rm -rf v2alpha1/go
rm -f v2alpha1/google/rpc/status.proto

# Generate Python package.
# Build and locally install Python package.
.PHONY: python
python: v2alpha1/pipeline_spec.proto v2alpha1/google/rpc/status.proto
python3 v2alpha1/python/generate_proto.py && cd v2alpha1/python && python3 setup.py bdist_wheel

# Build and locally install Python package using editable mode for development.
.PHONY: python-dev
python-dev: v2alpha1/pipeline_spec.proto v2alpha1/google/rpc/status.proto
python3 v2alpha1/python/generate_proto.py && cd v2alpha1/python && pip install -e .

# Delete all generated Python packages
.PHONY: clean-python
clean-python:
Expand Down
7 changes: 6 additions & 1 deletion kubernetes_platform/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,16 @@ clean-go:
rm -rf kubernetes_platform/go


# Generate Python package.
# Build and locally install Python package.
.PHONY: python
python: proto/kubernetes_executor_config.proto
python3 python/generate_proto.py && cd python && python3 setup.py bdist_wheel

# Build and locally install Python package using editable mode for development.
.PHONY: python-dev
python-dev: proto/kubernetes_executor_config.proto
python3 python/generate_proto.py && cd python && pip install -e .[dev]
HumairAK marked this conversation as resolved.
Show resolved Hide resolved

# Delete all generated Python packages
.PHONY: clean-python
clean-python:
Expand Down
65 changes: 61 additions & 4 deletions sdk/CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,69 @@
## Contributing to the `kfp` SDK
KFP v2 for SDK is under development on the `master` branch.

To contribute to v1, please go to [sdk/release-1.8 branch](https://github.com/kubeflow/pipelines/tree/sdk/release-1.8) to submit your change.
For developing KFP v2 SDK, use the `master` branch.

For developing KFP v1 SDK, use the [sdk/release-1.8 branch](https://github.com/kubeflow/pipelines/tree/sdk/release-1.8).

For general contribution guidelines including pull request conventions, see [pipelines/CONTRIBUTING.md](https://github.com/kubeflow/pipelines/blob/master/CONTRIBUTING.md).

### Requirements
All development requirement versions are pinned in [requirements-dev.txt](https://github.com/kubeflow/pipelines/blob/master/sdk/python/requirements-dev.txt).
### Pre-requisites
HumairAK marked this conversation as resolved.
Show resolved Hide resolved

Clone the repo:

```bash
git clone https://github.com/kubeflow/pipelines.git && cd pipelines
```

We suggest using a tool like [virtual env](https://docs.python.org/3/library/venv.html) or something similar for isolating
your environment and/or packages for you development environment. For this setup we'll stick with virtual env.

For supported python versions, see the sdk [setup.py](https://github.com/kubeflow/pipelines/blob/master/sdk/python/setup.py).

```bash
# optional, replace with your tool of choice
python -m venv env && source ./env/bin/activate
```

As with any Python package development, first ensure [wheels and setuptools](https://realpython.com/python-wheels/) are installed:

```bash
python -m pip install -U pip wheel setuptools
```

All sdk development requirement versions are pinned in [requirements-dev.txt](https://github.com/kubeflow/pipelines/blob/master/sdk/python/requirements-dev.txt).

```bash
pip install -r sdk/python/requirements-dev.txt
```

Install the SDK in development mode using the `--editable` or `-e` flag. This will allow you to implement and test changes iteratively.
Read more about it [here](https://setuptools.pypa.io/en/latest/userguide/development_mode.html).

```bash
pip install -e sdk/python
```

The SDK also relies on a couple other python packages also found within KFP.
These consists of the [api proto package](https://github.com/kubeflow/pipelines/tree/master/api) and the kfp [kubernetes_platform](https://github.com/kubeflow/pipelines/tree/master/kubernetes_platform) package.

For the proto code, we need protobuf-compiler to generate the python code. Read [here](../kubernetes_platform#dependencies) more about how to install this
dependency.

You can install both packages either in development mode if you are planning to do active development on these, or simply do a regular install.

To install the proto package:
```bash
pushd api
make generate python-dev # omit -dev for regular install
popd
```

To install the kubernetes_platform package:
```bash
pushd kubernetes_platform
make python-dev # omit -dev for regular install
popd
```

### Testing
We suggest running unit tests using [`pytest`](https://docs.pytest.org/en/7.1.x/). From the project root, the following runs all KFP SDK unit tests:
Expand Down
Loading