Build a Pipeline

This page describes how to create a pipeline for deployment in Kubeflow, using the provided sample pipelines.

Create a python3 environment

Python 3.5 or above is required. If you don't have Python3 set up, we suggest the following steps to install Miniconda.

In a Debian/Ubuntu/Cloud shell environment:

apt-get update; apt-get install -y wget bzip2
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh

In a Windows environment, download the installer and make sure you select "Add Miniconda to my PATH environment variable" option during the installation.

In a Mac environment, download the installer and run the following command:

bash Miniconda3-latest-MacOSX-x86_64.sh

Create a clean python3 environment:

conda create --name mlpipeline python=3.6
source activate mlpipeline

If the conda command is not found, be sure to add the Miniconda path:

export PATH=MINICONDA_PATH/bin:$PATH

Install the Kubeflow Pipelines SDK

Run the following:

pip install https://storage.googleapis.com/ml-pipeline/release/0.0.26/kfp-0.0.26.tar.gz --upgrade

After successful installation the command "dsl-compile" should be added to your PATH.

Compile the samples

The pipelines are written in Python, but they must be compiled to an intermediate representation before submitting to the Kubeflow pipelines service.

dsl-compile --py [path/to/py/file] --output [path/to/output/tar.gz]

For example:

dsl-compile --py [ML_REPO_DIRECTORY]/samples/basic/sequential.py --output [ML_REPO_DIRECTORY]/samples/basic/sequential.tar.gz

Deploy the samples

Upload the generated .tar.gz file through the Kubeflow pipelines UI.

Advanced: Build your own components

See how to build your own pipeline components.