Kubeflow examples - Notebooks, Pipelines, Models, Model tuning and more
Storage on k8s is a complex topic and deep dive is outside of scope of this repository. There are two types of storage you might encounter here - block and object storage.
Is the usual type of storage a k8s pod might mount to persist data. An example in the context of this repo are Kubeflow Notebook volumes.
Object storage is any S3-like type of storage. Kubeflow Pipelines use object storage
extensively to store intermediate and final task/pipeline artefacts. Furthermore, KServe can be configured
to serve models directly from object storage.
Kubeflow (if deployed viamanifests) comes with MinIO
built in. Due to MinIO license change the bundled version is quite old, but still functional. Alternatively,
admins can configure Kubeflow Pipelines to use other instances of object storage (e.g. self-served MinIO, AWS S3,
GCS, etc.). Relevant for end users it to know that many S3 libraries use environmental variables for their
configuration.
Those are usually: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and S3_ENDPOINT. They are likely already
available in your environment. You can also ask your admin about them or reach out to us -> prokube comes
pre-configured with integrated storage.
All code contributions should go via pull requests. Make sure your code is clearly documented and that it adheres to established standards (e.g. PEP).
Since this repo contains Jupyter notebooks we use nbstripout as pre-commit hook so all notebooks are stripped of cell outputs. Set it up locally for yourself with:
pip install --upgrade nbstripout
pip install pre-commit
pre-commit installThis should enable the hooks.
Use pre-commit run --all-files to run the hooks.