Kubeflow 1.10 Enterprise grade Agenda

[juliusvonkohout]

Validation Checklist

• Is this a Kubeflow issue?
• Are you posting in the right repository ?
• Did you follow the installation guide https://github.com/kubeflow/manifests?tab=readme-ov-file ?
• Is the issue report properly structured and detailed with version numbers?
• Is this for Kubeflow development ?
• Would you like to work on this issue?
• Join our slack channel using wg-manifests.

Version

master

Describe your issue

For general Kubeflow:

• Missing namespace in authorizationpolicy Missing namespace in authorizationpolicy #2298
• performance issues with admission webhook kubeflow#6708
• Renaming Notebooks to workbenches as discussed with Kimonas Sotirchos. There are not only Jupyterlabs but also Rstudios and Vscodes and other customer specific applications (e.g. Labelstudio). If possible make the istio-virtualservice redirect path configurable instead of always using NB_PREFIX.
• Docker -> OCI Update form-image.component.html Docker -> OCI Update form-image.component.html kubeflow#6673
• The kserve logo does have a different color than the other logos chore(icons): the model icon is grey instead of black kubeflow#6656
• oidc-authservice https://github.com/kubeflow/manifests/pull/2150 and https://github.com/kubeflow/kubeflow/pull/6609 for proper programatic authentication via tokens and serviceaccounts
• Remember the most recently selected namespace Remember the most recently selected namespace kubeflow#6663
• There is also another meta issue now Notebooks WG Roadmap for KF 1.7 kubeflow#6672

Security:

• Harden the istio 1.16 configuration #2357
• Add authorization mechanisms in new Katib UI backend katib#1983.
• Protection from fake kubeflow-userid header impersonation kubeflow#6702
• Add tensorboards-web-app.yaml networkpolicy #2304
• Are you open to a rootless istio (istio-cni) for Kubeflow 1.7 ? Currently almost everything runs insecurely as root. I could also add offical Kubernetes podsecuritystandards: restricted then for Kubernetes 1.23+. https://www.arrikto.com/blog/securing-your-kubeflow-deployment-with-kyverno-policies/?utm_content=222104897&utm_medium=social&utm_source=linkedin&hss_channel=lis-ZPwp5HFOSx is also interesting. But they seem to not know how dangerous root containers are. I think for kubeflow PSA /PSS/ Kyverno are the best options.
• Namespace sharing is completely broken from a security perspective. You can easily escalate from default-viewer to default-editor. Furthermore a malicious collaborator can easily extract the serviceaccount token, leave the company and still login with that as bearer token and impersonate other users. Very dangerous. I tried a lot, but the only sane way if found is to add a checkbox on the collaborator page that says "Delete all secrets, serviceaccounts and pods when removing a collborator via kubectl delete serviceaccounts,secrets,pods --all --force. The default secrets and serviceaccounts will be immideately recreated by the Kubeflow profile controllers, but manually created ones are your own resposibility. This might lead to a very short service interuption for e.g. KFP and other Kubeflow components." We could just display this as Information or try to implement it in the profile controller.
• Katib Narrow and explicit RBAC Narrow and explicit RBAC katib#1639
• The pipeline UI allows reading other peoples artifacts. The artifact proxy in the user namespace is insecure and obsolete. In the UI you can just get the artifact link from another user, remove the ?namespace=xxx parameter at the end and the UI server will fake the corresponding user for you. So if you know the S3/GCS path you can read other guys artifacts.

For Pipelines maybe :

• [feature] Seek alternative for object store solution other than MinIO [feature] Seek alternative for object store solution other than MinIO pipelines#7878
• Proper cache configuration for v2 pipelines too feat(backend) V1 Pipelines add maximum_cache_staleness and default_cache_staleness pipelines#8270
• Support big (5+ GB) s3 artifacts in pipelines EOF when many steps using same big input artifact argoproj/argo-workflows#9525 and Download large file from S3 cause memory rise in the init container argoproj/argo-workflows#1322 (they are not solved)
• feat(backend): isolate artifacts per namespace/profile/user using only one bucket feat(backend): isolate artifacts per namespace/profile/user using only one bucket pipelines#7725 We need to add gcs and aws s3 support to get it merged
• implement a caching strategy for PVC data_passing_method. (https://kubeflow-pipelines.readthedocs.io/en/latest/_modules/kfp/dsl/_pipeline.html?highlight=data_passing_method#). For example if one uses a pvc named after the pipeline for all executions of that pipeline versions this could be useful.
• [Multi User] Support separate metadata for each namespace pipelines#4790

Good first issues

• listruns performance is bad because of shitty SQL queries [backend] severe performance problem in listruns API pipelines#9890 please use fix(backend): Fix performance issue within a mysql request pipelines#9680 as a reference. Here the tale begins https://github.com/kubeflow/pipelines/blob/ad058b5321f5fe74bbd10845778e6627fb9aa5cb/backend/src/apiserver/storage/run_store.go#L109. Maybe the frontend is not using pagination as well. I do not know where the bottleneck is.
• Try out kfp v2 rootless fix(backend): set correct permissions for local directory pipelines#7916
• SDK support for v1 namespaced pipelines feat(sdk): Upload namespaced pipeline definitions. Part of #4197 pipelines#8512
• Reviewing Improve the performance of most Kubeflow backends web-apps: Implement watch to avoid constant polling from backend to K8s kubeflow#7080
• Workbenches as successor to notebooks Kubeflow Notebooks 2.0 (Kubeflow Workspaces) notebooks#85
• General culling in old "noteboks" Notebook Culling for VSCode/CodeServer style notebooks notebooks#93
• Support big (5+ GB) s3 artifacts in pipelines EOF when many steps using same big input artifact argoproj/argo-workflows#9525 and Download large file from S3 cause memory rise in the init container argoproj/argo-workflows#1322 (they are not solved)
• admission webhook performance performance issues with admission webhook kubeflow#6708
• Ray istio/headless [Feature] Support Istio ray-project/kuberay#1005

Steps to reproduce the issue

Install Kubeflow

Put here any screenshots or videos (optional)

No response

[rimolive]

Can we work on an alternative Container Registry to overcome the Docker Hub rate limit issues?

[juliusvonkohout]

@rimolive Yes, @thesuperzapper wants ghcr if I remember correctly and I also think that GitHub itself as registry might be a good choice.

[andreyvelich]

I think, we should discuss pros and cons to use Docker Hub vs ghcr.
Since we can ask CNCF for subscription for Docker Hub account for our official Kubeflow DockerHub: https://hub.docker.com/orgs/kubeflow, it might solve an issue with Docker Hub rate limits. It gives 5000 pulls per day for every user.

[kimwnasptd]

@juliusvonkohout should we also warm up again the discussion about groups support, now that the first step of oauth2-proxy is merged and we are passing id-tokens on requests?

kubeflow/dashboard#42
#2910

[juliusvonkohout]

@juliusvonkohout should we also warm up again the discussion about groups support, now that the first step of oauth2-proxy is merged and we are passing id-tokens on requests?

kubeflow/dashboard#42 kubeflow/manifests#2910

Yes please :-) Lets discuss it in the Kubeflow platform/manifests meeting. We really have to think the architecture trough.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[juliusvonkohout]

/lifecycle frozen

[juliusvonkohout]

I merged the remaining issues into #2763