Create recommend base image tutorial

[pacospace]

Is your feature request related to a problem? Please describe.
As Data Scientist,

I have a notebook for Computer Vision that I want to use in ODH.

As Data Scientist,

I have a stack with opencv and pytorch. I want to know which image should I use to run my Computer Vision Application.

High-level Goals
Have a tutorial on using a recommended base image for my ML application that can be run on Operate First with Meteor.

Describe the solution you'd like
Create steps in the tutorial.

Pre-requisite:

• have prescription for ps-cv-stack (Introduce ps-stacks prescription for ps-cv-pytorch thoth-station/prescriptions#17213)

Steps:

• Fork repo
• Add GitHub Template on the repo
• Set dependencies for the notebook.
• Ask recommendation to Thoth service and take the base image recommended and add it to the .thoth.yaml file (Depends-On: Update thoth config with base image recommended by thoth thoth-station/jupyterlab-requirements#345)
• Run notebook with base image recommended.
• Install AICOE-CI, enable it and adjust .aicoe.yaml.
• Setup images on container registry (e.g. Quay). (Depends-On: How to avoid contacting aicoe-ci maintainers when a new secret for registries is required? aicoe-ci#137)
• Install Khebut and enable issues.
• Create a release.
• Add _toc.yml and _config.yml similar to those in the Elyra Tutorial so that project meteor can build a jupyterbook out of it.

Describe alternatives you've considered

Additional context
Related-To: thoth-station/core#308

Acceptance Criteria

• The tutorial can run on Operate First.

[pacospace]

/assign @pacospace

[harshad16]

More details:
https://chat.google.com/room/AAAAVjnVXFk/YwV3qk8eoxA

[harshad16]

Ideas:

• Having prescriptions (see the above link)
• another use container analysis , lookup on the packages and recommend based on that .

[pacospace]

pre-requisities

• create prescription as suggested by @fridex using advised_manifest_changes https://thoth-station.ninja/docs/developers/adviser/prescription/wraps.html#wrap-run-advised-manifest-changes
• extend Thamos with a new flag to get the base image recommended, only show to user initially

extra steps (not required)
to be converted to issues

• create spike on automation of creation of the prescriptions for ps-stacks and main libraries involved (e.g PyTorch and open-cv for ps-cv)
• next extension for Thamos to actually apply the base image recommended

[fridex]

pre-requisities

• create prescription as suggested by @fridex using advised_manifest_changes https://thoth-station.ninja/docs/developers/adviser/prescription/wraps.html#wrap-run-advised-manifest-changes
• extend Thamos with a new flag to get the base image recommended, only show to user initially

extra steps (not required)

to be converted to issues

• create spike on automation of creation of the prescriptions for ps-stacks and main libraries involved (e.g PyTorch and open-cv for ps-cv)
• next extension for Thamos to actually apply the base image recommended

Thanks for the meeting minutes write-up. First, we can start with prescriptions - they will be easy to write, maintain and can state the main packages for which we want to recommend the base image (e.g. pytorch stack).

Besides that, we should prioritize fix thoth-station/thoth-application#1788 as it is causing wrong data to be synced and associated with the container images we have in the database. It will unlock thoth-station/thoth-application#1788 and a possibility to query packages in container images we build in a separate pipeline unit that can recommend a base image with an exact/partial match for the stack. The prescriptions scenario is still applicable as this solution just extends what we offer to users.

On the user's side, we can have the mentioned flag that will apply changes to the .thoth.yaml and eventually .aicoe-ci.yaml file. We should revisit the current design of advised_manifest_changes to distinguish what data are propagated from the recommender, how they are recognized and subsequently applied to the corresponding files.

[pacospace]

pre-requisities

• create prescription as suggested by @fridex using advised_manifest_changes https://thoth-station.ninja/docs/developers/adviser/prescription/wraps.html#wrap-run-advised-manifest-changes
• extend Thamos with a new flag to get the base image recommended, only show to user initially

extra steps (not required)
to be converted to issues

• create spike on automation of creation of the prescriptions for ps-stacks and main libraries involved (e.g PyTorch and open-cv for ps-cv)
• next extension for Thamos to actually apply the base image recommended

Thanks for the meeting minutes write-up. First, we can start with prescriptions - they will be easy to write, maintain and can state the main packages for which we want to recommend the base image (e.g. pytorch stack).

Thanks @fridex, let's do it!!

Besides that, we should prioritize fix thoth-station/thoth-application#1788 as it is causing wrong data to be synced and associated with the container images we have in the database.

Sounds good, @harshad16 what is ETA for the mentioned issue?

It will unlock thoth-station/thoth-application#1788 and a possibility to query packages in container images we build in a separate pipeline unit that can recommend a base image with an exact/partial match for the stack. The prescriptions scenario is still applicable as this solution just extends what we offer to users.

@fridex it will unlock thoth-station/thoth-application#1788 ?

Do we still want to use prescriptions in that case?

On the user's side, we can have the mentioned flag that will apply changes to the .thoth.yaml and eventually .aicoe-ci.yaml file. We should revisit the current design of advised_manifest_changes to distinguish what data are propagated from the recommender, how they are recognized and subsequently applied to the corresponding files.

Perfect!

[fridex]

@fridex it will unlock thoth-station/thoth-application#1788 ?

thoth-station/thoth-application#1788 will unlock thoth-station/adviser#1533 (also discussed in thoth-station/adviser#1943).

Do we still want to use prescriptions in that case?

Yes, they provide a different view of container images and Python packages shipped in them.

[pacospace]

@fridex it will unlock thoth-station/thoth-application#1788 ?

thoth-station/thoth-application#1788 will unlock thoth-station/adviser#1533 (also discussed in thoth-station/adviser#1943).

Thanks, in the above comment was not linked, now I get it :)

Do we still want to use prescriptions in that case?

Yes, they provide a different view of container images and Python packages shipped in them.

Perfect, then we also want to introduce a mechanism to automatically create prescriptions from ps-stacks analysis.

[fridex]

Perfect, then we also want to introduce a mechanism to automatically create prescriptions from ps-stacks analysis.

A simple script that takes direct dependencies from Pipfiles and generates corresponding prescriptions out of them (as direct dependencies are the ones that in most cases define the container image use) should be good enough here. Later it could be plugged into a bigger workflow if necessary.

[pacospace]

pre-requisities

• create prescription as suggested by @fridex using advised_manifest_changes https://thoth-station.ninja/docs/developers/adviser/prescription/wraps.html#wrap-run-advised-manifest-changes
• extend Thamos with a new flag to get the base image recommended, only show to user initially

extra steps (not required)
to be converted to issues

• create spike on automation of creation of the prescriptions for ps-stacks and main libraries involved (e.g PyTorch and open-cv for ps-cv)
• next extension for Thamos to actually apply the base image recommended

Issues created based on conversations with @fridex .

[pacospace]

blocked by thoth-station/python#418 and quay issues.

[goern]

/kind documentation
/priority critical-urgent

[pacospace]

Depends-On: thoth-station/prescriptions#18523

[pacospace]

Missing steps from Thoth side

• Extend AICOE-CI to store requirements file ( Propogate requirements info to a specific dir aicoe-ci#154 @harshad16 )
• Extend package-extract to store requirements files ( Propagate requirements info as used by AICoE-CI thoth-station/package-extract#459 @fridex )
• Create prescription-refresh-job handler to create specific wrap prescriptions (Add handler for thoth image analysis command thoth-station/prescriptions-refresh-job#97)
• Provide new workflow for prescription refresh in thoth-application (Add prescription refresh workflow for image analysis handler thoth-station/thoth-application#2230)
• Extend adviser logic ([3pt]Provide a generic advise_manifest_changes in results that can be retrieved and used by thamos thoth-station/adviser#2211)
• Provide logic in Thamos to patch base image recommended from prescription (Extend Thamos with a new flag to get the base image recommended thoth-station/thamos#910)

Missing steps from Operate First side:

• Provide an OPF Kubeflow Pipelines instance operate-first/support#435

Tutorial specific:

• create tutorial

[mayaCostantini]

/assign @mayaCostantini

[sesheta]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[sesheta]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten