Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce ps-stacks prescription for ps-cv-pytorch #17213

Merged
merged 4 commits into from
Sep 22, 2021
Merged

Introduce ps-stacks prescription for ps-cv-pytorch #17213

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7b2a952
Introduce ps-stacks prescriptions
Sep 16, 2021
fe540af
Update prescription to just show justification
Sep 17, 2021
eeea6ee
Add manifest change
Sep 17, 2021
d3803df
Use boot not wrap unit
Sep 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions prescriptions/_predictable_stacks/ps_cv_torch.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
units:
boots:
- name: ThothPSCVTorchBaseImageBoot
type: boot
should_include:
adviser_pipeline: true
runtime_environments:
base_images: [null] # Only if no base image is used.
match:
- package_name: opencv-python
- package_name: torch
run:
log:
type: INFO
message: Found predictive stack image that can be used with these dependencies
stack_info:
- type: INFO
message: Consider using predictive stack for computer vision with Pytorch
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the match part, this message will be shown also when tensorboard, jupyter-tensorboard or pillow will be resolved. Is it expected?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe not, maybe only when any of those are combined with the main packages, Pytorch and opencv. I will remove them then. Thanks @fridex !

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about adjusting it to make it more generic - not specific to Pytorch, but to a computer vision stack?

Copy link
Contributor Author

@pacospace pacospace Sep 21, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe not, maybe only when any of those are combined with the main packages, Pytorch and opencv. I will remove them then. Thanks @fridex !

Actually thinking a little bit more on this, if their software stacks is a subset of the list of packages stated in the prescription, we could recommend a base image (or more in the future maybe, sorted by?), so it might make sense to keep also the others because if they have have more packages that those ones in the prescription (direct one) we won't recommend it right?, wdyt?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The packages can be kept in the prescription as the recommender system can provide suggestions on which base image to use. The justification message can be adjusted to be more generic, not specific to Pytorch. Users can use the container image if it provides used packages.

if they have have more packages that those ones in the prescription (direct one) we won't recommend it right?, wdyt?

We can still advertise the base images and users can build on top of them. If the intention is to ecommend the base image based on the direct dependencies, you can provide boot prescription unit that can act on direct dependencies used in the application - see https://thoth-station.ninja/docs/developers/adviser/prescription/boots.html#boot-match

Copy link
Contributor Author

@pacospace pacospace Sep 21, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The packages can be kept in the prescription as the recommender system can provide suggestions on which base image to use. The justification message can be adjusted to be more generic, not specific to Pytorch. Users can use the container image if it provides used packages.
Well this prescription is for Computer Vision with Pytorch image:

      justification:
      - type: INFO
        message: Consider using predictive stack for computer vision with Pytorch
        link: https://quay.io/repository/thoth-station/ps-cv-pytorch

if they have have more packages that those ones in the prescription (direct one) we won't recommend it right?, wdyt?

We can still advertise the base images and users can build on top of them. If the intention is to ecommend the base image based on the direct dependencies, you can provide boot prescription unit that can act on direct dependencies used in the application - see https://thoth-station.ninja/docs/developers/adviser/prescription/boots.html#boot-match

Is this a different prescription to be prepared? Isn't the type: wrap doing that purpose?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a different prescription to be prepared? Isn't the type: wrap doing that purpose?

The same way as Kubernetes provides different objects, such as Job or Deployment, the same way the resolver provides types of pipeline units as an abstraction to configure the resolution process. wrap is executed at the end of the resolution process on a fully resolved set of packages whereas boot is executed at the beginning of the resolution process on direct dependencies (just for completeness: there are other inputs that they can act on, but that is not relevant for this case).

Now, the difference in wrap.match part - this configuration will add the justification mentioned when jupyter-tensorboard OR torch is present:

    match:
    - state:
        resolved_dependencies:
        - name: jupyter-tensorboard
     - state:
        resolved_dependencies:
        - name: torch

Whereas this configuration will add the justification when jupyter-tensorboard AND torch are present in the resolved set of packages at the same time:

    match:
    - state:
        resolved_dependencies:
        - name: jupyter-tensorboard
        - name: torch

Additional version ranges and index configuration can be applied.

To recommend the base image, it might be good to pick "representatives" that describe the container image the best. This way, we can recommend a container image for computer vision with pytorch if pytorch is present. On the other hand, jupyter-tensorboard might not be a good package to justify recommending pytorch specific computer vision container images.

This might be a good start, then we can perform a more complicated things as discussed in AICoE/recommend-base-image-tutorial#1

Copy link
Contributor Author

@pacospace pacospace Sep 22, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the detailed explanation @fridex, now I understand :) and I think we can start with the boot then and when we will do something more complex later we can make it a warp (maybe when we analyze ps- images and learn specific versions per tags, so we can recommend more precise image tag), wdyt?

I agree with you on the use of "representatives"! I will modify the PR, thanks!!

link: https://quay.io/repository/thoth-station/ps-cv-pytorch