Include Galaxy tool in container

[innovate-invent]

This is a proposal to expand this project to include the Galaxy tool folder in the container. This is necessary to achieve full Cloud support for Galaxy.

To optimize the size of the containers, these containers can be based off of the mulled container available in biocontainers. The container will need to include in its specification a standardized environment variable pointing to the equivalent of the $__tool_directory__ variable in the tool wrapper xml. The Galaxy job runner can then be extended to refer to this variable in the job script.

An alternative is to generate storage containers that have no other purpose than to store the Galaxy tool folder. This folder is then reverse mounted outside of the container to be mounted by the mulled tool container. Docker in particular supports mounting containers together directly bypassing the host filesystem.

[mvdbeek]

I assume you want this because you don't want to have a shared filesystem that contains tool scripts ?
In that case the answer is pulsar, I don't think we want to build specialized containers for Galaxy.

[innovate-invent]

Pulsar still requires a NFS as far as I can see.

[mvdbeek]

Pulsar can stage the tool dir and needs no NFS.

[innovate-invent]

I looked into Pulsars implementation of operating in a pod and it seems to be hacky at best. Where does it fetch the tool directory from?

Also, including the tool folder in a container layer allows the node to cache it.

[mvdbeek]

https://pulsar.readthedocs.io/en/latest/galaxy_conf.html#data-staging

[innovate-invent]

Ah, it fetches it from the Galaxy app. That isn't optimal.

What is the downside of adding a layer to the mulled tool containers to include the tool folder?
The mulled containers basically already only exist to serve that tool.

[mvdbeek]

So you'd need to generate a container for each tool version, and these containers are used by CWL and nextflow as well. Galaxy can scale serving these datasets pretty well, you'd hit a bottleneck with the database long before serving tool files becomes an issue.

[innovate-invent]

I am proposing adding a layer on top of the mulled container. CWL and nextflow can freely pull the mulled container without the Galaxy tool layer.