Accessing plugin dependencies that are contained within a docker

[callum-jpg]

When running deep learning-based plugins the installation can be tricky. Moreover, running StarDist and Cellpose in the same environment is also problematic, given the tensorflow and pytorch dependencies, respectively.

So, does it make sense to have a version of these plugins that interacts with a docker image (eg. RunCellposeDocker)? That is, StarDist/Cellpose are contained within a docker image and images are passed through to the docker via a mounted volume.

I quickly tried this with Cellpose using subprocess to pull and run the image and it works well. There's also a python api for docker. In this process, the input image is saved using skimage to the mounted directory, segmented using Cellpose CLI, and then the output can be read into the plugin again using skimage. As a result, the only lift for users would be for users to install Docker, which is quite easy.

Of course, the saving and reading of files in the middle of a module is a bit weird and slower than creating an environment, but if a user just wants access to Cellpose segmentation with CellProfiler's other modules, I think this makes it as easy as possible. You can also imagine a scenario in which you have multiple plugin modules with conflicting dependencies working together, eg. StarDist + Cellpose + DeepProfiler in the same pipeline, all calling out to their respective docker containers.

Another limitation of this is that the docker image size will be considerably larger on disk than the equivalent in the users home python environment. However, I'd also think this downside is negligible given the ease of use of the docker.

You could even abstract this idea further by having some RunDocker module where you input a docker image name, the command you want to run and the data that will be passed to it.

Here's what the subprocess docker run looks like for Cellpose:
subprocess.check_call("docker run -it --rm -v ${PWD}/img:/img ctromanscoia/cellpose:0.1 cellpose --dir /img --pretrained_model nuclei --verbose", shell=True)

[bethac07]

I really REALLY like this idea! I had brought this up in the context of some of our other workflows like the profiling recipe Broad-only link 1 Broad-only link 2, but it has obvious advantages for plugins as well. Both the Docker and the Singularity API (which is what the code snippet I linked below is from ) have minimal dependencies (Singularity looks like it may not even actually have any?), we'd need to dig into license compatibility, but subprocess obviously works too.

Certainly performance is going to be a major thing requiring investigation - startup times may be non-trivial, and is the system going to go crazy if we're spinning up Dockers everywhere? But rather than having two plugins, I wonder if we could just have a single plugin with a toggle switch ("Docker/python"); it seems like that would be ideal.

spython shell docker://cytominer/profiling-recipe:tag
client.run(bind=['/local/input:/container/input', '/local/output:/container/output'])

[callum-jpg]

I was wondering about the spinning up of lots of dockers too. It seems like you can handle this scenario with the docker python API with a conditional like: if any("cellprofiler/cellpose-plugin" in c.name for c in client.containers.list()):, which would check if the container was already running. If so, pass the command with container.exec_run("cellpose..."), else start the container with client.containers.run(cellprofiler/cellpose-plugin...).

Passing of files would have to be standardized in some way, but I was imagining either overwriting the file if it exists, or deleting it after it's been segmented. Unsure what the typical strategy is for temp files like this.

I like the docker/python option in the module. I'll have a go at experimenting with docker and singularity to see what it's like to use and how the performance/start-up times etc are.

[bethac07]

Yeah, you'd want to benchmark starting up containers vs leaving containers open.

I think in the long run the APIs are likely better, but the advantage of using subprocess is that it's immediately useable in any CellProfiler (4) out there, no extra dependencies. We could think about a subprocess version in 4 and then moving to a nicer, api-based version for 5.

FWIW, the RunImageJMacro (not Script) module is basically exactly what you want to do here- take in images and parameters, save out an image, give those images and parameters to some external program with subprocess, bring in the file it makes, load in the file and make it a CellProfiler image - so I would borrow from that HEAVILY in your version, if you write one.