rudimentary SIF image cache for apptainer/singularity runners

[tomeichlersmith]

Is your feature request related to a problem? Please describe.
I don't like how many copies of a SIF image I need to have. For example, if I have two projects that both want to use the python:3.12 image for the denv, they both need to have their own copy of it within .denv/images.

Describe the solution you'd like
I think we can piggy-back on APPTAINER_CACHEDIR and keep the denv images within some subdirectory there so images can be shared between multiple denvs held by a single user. The downside then is that if one denv chooses to update an image (with denv config image pull) this could effect other denvs using the same image.

Describe alternatives you've considered
Users can still avoid image duplicated by using symlinks. They could create their own location of SIF files and then pass the full path to the SIF file to denv when initializing or configuring. I don't like this solution because it requires the user to apptainer build themselves which is less ergonomic than letting denv get the OCI image they want into the correct format.

Additional context
If implemented, this will also hopefully improve the speed of the apptainer/singularity tests since the OCI-SIF image conversion will only need to happen once per image name rather than for every test.

[tomeichlersmith]

I'm still thinking about if this is a good idea. The main reason I chose to avoid this when first writing denv was because cache maintenance is hard.

• What happens if the cache is on a filesystem that goes down?
• How to handle cache hits/misses?
• Can users choose to disable the cache? Where should the file be stored then?
• What about asynchronous updates to the cache (like the example I give with denv config image pull)?

These questions still leave me unconvinced that have a shared cache is a good idea. Especially this last point which goes almost completely against the whole idea of denv enabling the user a fixed environment. I could avoid this issue by having the deduced SIF filename be much more specific (e.g. using the image SHA or something) but that introduces a lot of complexity into this code that is meant to be simple. I might just close this issue after writing some more tips and tricks into the documentation. I could also see OCI-like caching being supported by apptainer in the future which would then allow denv to avoid building the SIF image at all (letting apptainer worry about if a new one is needed or not).

Currently, apptainer does support building a cached SIF image for you from an OCI image. The downside is two fold.

1. There is an ugly INFO statement printed by apptainer which may be confusing to some users.
2. I think the registry is checked on every run even if the SIF image is in the cache. This is the actual problem since it means denv couldn't (for example) be run in scripts using apptainer's SIF image caching without potentially running into rate limiting from DockerHub or GitHub Container Registry (I've run into this before using apptainer on a different containerization project) and presumably means that the SIF image would be updated if the image tag on the registry is moved (e.g. latest moving to a new release) which is undesirable from a user who wants their image fixed. I have not been able to reproduce this rate limiting issue so this might have been patched since I saw it a few years ago.

eichl008@spa-cms016 /export/scratch/users/eichl008/denv/apptainer-oci> apptainer run alpine:latest /bin/sh                                                                                   
FATAL:   While checking container encryption: could not open image /export/scratch/users/eichl008/denv/apptainer-oci/alpine:latest: failed to retrieve path for /export/scratch/users/eichl008/denv/apptainer-oci/alpine:latest: lstat /export/scratch/users/eichl008/denv/apptainer-oci/alpine:latest: no such file or directory                                                          
eichl008@spa-cms016 /export/scratch/users/eichl008/denv/apptainer-oci> apptainer run docker://alpine:latest /bin/sh                                                                          
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Copying blob 43c4264eed91 skipped: already exists
Copying config d5dccdd0a1 done   |
Writing manifest to image destination
2024/10/25 15:34:49  info unpack layer: sha256:43c4264eed91be63b206e17d93e75256a6097070ce643c5e8f0379998b44f170                                                                              
INFO:    Creating SIF file...
Apptainer> exit
eichl008@spa-cms016 /export/scratch/users/eichl008/denv/apptainer-oci> apptainer run docker://alpine:latest /bin/sh                                                                          
INFO:    Using cached SIF image
Apptainer> exit
eichl008@spa-cms016 /export/scratch/users/eichl008/denv/apptainer-oci> apptainer version
1.3.4-1.el8

Edit: I've confirmed via netstat that apptainer still makes some internet connection even after the image is in the cache. I still can't get a rate limit error even after running for awhile but this still seems undesirable to me. More thorough checking is required before I see if that is a problem.