Is it possible/supported/problematic to use cache build mounts (--mount=type=cache) to store installed packages?

[tboddyspargo]

My team has been experiencing some pain with building and re-building images locally during development when layers are invalidated. We have a very large python+R service with many dependencies which can take up to 40mins to build. I've leveraging build cache mounts as mitigation tools to get that time down to 10-20mins depending on which layers are invalidated, but we have generally been sticking with the examples in documentation and from the wider community.

At this point, the slowest part of the build process for us is the installation of python packages (not downloading or building wheels). In contrast, when we install these same packages with pip locally, there is rarely a repeated time cost because pip can tell when a package has already been installed and doesn't need to be re-installed. In order to accomplish similar behavior during image build, we would need a mechanism to persist the package installation directory between builds so that pip could reuse them instead of re-installing. I decided to try using build cache mounts to accomplish this.

However, by installing all pip dependencies into a cache mount, they were no longer part of the image. To work around this, I chose to copy the virtual environment to a directory outside the cache mount at the end of the RUN command that used it. This way, I could take advantage of the cache mount to reduce time spent installing dependencies, but keep a copy of the virtual environment in the image to be picked up by another stage in our multi-stage image.

Solution

Simplified for readability

FROM base as pip_install

RUN python -m venv /opt/venv

RUN --mount=type=cache,target=/root/.cache \
    --mount=type=cache,target=/opt/venv \
    python -m pip install -r requirements.txt \
    # Copy the venv directory out of the cache mount so it can be used in future stages.
    && mkdir -p /output \
    && cp -r /opt/venv /output

FROM base as final

# Place the venv directory back in its original location so that absolute paths to packages/shims are accurate again.
COPY --from=pip_install /output/venv /opt/venv

COPY /app ./

ENTRYPOINT ['python', '-m', 'main']

Using this approach, I was able to reduce image re-builds to 2-4 minutes, which would go a long way to improving our developer experience when cached image layers are invalidated.

Questions

I feel a bit uncomfortable about this solution because I haven't heard of other people caching python virtual environments or other package installation locations to improve build speed. Additionally, it doesn't really seem like the way this feature was intended to be used, though it seems to work well enough. What I'm wondering is:

1. Do you see anything concerning about my approach?
2. Can you point me to other examples of community members using build cache mounts to improve image build speed by caching installed packages?
3. Would you have other suggestions for different/better ways to reduce the time spent installing dependencies during image build?

Thank you very much in advance for any input/advice you might have!

[lagergren]

Did you ever get a good answer for this?

[tboddyspargo]

No, but we've been using in practice for a while and it's been fine.

I will say that for pip packages that have pre-built wheels, the installation time shouldn't be too bad. It's only the ones that need to be built from source whose installation time is particularly painful. As we upgrade our python and package versions to take more advantage of pre-built wheels, I'll be re-evaluating this approach and trying to avoid a cache mount for the venv and copying it around multiple times.

[lagergren]

Thanks for the info!

Yes I have some source installations, but not being too familiar with this ecosystem, I have trouble finding a "best case solution". The thing that confused me the most is that the actual copy operation to a new stage is so incredibly slow.