Use python slim instead of Cuda base image

[technillogue]

One weird trick to save 20GB of user bandwidth and cut our cold start times in half.

I'm not sure how to set up the config to default to false though.

[andreasjansson]

Defaulting use_cuda_base_image to true means that we're completely backwards compatible but people might not discover that they can get a much smaller image.

Defaulting to false means that some high percentage of users will get smaller images, but for some low percentage they'll get a really hard-to-debug error message.

This is tricky... We could improve those percentages by magically setting use_cuda_base_image based on presence of tensorflow or jax, but that makes the hard-to-debug error message even more opaque for people who actually need the cuda devel base image.

Unrelated, but we've also had people ask to use custom base images. I'm not sure if we should encourage that since it most likely creates even harder-to-debug errors.

[mattt]

@technillogue Thanks so much for putting this together. I'm encouraged by how far we can get by moving away from CUDA dev base images.

I originally imagined this as a build flag (cog build --use-cuda-base-image=false). Making that work is more involved, but may be a better fit. mattt/use-cuda-base-image-flag

Expanding on what @andreasjansson said, and following up on our conversation yesterday, I think we should start by implementing this as an opt-in feature. That solves an immediate need we have, and gives us a good opportunity to continue validating this functionality without the risk of breaking existing Cog models.

At the same time, this should be a big improvement for a majority of users, and it'd be nice to migrate everyone to this sooner than later. I wrote up an idea for how we could solve that here: #1207

[technillogue]

mattt/use-cuda-base-image-flag looks really good, I would be really happy to have that while we figure out the right way to handle #1207. I hadn't thought about what to do with the cuda and cudnn options. We might still need cuda, separately from the base image, for picking the right torch for users (since we're relying on torch's CUDA libraries).

Like we discussed, we'd like to also support custom dockerfiles that we add labels to, but not custom base images that might not have apt.

[bfirsh]

Could this be detected automatically? Andreas hints at a hard to debug error message. Could we automatically detect that error message and hint at the solution?

I like the idea of trying to do as much automatically as possible, then giving users a way to bail out if they need to.

[mattt]

#1214 implements this as a --use-cuda-base-image build flag that defaults to "auto", which we currently take to be true in all cases.

@bfirsh, to your point, we can absolutely iterate towards making automatic behavior better. We should eventually have it so that "auto" implies false in most cases, except the ones @andreasjansson alluded to where there are known issues.

@technillogue has a PR to allow users to specify a custom base image in #1229.