Bumping from static latest yesterday to the release 18 hours ago breaks build

[tomski0]

Describe the bug
FROM gcr.io/distroless/static COPY bin/binary /app CMD [ "/app" ]

This worked yesterday.

Running it today returns an error when executing the container:

InvalidImage(MissingParentDirectory: Parent directory does not exist for file: ./usr/lib/ssl/cert.pem)

To Reproduce

docker build
docker run

Expected behavior
The container should still build with the latest version

Console Output
InvalidImage(MissingParentDirectory: Parent directory does not exist for file: ./usr/lib/ssl/cert.pem)

Additional context
Replacing FROM gcr.io/distroless/static with gcr.io/distroless/static:latest@sha256:95eb83a44a62c1c27e5f0b38d26085c486d71ece83dd64540b7209536bb13f6d returns to a working state.

For completion the build used FROM gcr.io/distroless/static:latest@sha256:b033683de7de51d8cce5aa4b47c1b9906786f6256017ca8b17b2551947fcf6d8 on the failed build so it is referencing back to the latest tag.

[loosebazooka]

Ah interesting. Must've been a rules distroless change.

[loosebazooka]

This doesn't seem to be affecting my builds. Are you using a old or very strict runtime?

[tomski0]

Runtime is Go 1.22. It's an AWS Lambda function backed by Docker if that helps

[slagiewka]

We had the same issues with Lambda recognising invalid layers. Also Go (1.23) builds running on static.

Pinning to gcr.io/distroless/static@sha256:95eb83a44a62c1c27e5f0b38d26085c486d71ece83dd64540b7209536bb13f6d helped.

[Janosch]

I encountered the same issue with gcr.io/distroless/python3. I would like to pin distroless/python3 to a previous version to fix this. How can I find out the hashes of previous versions of distroless/python3 for linux/amd64?

[slagiewka]

I encountered the same issue with gcr.io/distroless/python3. I would like to pin distroless/python3 to a previous version to fix this. How can I find out the hashes of previous versions of distroless/python3 for linux/amd64?

We were lucky enough to have CodeBuild logs with docker build outputs where the previously used digest was available. It removed some guess work as to which digest was the exact one in use.

You can browse all of them here: https://console.cloud.google.com/artifacts/docker/distroless/us/gcr.io/python3

There's a chance that you still have a previous version stored locally so try browsing there, too.

[Janosch]

@slagiewka I just started using distroless therefore I don't have any previous versions locally.

I know that I can browse them online, but it is not really helpful as I would need to randomly try hashes, hoping that I pick the right one (root/nonroot, debug/non-debug, amd64/arm64). Or am I missing something?

[loosebazooka]

Yeah, we don't have a great system for this: see #1360 and #686

[loosebazooka]

Kk I think this is fixed. Can you try again?

[tomski0]

@loosebazooka looking good at my end.