Tags are updated for old images

[ianlewis]

It seems that when some kind of change is made all the old image tags are re-built and pushed. We check the image sha for our based images because we don't expect tags in the docker repository to change.

We can pull the old sha, but Docker Content Trust verification fails in that case.

Is it necessary to continuously re-build and push all old image tags?

[ianlewis]

We can pull the old sha, but Docker Content Trust verification fails in that case.

I should say that DCT succeeds but it succeeds with a different SHA than the one we pin.

[ianlewis]

We verify our base images as follows.

img="golang:1.19.4@sha256:cfaad8202aed5121121dfe3a252e98d5c89cc67fc456cc69fe70eb7dcc1b8cff"
image_full=$(echo "$img" | awk '{ print $2 }')
image_name=$(echo "$image_full" | cut -d '@' -f 1)
image_sha=$(echo "$image_full" | cut -d '@' -f 2- | cut -d ':' -f 2-)

DOCKER_CONTENT_TRUST=1 docker trust inspect --pretty "$image_name" | grep "$image_sha"

The full script we use is here:
https://github.com/slsa-framework/slsa-github-generator/blob/f7061e6c0aed5c492527fce8349394cb87b6b4fc/.github/workflows/scripts/verify-base-images.sh

[imjasonh]

I'm going to guess this is happening because other stuff in the image has updates, and those are being rebuilt and pushed back to golang:1.19.4.

As an example, that image contains svn to support some go get scenarios -- if svn has a critical vulnerability, or even just a regular release, the latest version of it should be included in the image. The Go version didn't change when svn was updated, so it's still tagged :1.19.4.

In other words, I don't think Docker shares your expectation that the tag won't change after it's been produced.

If the images had SBOMs it could help narrow down exactly what changed between :1.19.4@sha256:cfaa... and :1.19.4@sha256:660f.... In the absence of those, you can diff the filesystem and try to piece it together.

edit: In this case, it looks like most of the images in the manifest are unchanged between those two digests, only the arm and mips64le images have changed:

diff -C3 \
  <(crane manifest golang@sha256:cfaad8202aed5121121dfe3a252e98d5c89cc67fc456cc69fe70eb7dcc1b8cff | jq ) \
  <(crane manifest golang@sha256:660f138b4477001d65324a51fa158c1b868651b44e43f0953bf062e9f38b72f3 | jq)

[tianon]

Thanks Jason! There are indeed a couple misunderstandings at play here: 😅

1. tags in a container registry are not (and have never been) officially immutable -- they're more similar to Git's branches than Git's tags in that respect (but without an explicit standardized "history")

2. DCT (aka Notary v1) is a signature of a specific tag to a specific digest (in fact, when you enable Notary v1 / DCT, the client bypasses the registry completely for tag resolution and only consults the Notary TUF database instead, which has fun failure modes if the tags stop being signed at some point)

3. official images are (by design) rebuilt frequently, specifically to pick up things like package security updates (as noted by Jason); see How to implement reproducible deployments if images are getting rebuilt? official-images#12277 (comment) for a somewhat longer-form explanation of this and how we suggest users work within it to accomplish their goals -- this most recent rebuild would've been caused by the recent updates to the Debian images (which happen roughly monthly)

Can you elaborate a bit on what you're trying to verify by cross-referencing the current tag's value back to the saved digest? (I don't think I'm fully understanding the use-case.)

[ianlewis]

Hi thanks.

1. tags in a container registry are not (and have never been) officially immutable -- they're more similar to Git's branches than Git's tags in that respect (but without an explicit standardized "history")

I suppose I didn't have this expectation either, but nor did I expect that they would change as frequently as they sometimes do. e.g. daily.

2. DCT (aka Notary v1) is a signature of a specific tag to a specific digest (in fact, when you enable Notary v1 / DCT, the client bypasses the registry completely for tag resolution and only consults the Notary TUF database instead, which has fun failure modes if the tags stop being signed at some point)

This I think is a real downside of DCT. Verifying an image at a tag that changes all the time makes it pretty hard to ensure that you are using an image that has been verified as the tag could change out from under you. With a digest you have guarantees that the image hasn't changed. It's also hard to map between the tag and digest if you are pinning images by digest, because DCT only supports verifying by tag.

It also makes it hard to avoid TOCTOU issues unless you can really ensure that you are using the same image as was verified. For docker this means you need control over docker build and/or docker run and to ensure that DCT is enabled.

Can you elaborate a bit on what you're trying to verify by cross-referencing the current tag's value back to the saved digest? (I don't think I'm fully understanding the use-case.)

Yeah, Right now we are using a Docker container action which builds the image on every GHA run. It would pick up the changes in the base image tag immediately and AFAIK has no way of verifying the images before using it.

We pin the base images by digest and verify the base images using DCT in a pre-submit action. So, since we verify the images used when they are changed in the repo, we want to make sure that the image hasn't changed since the last time it's been verified (thus pinning by digest).

It sounds like, if we want to continue using Docker container action on GitHub we'll need to use pre-built images and verify the base images at build time. And then pin our pre-built image by digest to make sure it hasn't changed since verification (as there is no way to sign our own image and verify it when used by GHA AFAICT).