Make oras push / oras attach deterministic

[shizhMSFT]

What is the version of your ORAS CLI

v1.2.0-beta.1

What would you like to be added?

Deterministically generate manifests for oras push and oras attach if the same content (e.g. blobs, annotations) are packed.

Related issue: oras-project/oras-go#748

The request is to document how to do deterministic build. It would be better if we can introduce a flag to make --annotation org.opencontainers.image.created=$(date -d @$(stat -c %Y <artifact_path>) -u +"%Y-%m-%dT%H:%M:%SZ") simpler.

Why is this needed for ORAS?

With deterministic builds (a.k.a. reproducible builds), the oras push command will not push two different manifests. Deterministic builds also play an important role in CSSC (see blog).

Current behavior:

$ echo hello world > hello.txt
$ oras push --oci-layout hello hello.txt
✓ Uploaded  application/vnd.oci.empty.v1+json                                                      2/2  B 100.00%    5ms
  └─ sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
✓ Uploaded  hello.txt                                                                            12/12  B 100.00%    4ms
  └─ sha256:a948904f2f0f479b8f8197694b30184b0d2ed1c1cd2a1ec0fb85d299a192a447
✓ Uploaded  application/vnd.oci.image.manifest.v1+json                                         588/588  B 100.00%    1ms
  └─ sha256:081c07dd781982968e29c2a1c6a5879511dec81f916913c5eeb28f521aed7793
Pushed [oci-layout] hello
ArtifactType: application/vnd.unknown.artifact.v1
Digest: sha256:081c07dd781982968e29c2a1c6a5879511dec81f916913c5eeb28f521aed7793
$ oras push --oci-layout hello hello.txt
✓ Exists    hello.txt                                                                            12/12  B 100.00%     0s
  └─ sha256:a948904f2f0f479b8f8197694b30184b0d2ed1c1cd2a1ec0fb85d299a192a447
✓ Exists    application/vnd.oci.empty.v1+json                                                      2/2  B 100.00%     0s
  └─ sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
✓ Uploaded  application/vnd.oci.image.manifest.v1+json                                         588/588  B 100.00%  341µs
  └─ sha256:faf86564f14a14aac0cb377437d9129ce26fb69b75c6269117810331a8cf32b4
Pushed [oci-layout] hello
ArtifactType: application/vnd.unknown.artifact.v1
Digest: sha256:faf86564f14a14aac0cb377437d9129ce26fb69b75c6269117810331a8cf32b4

With deterministic builds:

$ echo hello world > hello.txt
$ oras push --annotation org.opencontainers.image.created=$(date -d @$(stat -c %Y hello.txt) -u +"%Y-%m-%dT%H:
%M:%SZ") --oci-layout hello hello.txt
✓ Uploaded  hello.txt                                                                            12/12  B 100.00%    2ms
  └─ sha256:a948904f2f0f479b8f8197694b30184b0d2ed1c1cd2a1ec0fb85d299a192a447
✓ Uploaded  application/vnd.oci.empty.v1+json                                                      2/2  B 100.00%    2ms
  └─ sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
✓ Uploaded  application/vnd.oci.image.manifest.v1+json                                         588/588  B 100.00%  716µs
  └─ sha256:a7580b6441d5e96f80076ee7cbbace168c4efecadfca299bad4f15a635d5a7b1
Pushed [oci-layout] hello
ArtifactType: application/vnd.unknown.artifact.v1
Digest: sha256:a7580b6441d5e96f80076ee7cbbace168c4efecadfca299bad4f15a635d5a7b1
$ oras push --annotation org.opencontainers.image.created=$(date -d @$(stat -c %Y hello.txt) -u +"%Y-%m-%dT%H:%M:%SZ") --oci-layout hello hello.txt
✓ Exists    application/vnd.oci.image.manifest.v1+json                                         588/588  B 100.00%     0s
  └─ sha256:a7580b6441d5e96f80076ee7cbbace168c4efecadfca299bad4f15a635d5a7b1
Pushed [oci-layout] hello
ArtifactType: application/vnd.unknown.artifact.v1
Digest: sha256:a7580b6441d5e96f80076ee7cbbace168c4efecadfca299bad4f15a635d5a7b1
$ touch hello.txt
$ oras push --annotation org.opencontainers.image.created=$(date -d @$(stat -c %Y hello.txt) -u +"%Y-%m-%dT%H:%M:%SZ") --oci-layout hello hello.txt
✓ Exists    hello.txt                                                                            12/12  B 100.00%     0s
  └─ sha256:a948904f2f0f479b8f8197694b30184b0d2ed1c1cd2a1ec0fb85d299a192a447
✓ Exists    application/vnd.oci.empty.v1+json                                                      2/2  B 100.00%     0s
  └─ sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
✓ Uploaded  application/vnd.oci.image.manifest.v1+json                                         588/588  B 100.00%  465µs
  └─ sha256:ec75bba78811a3046d9676b2ef8c92e859cfdc15deaddcf94d0d67db75c20835
Pushed [oci-layout] hello
ArtifactType: application/vnd.unknown.artifact.v1
Digest: sha256:ec75bba78811a3046d9676b2ef8c92e859cfdc15deaddcf94d0d67db75c20835
$ oras push --annotation org.opencontainers.image.created=$(date -d @$(stat -c %Y hello.txt) -u +"%Y-%m-%dT%H:%M:%SZ") --oci-layout hello hello.txt
✓ Exists    application/vnd.oci.image.manifest.v1+json                                         588/588  B 100.00%     0s
  └─ sha256:ec75bba78811a3046d9676b2ef8c92e859cfdc15deaddcf94d0d67db75c20835
Pushed [oci-layout] hello
ArtifactType: application/vnd.unknown.artifact.v1
Digest: sha256:ec75bba78811a3046d9676b2ef8c92e859cfdc15deaddcf94d0d67db75c20835

Are you willing to submit PRs to contribute to this feature?

• Yes, I am willing to implement it.

[cunningr]

I'd like to add to the above the case where we create a manifest from dir with multiple files and dirs. What would be the recommended solution there?

[qweeah]

@cunningr Currently there is a workaround via specifying the org.opencontainers.image.created explicitly.

> ls dir
a  b  c
> time=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
> oras push localhost:5000/test:pushed dir --annotation org.opencontainers.image.created=$time
✓ Uploaded  dir                                                                                                              146/146  B 100.00%   63ms
  └─ sha256:f20484fafe471d3d0542f2643c087064fd0b16ab44b174ca844cb3a352260900
✓ Uploaded  application/vnd.oci.empty.v1+json                                                                                    2/2  B 100.00%   59ms
  └─ sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
✓ Uploaded  application/vnd.oci.image.manifest.v1+json                                                                       729/729  B 100.00%   44ms
  └─ sha256:bc7d73dcb17084e35d17c1fb4c17cce0ffcca6c19e805676b95014dc23172e12
Pushed [registry] localhost:5000/test:pushed
Digest: sha256:bc7d73dcb17084e35d17c1fb4c17cce0ffcca6c19e805676b95014dc23172e12
> oras push localhost:5000/test:pushed dir --annotation org.opencontainers.image.created=$time
✓ Uploaded  application/vnd.oci.image.manifest.v1+json                                                                       729/729  B 100.00%   29ms
  └─ sha256:bc7d73dcb17084e35d17c1fb4c17cce0ffcca6c19e805676b95014dc23172e12
Pushed [registry] localhost:5000/test:pushed
Digest: sha256:bc7d73dcb17084e35d17c1fb4c17cce0ffcca6c19e805676b95014dc23172e12

You can see that both pushing generates the same manifest.

[cunningr]

I encountered a different issue related to this. Maybe it is expected (and desirable) but for us it was not desirable so thought it was worth calling out.

We generate a bunch of manifests using templating and input vars. The resulting manifests are packed and pushed using ORAS. I was able to get deterministic builds using the above and the same instance (generation output run of manifests).

However, if we make two different generation runs of our manifests with exactly the same inputs ORAS will detect changes, presumably due to the file attributes (e.g. create/access timestamp). The contents of the input files are exactly the same but we still get a different resulting hash.

[qweeah]

If file contents are identical, the generated manifest should be the same. Can you double check the org.opencontainers.image.created annotation in the generated manifest to confirm?

[cunningr]

So here is my test:

generate the files and do;

oras push <repo-redacted>\
-a "org.opencontainers.image.created=1970-01-01T00:00:00Z" \
        --config /platform-config/config.json:application/vnd.cncf.flux.config.v1+json \
        ./flux-apps/:application/vnd.cncf.flux.content.v1.tar+gzip; \

I get returned a digest "abc123"

Now i delete my files and regenerate them and do exactly the same oras push command. Now I get a digest "def456". If I oras pull these two manifests and compare the contents using a diff, I see no diffs using git diff --no-index dir1 dir2:

Now I pull the manifest data using oras manifest fetch .... The manifests differ in the digests of the (single) layer:

    {
      "mediaType": "application/vnd.cncf.flux.content.v1.tar+gzip",
      "digest": "sha256:d4f9828c18f2b69b385b7229e17c1039d73b2d6bd9e0c3ff1bc309d3c22335aa",
      "size": 121463,
      "annotations": {
        "io.deis.oras.content.digest": "sha256:c137410faacbc4ba7f69aee6003d47eb86dc5650ad1931c9cc41574817bf8674",
        "io.deis.oras.content.unpack": "true",
        "org.opencontainers.image.title": "flux-apps"
      }
    }

vs

    {
      "mediaType": "application/vnd.cncf.flux.content.v1.tar+gzip",
      "digest": "sha256:ac6b46007ae4b6072c81430503dd2f4122e10d1863f6e284469fb821e97d3fc1",
      "size": 121499,
      "annotations": {
        "io.deis.oras.content.digest": "sha256:89b03ac75587667b858b119c54a18f4f052eb554d77c8e3ae92859bd286416fb",
        "io.deis.oras.content.unpack": "true",
        "org.opencontainers.image.title": "flux-apps"
      }
    }

The annotations are the same in each manifest:

  "annotations": {
    "org.opencontainers.image.created": "1970-01-01T00:00:00Z"
  }

but my git diff on the content of the manifests shows not a single diff. I am using ECR if that might have any relevance.

[qweeah]

I think the mtime(Last Modified Time) of the files are different in different pipelines and causes the digest difference. oras-go provides an option to strip that out but it's not exposed by ORAS CLI.

[qweeah]

@cunningr I have added an enhancement issue oras-project/oras#1464