Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support multi-arch image/artifact push #1053

Open
sajayantony opened this issue Aug 8, 2023 · 8 comments · May be fixed by #1514
Open

Support multi-arch image/artifact push #1053

sajayantony opened this issue Aug 8, 2023 · 8 comments · May be fixed by #1514
Assignees
Labels
enhancement New feature or request multi-arch spec required Issues that require specifications
Milestone

Comments

@sajayantony
Copy link
Contributor

sajayantony commented Aug 8, 2023

What is the version of your ORAS CLI

1.0.0

What would you like to be added?

As artifacts become increasingly accepted, it would be useful for ORAS to support multi-arch artifacts as well.
There are already some standard conventions of tools like homebrew implementing this.

Multi-arch manifest below - https://explore.ggcr.dev/?image=ghcr.io%2Fhomebrew%2Fcore%2Foras:1.0.1

{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:d0e42e6f6d564a9d1e178e72b42ff12b7e2e3c36cd32b44b150f7465c171db41",
      "size": 1914,
      "platform": {
        "architecture": "arm64",
        "os": "darwin",
        "os.version": "macOS 12.6"
      },
      "annotations": {
        "org.opencontainers.image.ref.name": "1.0.1.arm64_monterey",
        "sh.brew.bottle.digest": "8690763d0a9d0be6574ee4cfdb8edbab302f232c8056bae2d0083f170719afed",
        "sh.brew.bottle.size": "3436422",
        "sh.brew.tab": "{\"homebrew_version\":\"4.1.4-6-ga8022e6\",\"changed_files\":[],\"source_modified_time\":1691075089,\"compiler\":\"clang\",\"runtime_dependencies\":[],\"arch\":\"arm64\",\"built_on\":{\"os\":\"Macintosh\",\"os_version\":\"macOS 12.6\",\"cpu_family\":\"dunno\",\"xcode\":\"14.2\",\"clt\":\"14.2.0.0.1.1668646533\",\"preferred_perl\":\"5.30\"}}"
      }
    },

Each binary is uploaded with the following manifest

 oras manifest get --pretty ghcr.io/homebrew/core/oras@sha256:d0e42e6f6d564a9d1e178e72b42ff12b7e2e3c36cd32b44b150f7465c171db41
{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:0e7e738a321110bc395c82868cd93c6dbd0e4afa7b8931f08a6039ed59544249",
    "size": 222
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:8690763d0a9d0be6574ee4cfdb8edbab302f232c8056bae2d0083f170719afed",
      "size": 3436422,
      "annotations": {
        "org.opencontainers.image.title": "oras--1.0.1.arm64_monterey.bottle.tar.gz"
      }
    }
  ],
  "annotations": {
    "com.github.package.type": "homebrew_bottle",
    "org.opencontainers.image.created": "2023-08-07",
    "org.opencontainers.image.description": "OCI Registry As Storage",
    "org.opencontainers.image.documentation": "https://formulae.brew.sh/formula/oras",
    "org.opencontainers.image.license": "Apache-2.0",
    "org.opencontainers.image.ref.name": "1.0.1.arm64_monterey",
    "org.opencontainers.image.revision": "7716dfd65ccfe0502f5acfc35b3e8c6aa4cc52b1",
    "org.opencontainers.image.source": "https://github.com/homebrew/homebrew-core/blob/7716dfd65ccfe0502f5acfc35b3e8c6aa4cc52b1/Formula/oras.rb",
    "org.opencontainers.image.title": "oras 1.0.1.arm64_monterey",
    "org.opencontainers.image.url": "https://github.com/oras-project/oras",
    "org.opencontainers.image.vendor": "homebrew",
    "org.opencontainers.image.version": "1.0.1",
    "sh.brew.bottle.digest": "8690763d0a9d0be6574ee4cfdb8edbab302f232c8056bae2d0083f170719afed",
    "sh.brew.bottle.size": "3436422",
    "sh.brew.tab": "{\"homebrew_version\":\"4.1.4-6-ga8022e6\",\"changed_files\":[],\"source_modified_time\":1691075089,\"compiler\":\"clang\",\"runtime_dependencies\":[],\"arch\":\"arm64\",\"built_on\":{\"os\":\"Macintosh\",\"os_version\":\"macOS 12.6\",\"cpu_family\":\"dunno\",\"xcode\":\"14.2\",\"clt\":\"14.2.0.0.1.1668646533\",\"preferred_perl\":\"5.30\"}}"
  }
}

Why is this needed for ORAS?

ORAS does not have CLI commands to author a manifest. Being able to author this multi arch manifest or index-based manifest would enable artifact authors to use ORAS CLI for scenarios like above.

Things to keep in mind

  1. Multi-arch platform property is one way for clients to determine the artifact. This is already supported in ORAS throught the --platform flag.
  2. Clients can choose to use annotations or any custom filter from index.
  3. There could be an ability for clients to compose the manifest using a tag and add more artifacts in a multi step script and finally copy the full index and related manifests in one shot.

Composotion

sequenceDiagram
    participant User
    participant LocalRegistry as Local Store
    participant RemoteRegistry as Remote Registry

    User->>LocalRegistry: Begin creation of artifact with tag mybinary:v1 for amd64 linux
    Note right of LocalRegistry: Artifact: mybinary:v1 (linux/amd64) initiated

    User->>LocalRegistry: Add binary for a different platform (e.g. arm64)
    Note right of LocalRegistry: Artifact: mybinary:v1 (linux/amd64, linux/arm64) updated

    User->>LocalRegistry: Add binary with a different annotation
    Note right of LocalRegistry: Artifact: mybinary:v1 (linux/amd64, linux/arm64, annotations)

    User->>RemoteRegistry: Copy artifact mybinary:v1 from Local Registry
    Note right of RemoteRegistry: Artifact: mybinary:v1 (linux/amd64, linux/arm64, annotations)
Loading

Appending

Consider appending a remote manifest and add a new item to the index

sequenceDiagram
    participant User
    participant RemoteRegistry1 as Local Store
    participant RemoteRegistry2 as Remote Registry

    User->>RemoteRegistry1: Inspect artifact with tag mybinary:v1
    Note right of RemoteRegistry1: GET manifest details of mybinary:v1

    User->>User: Determine modifications without pulling full artifact

    User->>RemoteRegistry1: Add binary for a different platform (e.g. linux/arm64) directly
    Note right of RemoteRegistry1: Artifact: mybinary:v1 (linux/amd64, linux/arm64)  updated

    User->>RemoteRegistry: Copy artifact mybinary:v1 from Local Registry
    Note right of RemoteRegistry2: Artifact: mybinary:v1 (linux/amd64, linux/arm64) updated
Loading

Note

  • Currently Index implementations don't support artifactType in some registries and hence we should consider if tools need this even through it will be a part of the image spec.
  • Index support subject and we should consider how we will support subject properties to link and index to another artifact.

/cc @siggy

@sajayantony sajayantony added the enhancement New feature or request label Aug 8, 2023
@shizhMSFT shizhMSFT added this to the future milestone Aug 8, 2023
@shizhMSFT
Copy link
Contributor

/cc @FeynmanZhou @yizha1

@FeynmanZhou
Copy link
Member

There is another request for attaching artifacts to a multi-arch image with ORAS raised by an ORAS user @SamirPS described in #813. Designing a full user story for multi-arch image distribution (pull, push, discover, attach) in ORAS sounds reasonable to me.

@qweeah
Copy link
Contributor

qweeah commented Aug 10, 2023

There is another request for attaching artifacts to a multi-arch image with ORAS raised by an ORAS user @SamirPS described in #813. Designing a full user story for multi-arch image distribution (pull, push, discover, attach) in ORAS sounds reasonable to me.

#813 requires ability to find the correct subject, this issue is proposing ability of packing(composing or authoring) an index, which is different.

@qweeah
Copy link
Contributor

qweeah commented Aug 10, 2023

Composing an index is easy and the composed index can be output to stdout for uploading like:

oras index my.reg/repo tag1 tag2 dgst2 --subject tag3 | oras manifest push -

Platform info can be easily obtained from the predecessor manifest's config, the tricky part is customizing descriptor annotations in manifests. e.g. we can found that the annotations of oras mac build doesn't all presented in the referring index.

I don't have any idea to accommodate the need for selectively add annotation into the index, I suggest we can introduce the index composing feature first.

@qweeah
Copy link
Contributor

qweeah commented Aug 18, 2023

  1. For composition scenario, oras should be able to let user specify platform info when it's not in the artifact.
  2. oras push and oras attach should support --platform to allow user bake artifacts with platform metadata, see support --platform in oras push and oras attach #1066
  3. For appending scenario, oras should not do garbage collection.

@sajayantony sajayantony changed the title Support multi-arch image push Support multi-arch image/artifact push Aug 30, 2023
@yizha1 yizha1 modified the milestones: future, v1.3.0 Mar 4, 2024
@FeynmanZhou FeynmanZhou added the spec required Issues that require specifications label Mar 14, 2024
DelusionalOptimist added a commit to DelusionalOptimist/KubeArmor that referenced this issue May 23, 2024
Currently OS and arch is stored in the tag however once oras CLI
starts to support pushing multi-arch artifacts, we'll use that.
Ref - oras-project/oras#1053

Signed-off-by: Rudraksh Pareek <[email protected]>
Shreyas220 pushed a commit to Shreyas220/KubeArmor that referenced this issue May 31, 2024
Currently OS and arch is stored in the tag however once oras CLI
starts to support pushing multi-arch artifacts, we'll use that.
Ref - oras-project/oras#1053

Signed-off-by: Rudraksh Pareek <[email protected]>
@sajayantony
Copy link
Contributor Author

Adding 2 scripts to show the challenges
https://gist.github.com/sajayantony/0c9d6436c03d531b1bbebe43249381cf
and from @theory https://gist.github.com/theory/7dc164e5772cae652d838a1c508972ae#file-push_trunk

@TerryHowe
Copy link
Member

Blog post https://justatheory.com/2024/06/trunk-oci-poc/

@qweeah
Copy link
Contributor

qweeah commented Jul 24, 2024

This special case of composition scenario might need a index merge command:

The tool chain I used is listed as below

✗ docker version
Client: Docker Engine - Community
 Version:           27.1.0
 API version:       1.46
 Go version:        go1.21.12
 Git commit:        6312585
 Built:             Fri Jul 19 17:43:11 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.1.0
  API version:      1.46 (minimum version 1.24)
  Go version:       go1.21.12
  Git commit:       a21b1a2
  Built:            Fri Jul 19 17:43:11 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.19
  GitCommit:        2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc:
  Version:          1.1.1
  GitCommit:        v1.1.0-20-g52de29d7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

I run docker build and docker save and got below manifest:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:f4c8fbb580a9de23437857b686a96394451e5bd66c464ba025d09c8aee3de9d6",
      "size": 665,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:cea0a71c4a8214c54e655c587607687a112ab1ed852a68add9863fbbb1b93774",
      "size": 566,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:f4c8fbb580a9de23437857b686a96394451e5bd66c464ba025d09c8aee3de9d6",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    }
  ]
}

Worth pointing out that even though I was trying to build a single arch image, still the builder(buildx) generates me an index, whose manifests field points to the image manifest and attestation manifest. The experience should apply to all users building from non-overlayfs image store (in my case, containerd image store)

To better help such user compose multi-arch image from those indexes. We can add a new command to accept index references and flatten the input into one index. Comparing to index create, the advantages are

  1. Performance is better: as it requires less network request to fetch platform information from the config
  2. Annotations added by building tools (like vnd.docker.reference.digest and vnd.docker.reference.type) will be kept.

navin772 pushed a commit to navin772/KubeArmor that referenced this issue Jul 27, 2024
Currently OS and arch is stored in the tag however once oras CLI
starts to support pushing multi-arch artifacts, we'll use that.
Ref - oras-project/oras#1053

Signed-off-by: Rudraksh Pareek <[email protected]>
Signed-off-by: Navin Chandra <[email protected]>
@FeynmanZhou FeynmanZhou linked a pull request Oct 15, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request multi-arch spec required Issues that require specifications
Projects
None yet
7 participants