Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable multi-arch build for cilium-cli image #2782

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 15 additions & 10 deletions .github/workflows/images.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,19 @@ concurrency:
jobs:
build-and-push-prs:
if: ${{ github.repository == 'cilium/cilium-cli' }}
environment: ci
environment: ${{ matrix.gh-env }}
runs-on: ubuntu-24.04
strategy:
aanm marked this conversation as resolved.
Show resolved Hide resolved
matrix:
include:
- name: cilium-cli
dockerfile: ./Dockerfile
platforms: linux/amd64,linux/arm64
gh-env: release
- name: cilium-cli-ci
dockerfile: ./Dockerfile
platforms: linux/amd64
gh-env: ci

steps:
- name: Set up Docker Buildx
Expand Down Expand Up @@ -57,10 +62,10 @@ jobs:
ref: ${{ steps.tag.outputs.tag }}

# main branch or tag pushes
- name: CI Build ${{ matrix.name }}
marcofranssen marked this conversation as resolved.
Show resolved Hide resolved
- name: Build ${{ matrix.name }}
if: ${{ github.event_name != 'pull_request_target' }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's run step only on tag push, not branch push. https://quay.io/repository/cilium/cilium-cli?tab=tags should only contain released images 🙏

Suggested change
if: ${{ github.event_name != 'pull_request_target' }}
if: ${{ github.ref_type == 'tag' }}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@michi-covalent maybe github.ref_type == 'tag' && github.event_name == 'tag'?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was never changed. Happy to change it, but didn't want to modify the current pipeline behavior.

Let me know and I can perform the update.

Copy link
Contributor Author

@marcofranssen marcofranssen Nov 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not even sure if you want to change it, because this would allow the release of the image on merges to main using the commit hash.

See https://github.com/cilium/cilium-cli/pull/2782/files#diff-d0a3d6684c78a148cbf0725d5fe8b5aab6431da05b698a82c9e015516f3020baR48-R57

So changing this would change the existing behavior as that step output is used during the Build here https://github.com/cilium/cilium-cli/pull/2782/files#diff-d0a3d6684c78a148cbf0725d5fe8b5aab6431da05b698a82c9e015516f3020baR99

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the latest workflow file looks good to me. just wanted to make sure we don't push to quay.io/cilium/cilium-cli on push to main branch 🚀🙏

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm don't we still end up pushing to quay.io/cilium/cilium-cli repo on push to main?

# main branch or tag pushes
- name: Build ${{ matrix.name }}
if: ${{ github.event_name != 'pull_request_target' }}
uses: docker/build-push-action@48aba3b46d1b1fec4febb7c5d0c644b249a11355 # v6.10.0
id: docker_build_main
with:
context: .
file: ${{ matrix.dockerfile }}
target: ${{ matrix.name }}
platforms: ${{ matrix.platforms }}
push: true
tags: |
quay.io/${{ github.repository_owner }}/${{ matrix.name }}:latest
quay.io/${{ github.repository_owner }}/${{ matrix.name }}:${{ steps.tag.outputs.tag }}
💭

i suggest keeping this pull request for changing Dockerfile only. let's forcus on getting Dockerfile change merged in this PR. @tklauser and i will set up a release image workflow. we need to create a separate github environment anyways.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we'll do something like this for the release image workflow once the Dockerfile change gets merged => c6cfc8b

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i didn't mean to update the workflow file to refer to a release environment. i need to configure a github environment for this repo and make sure that it works.

let's limit the scope of this pull request to the Dockerfile change 🙏

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without the workflow it won't work right. Hence #2782 (comment)

The Github environment you have to configure basically needs to match the name here https://github.com/cilium/cilium-cli/pull/2782/files#diff-d0a3d6684c78a148cbf0725d5fe8b5aab6431da05b698a82c9e015516f3020baR31

Guess the only thing you need to do is configure the secrets for publishing the image int there.

Why don't you add the environment, (give it a different name to your desires) and then we update it here in the workflow.

I'm not fully getting the problem with the workflow here. Basically all required to use a different gh-env is in place. You only need to add the secret to it.

uses: docker/build-push-action@48aba3b46d1b1fec4febb7c5d0c644b249a11355 # v6.10.0
id: docker_build_ci_main
id: docker_build_main
with:
context: .
file: ${{ matrix.dockerfile }}
Expand All @@ -71,19 +76,19 @@ jobs:
quay.io/${{ github.repository_owner }}/${{ matrix.name }}:latest
quay.io/${{ github.repository_owner }}/${{ matrix.name }}:${{ steps.tag.outputs.tag }}

- name: CI Image Releases digests
- name: Image Releases digests
if: ${{ github.event_name != 'pull_request_target' }}
shell: bash
run: |
mkdir -p image-digest/
echo "quay.io/${{ github.repository_owner }}/${{ matrix.name }}:latest@${{ steps.docker_build_ci_main.outputs.digest }}" > image-digest/${{ matrix.name }}.txt
echo "quay.io/${{ github.repository_owner }}/${{ matrix.name }}:${{ steps.tag.outputs.tag }}@${{ steps.docker_build_ci_main.outputs.digest }}" >> image-digest/${{ matrix.name }}.txt
echo "quay.io/${{ github.repository_owner }}/${{ matrix.name }}:latest@${{ steps.docker_build_main.outputs.digest }}" > image-digest/${{ matrix.name }}.txt
echo "quay.io/${{ github.repository_owner }}/${{ matrix.name }}:${{ steps.tag.outputs.tag }}@${{ steps.docker_build_main.outputs.digest }}" >> image-digest/${{ matrix.name }}.txt

# PR updates
- name: CI Build ${{ matrix.name }}
- name: Build ${{ matrix.name }}
if: ${{ github.event_name == 'pull_request_target' }}
uses: docker/build-push-action@48aba3b46d1b1fec4febb7c5d0c644b249a11355 # v6.10.0
id: docker_build_ci_pr
id: docker_build_pr
with:
context: .
file: ${{ matrix.dockerfile }}
Expand All @@ -93,12 +98,12 @@ jobs:
tags: |
quay.io/${{ github.repository_owner }}/${{ matrix.name }}:${{ steps.tag.outputs.tag }}

- name: CI Image Releases digests
- name: Image Releases digests
if: ${{ github.event_name == 'pull_request_target' }}
shell: bash
run: |
mkdir -p image-digest/
echo "quay.io/${{ github.repository_owner }}/${{ matrix.name }}:${{ steps.tag.outputs.tag }}@${{ steps.docker_build_ci_pr.outputs.digest }}" > image-digest/${{ matrix.name }}.txt
echo "quay.io/${{ github.repository_owner }}/${{ matrix.name }}:${{ steps.tag.outputs.tag }}@${{ steps.docker_build_pr.outputs.digest }}" > image-digest/${{ matrix.name }}.txt

# Upload artifact digests
- name: Upload artifact digests
Expand Down
31 changes: 23 additions & 8 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,26 +3,41 @@
# Copyright Authors of Cilium
# SPDX-License-Identifier: Apache-2.0

FROM docker.io/library/golang:1.23.4-alpine3.19@sha256:5f3336882ad15d10ac1b59fbaba7cb84c35d4623774198b36ae60edeba45fd84 AS builder
FROM --platform=${BUILDPLATFORM} docker.io/library/golang:1.23.4-alpine3.19@sha256:5f3336882ad15d10ac1b59fbaba7cb84c35d4623774198b36ae60edeba45fd84 AS base
RUN apk add --no-cache --update ca-certificates git make
WORKDIR /go/src/github.com/cilium/cilium-cli
RUN apk add --no-cache curl git make ca-certificates
COPY go.* .
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
RUN make

# xx is a helper for cross-compilation
FROM --platform=$BUILDPLATFORM tonistiigi/xx:1.5.0@sha256:0c6a569797744e45955f39d4f7538ac344bfb7ebf0a54006a0a4297b153ccf0f AS xx
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really necessary? It shouldn't be since we cross compile cilium without tonistiigi/xx as an example.

Copy link
Contributor Author

@marcofranssen marcofranssen Sep 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is required within Docker, to compile with CGO_ENABLED=0. Don't recall the exact reason, I think it was related to alpine based builds (muslc, vs glibc).

I also use some buildx features that allow caching and parallel builds for all the supported platforms to have a speedy and optimized build.

I contributed similar to the Spire project couple of years ago.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In essence the tool does following and is basically a small little wrapper.

https://github.com/tonistiigi/xx?tab=readme-ov-file#go--cgo

It wraps the go command by setting some variables accordingly based on the buildx args.

Saves a lot of plumbing.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-requested review from andre. let's see what he says 🙏

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can still use the caches without using this dependency. Personally, I would prefer to not have any wrappers on official builds.

Copy link
Contributor Author

@marcofranssen marcofranssen Nov 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this isn't related to the caching only. It is also to set the go flags accordingly for crosscompile based on the TARGETPLATFORM and TARGETARCH which is a buildx feature.

It is used here https://github.com/cilium/cilium-cli/pull/2782/files#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557R19

Without that we would have to do all the plumbing for that in the RUN statement. I'm not a fan of reinventing the wheel :)


FROM --platform=${BUILDPLATFORM} base AS builder
ARG TARGETPLATFORM
ARG TARGETARCH
COPY --link --from=xx / /
RUN --mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/go/pkg/mod \
xx-go --wrap && \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small little wrapper that sets the correct go build flags and such based on TARGETPLATFORM and TARGETARCH and such.

make && \
xx-verify --static /go/src/github.com/cilium/cilium-cli/cilium
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It also allows verifying the binary, see here.


# cilium-cli is from scratch only including cilium binaries
FROM scratch AS cilium-cli
ENTRYPOINT ["cilium"]
FROM --platform=${BUILDPLATFORM} scratch AS cilium-cli
ENTRYPOINT [""]
USER 1000:1000
LABEL maintainer="[email protected]"
WORKDIR /root/app
COPY --from=builder --chown=root:root --chmod=755 /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /go/src/github.com/cilium/cilium-cli/cilium /usr/local/bin/cilium
COPY --link --from=builder --chown=root:root --chmod=755 /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --link --from=builder --chown=1000:1000 --chmod=755 /go/src/github.com/cilium/cilium-cli/cilium /usr/local/bin/cilium
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Run as non root


# cilium-cli-ci is based on ubuntu with cloud CLIs
FROM ubuntu:24.04@sha256:80dd3c3b9c6cecb9f1667e9290b3bc61b78c2678c02cbdae5f0fea92cc6734ab AS cilium-cli-ci
ENTRYPOINT []
LABEL maintainer="[email protected]"
WORKDIR /root/app
COPY --from=builder /go/src/github.com/cilium/cilium-cli/cilium /usr/local/bin/cilium
COPY --link --from=builder /go/src/github.com/cilium/cilium-cli/cilium /usr/local/bin/cilium

# Install cloud CLIs. Based on these instructions:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to https://github.com/cilium/cilium/blob/78b2e1a189bccd7c863715d3f7708f8fdab1cf59/cilium-cli/Dockerfile, I don't think this will work because, for example, awscli, downloaded on line 53, is for x86_64 only.

Copy link
Contributor Author

@marcofranssen marcofranssen Nov 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can tell you it does work.

https://github.com/cilium/cilium-cli/pull/2782/files#diff-d0a3d6684c78a148cbf0725d5fe8b5aab6431da05b698a82c9e015516f3020baR33

The ci image is only build for amd64. If you want to add arm64 support for that image I'm happy to another PR for that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I meant was that it will not work for arm64 because of the binary downloaded on line 53. The build will work but the resulting image will not work for arm64.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me state it differently, because we are not building this image for arm64 in the workflow, it won't be a problem until we change that in the workflow.

For now this PR only focusses on building the cilium image in both architectures, the ci image will stay with amd64 support only.

# - https://cloud.google.com/sdk/docs/install#deb
Expand Down
Loading