Feat: Build container in Zuul

[joshuai96]

Use Zuul to build new containers.

scs-status-page-container-build in check: Build the whole application on every new PR, to verify build and container build is working.

scs-status-page-container-push in post: Build and push the container image to the registry, with latest and commit_id (commit hash) when the PR was merged.

scs-status-page-container-tag in tag: Retag the container image with a given tag.

Furthermore the scs-status-page-go-build job was removed, as the Containerfile is a multi stage file, that builds the binary itself.

This marks the go build GitHub action obsolete, too.

Closes: #15
Closes: #29

[joshmue]

I had a chat with with @gtema regarding secret management in Zuul. For central-api pipelines, he opted to implement central secret management first, not putting per-repo secrets into git.

This would be the case for this PR as well, @gtema?

[gtema]

yes, we would be managing registry credentials as well as required scanning/sbom building centrally

[joshuai96]

When can we expect these changes to be usable?

[joshuai96]

Removed the tag job, was we never used the tagging mechanism before, when releasing a new version.

[gtema]

using commit_id may be tricky and not as useful as you think. Reason for this is that depending on the merge type commit hash as such may never appear in the main branch (squash vs merge vs rebase). Using PR as a reference is more reliable (pr link is already present in the image labels as org.zuul-ci.change_url)
if you skip tags as such automatically "latest" is added (https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/build-container-image/tasks/build.yaml#L25). If you use something to how zuul is defining tags (https://opendev.org/zuul/zuul/src/branch/master/.zuul.yaml#L287) you'll automatically get proper tags also in the case when you put your job into the tag pipeline

[gtema]

Adding "latest" tag in upload but not in build makes no sense and makes job structure more complex. Generally there is no sense in using different tags/labels between jobs

[joshuai96]

Having them separated was a little oversight, fixing it. Wanted to have the possibility to easily set new tags. But I will use same config for both now.

[joshuai96]

Zuuls way of tagging relies on the usage of SemVer. We never talked about versioning and releasing the status-page-api or any other status page component. I am not opposed to defining a versioning and release system based on SemVer for the status page.

But I still need a more automated way of building a container image, after each PR, as some features are better tested in the local dev environment or test deployment, before "releasing" a versioned image.

In general, the way of promoting an image from a commit sha tag to a version tag, would be preferable, but using the opendev job which used skopeo may introduce new complexities regarding the secrets.

At this point, I do have a question @gtema : Do I need to configure anything else, regarding the secrets? I do have a robot account on harbor for the status page project. Do I need to share these secrets on the Zuul instance, if yes, how?

[gtema]

Zuuls way of tagging relies on the usage of SemVer. We never talked about versioning and releasing the status-page-api or any other status page component. I am not opposed to defining a versioning and release system based on SemVer for the status page.

I just referred at Zuul model for example how it could be used (disregarding of the chosen versioning model) to have single definition of tags working for regular PR and tagged releases

But I still need a more automated way of building a container image, after each PR, as some features are better tested in the local dev environment or test deployment, before "releasing" a versioned image.

Ideally you should be having functional tests here in the project. Local testing is never a better choice. But that is a different story and not in scope.

In general, the way of promoting an image from a commit sha tag to a version tag, would be preferable, but using the opendev job which used skopeo may introduce new complexities regarding the secrets.

promote job upstream is used more to copy image between registries (opendev often uses interim registry where artifacts from different projects are pushed to be tested together (cross-project functional testing). In that model there might be step of "promoting" image from interim registry to the general one (and for copying skopeo is being used). Otherwise normal container build/upload is being used.

At this point, I do have a question @gtema : Do I need to configure anything else, regarding the secrets? I do have a robot account on harbor for the status page project. Do I need to share these secrets on the Zuul instance, if yes, how?

No, you do not need to care about anything as long as your job is pushing to registry.scs.community. For other registry zuul admin would need to allocate new robot account in the target registry, put it in vault and update job to be supporting new registry as well. For you now there is no need to do anything about that

[gtema]

you do not really need provides/requires in jobs here. In Zuul it means something different and used for requirements between different projects and not between jobs in the same project (https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.requires)

[joshuai96]

Misunderstood the usage of it, we were thinking of setting an order, in witch the jobs run, to save on nodes.

We don't care if tests succeed, if the linting fails. We don't care for building, if testing fails, etc.

Is there a way of doing such a workflow?

[gtema]

yes, there is: https://github.com/opentelekomcloud-infra/system-config/blob/main/zuul.d/project.yaml#L68 or https://opendev.org/openstack/codegenerator/src/branch/master/zuul.d/openapi.yaml#L282 as example (https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.dependencies for reference). Building it this way, though, you make your CI longer. You generally should not do this unless jobs are very long running.

It makes much more sense to use pods for running lightweight jobs (https://github.com/SovereignCloudStack/zuul-scs-jobs/blob/main/zuul.d/jobs.yaml#L35) with nodeset: pod-fedora-40. Such jobs are being executed in K8 and do not require nodes to be provisioned. There are few cases where this should not be used:

• image building (docker in docker never worked good)
• very heavy jobs (requiring more cpu/mem).
• very long running jobs (networking bugs on k8 cause Zuul loose control over those)

[joshuai96]

That seems nice, instead of waiting for the nodes.

Is there a list, of node sets and labels, that are currently usable, documented somewhere?

[gtema]

https://github.com/SovereignCloudStack/zuul-scs-jobs/blob/main/zuul.d/nodesets.yaml

[gtema]

so do you plan to drop provides/requires? The only sense to use those would be in combination between status-page-api and status-page-web to build dependencies between projects

[joshuai96]

dropped the provides and requires in jobs

[joshuai96]

Ideally you should be having functional tests here in the project. Local testing is never a better choice. But that is a different story and not in scope.

The project is automatically tested. I meant to try out, like a test deployment with all components around it, like the local KinD deployment. I would prefer, to do it on a per feature (after PR) basis, then tagging and "releasing" a new version, with possible multiple new features. So I can, for example, get a new feature up in local or test deployment, so someone else can implement the corresponding frontend changes, as they depend on a working backend. So a working status page full package can be released.

I could stick to building each PRs container image locally and pushing to harbor manually, but then this whole endeavor would be in vain.

promote job upstream is used more to copy image between registries (opendev often uses interim registry where artifacts from different projects are pushed to be tested together (cross-project functional testing). In that model there might be step of "promoting" image from interim registry to the general one (and for copying skopeo is being used). Otherwise normal container build/upload is being used.

Still a promotion workflow seems more desirable to me. I want to re-tag the last container containing the recent changes, to mark it as "released" under a given version. Rebuilding the container, even with the same code from the same commit, will result in different hashes for the layers, and thus will be regarded as another container image, that overwrites exciting container images/tags.

[gtema]

Ideally you should be having functional tests here in the project. Local testing is never a better choice. But that is a different story and not in scope.

The project is automatically tested. I meant to try out, like a test deployment with all components around it, like the local KinD deployment. I would prefer, to do it on a per feature (after PR) basis, then tagging and "releasing" a new version, with possible multiple new features. So I can, for example, get a new feature up in local or test deployment, so someone else can implement the corresponding frontend changes, as they depend on a working backend. So a working status page full package can be released.

I could stick to building each PRs container image locally and pushing to harbor manually, but then this whole endeavor would be in vain.

And this is easily possible if you include a tag with PR number (like status-page-api:change_31 in case of this PR). Once you merge PR the image is being uploaded and pointed by tag latest and change_31 giving you possibility to test everything what you need. But also as I mentioned above - the case of testing how multiple projects incorporate together is precisely what interim registry is for.
Testing things locally is understandable but is not as good as full automated tests so that nobody needs to setup local env. In the case of this app it is pretty trivial.

promote job upstream is used more to copy image between registries (opendev often uses interim registry where artifacts from different projects are pushed to be tested together (cross-project functional testing). In that model there might be step of "promoting" image from interim registry to the general one (and for copying skopeo is being used). Otherwise normal container build/upload is being used.

Still a promotion workflow seems more desirable to me. I want to re-tag the last container containing the recent changes, to mark it as "released" under a given version. Rebuilding the container, even with the same code from the same commit, will result in different hashes for the layers, and thus will be regarded as another container image, that overwrites exciting container images/tags.

Principle of immutability forbids changing tags and I am not aware of any possibility to achieve that with docker/podman directly. Some registries allows this using "proprietary" api only but in every case it is meant a bad tone. Skopeo would also do a copy resulting in change of all hashes.

That is precisely a reason why you should be having sufficient tests in CI so that when you say you want to tag new release you are confident in it.

At the moment I have neither time nor desire to fiddle with harbor apis to find how to achieve image retagging, so I am not going to implement it in the next future.

[joshuai96]

Principle of immutability forbids changing tags and I am not aware of any possibility to achieve that with docker/podman directly. Some registries allows this using "proprietary" api only but in every case it is meant a bad tone. Skopeo would also do a copy resulting in change of all hashes.

latest is a tag, that get overwritten all the time.

Adding a new tag to an existing image is easily done with podman or docker:

podman pull <image>:<old-tag>
podman tag <image>:<old-tag> <image>:<new-tag>
podman push <image>:<new-tag>

That's exactly what I use in the Makefile of this project, to push my local images to harbor.

In CIs for example, where you can not run docker or podman in another container, crane can be used:

crane tag <image>:<old-tag> <new-tag>

This is no "far of" use case or something hacky. It's a built in functionality in most tools used to manage container images.

After a PR merge, there is a new commit hash -> Use this commit hash as tag for the new image.

After a while, when you want to release one or more changes, you tag the repository -> there is already an image with the last commit hash as tag, having all the changes -> re-tag with tag

Zuul seems to over complicate this, as it is at most 2 lines of bash in every other CI/CD I've ever used. (ignoring credential handling)

Edit:

I could confirm that even skopeo reuses the original layer hashes:

skopeo copy docker://registry.scs.community/status-page/status-page-api:687a28d docker://registry.scs.community/status-page/status-page-api:687a28d-test
Getting image source signatures
Copying blob fe0e581f9d15 skipped: already exists  
Copying blob 4b16a3ef091b skipped: already exists  
Copying blob 69681d43c853 skipped: already exists  
Copying blob f96a1577851c skipped: already exists  
Copying blob e3ca87ad3f63 skipped: already exists  
Copying blob ef8b9bca8ee8 skipped: already exists  
Copying config 0598895dfa done   | 
Writing manifest to image destination

As you can see in the harbor status-page-api registry, there are now 3 tags for the same image.

[gtema]

it's not about Zuul making it anyhow different, but about the fact that you typically should not do this in CI. If you still have such need - feel free to do this manually in harbor and any point in time.

[joshuai96]

it's not about Zuul making it anyhow different, but about the fact that you typically should not do this in CI. If you still have such need - feel free to do this manually in harbor and any point in time.

Why shouldn't I do this in CI?

We're using CI, in this case Zuul, as automation for workflows we do not want to do manually.

I would like to understand how this workflow differs from, for example running the lint and testing in Zuul?

Especially considering that the docs for the opendev container jobs, indicate exactly this workflow:

* :zuul:job:`build-container-image` in `check`
* :zuul:job:`upload-container-image` in `gate`
* :zuul:job:`promote-container-image` in `promote` with
  ``promote_container_method: tag``

opendev - container image playbooks

We're just using different GH pipelines, but the workflow is the same.

I do not expect you to implement any changes to Zuul, harbor, jobs or anything else. I just want to understand your reasoning behind it.

Edit:

The above example, especially handles the case of not using an intermediate registry.

[joshuai96]

@gtema there is still no way to use a promotional workflow like the Zuul example workflows? I think rebuilding the same container, instead of re-tagging, is wasteful and the inconsistencies in the image hashes is very undesirable.

Edit: I finally see the problem you've described.
That was a oversight by me.

The container is build inside of a pull request. Thats the problem. It must be build after the pull request was merged. Then, on a tag, we can re-tag the commit id container to a version container.

[gtema]

@gtema there is still no way to use a promotional workflow like the Zuul example workflows? I think rebuilding the same container, instead of re-tagging, is wasteful and the inconsistencies in the image hashes is very undesirable.

Not publishing tag before the PR got merged is exactly to prevent tag replacement. Publishing container on every commit is not a sane development workflow in my eyes.

[joshuai96]

Not publishing tag before the PR got merged is exactly to prevent tag replacement. Publishing container on every commit is not a sane development workflow in my eyes.

That's not what I planned to do, maybe this is where our discussion derailed.

AFAIK Zuul's docs recommend this workflow:

1. A container build with every change, to check if the change is able to build a container, but that container is not uploaded to any registry.
2. When the changes are reviewed, a "speculative" upload will be made. The last build change container gets (rebuild and) uploaded.
3. When the changes are merged, this "speculative" container gets promoted to a real tag, for example commit id AFTER the PR was merged.
4. Tagging and releasing the projects build a new container with given tags.

This is the workflow they call "Promotion via tags"

I want to modify the workflow to basically not build on a tag or release, just to promote again.

Basically:

1. Build a container, on change, to check if build is possible.
2. After PR is merged, promote last change container to commit container.
3. After Tag and Release promote last commit container to version container.

I did go down the rabbit hole of the original Zuul jobs. AFAIK right now, this should be possible.

My main concern right now is, there are only build and upload jobs for scs in the zuul container jobs.

Another way would be to still use the check pipeline and a container build as check if the container can be build. Not upload any post PR merge containers, only a fully versioned container on a tag.

[joshuai96]

@gtema @o-otte if you're all happy with the changes now, I would appreciate a review/approval of the changes.

[gtema]

Not publishing tag before the PR got merged is exactly to prevent tag replacement. Publishing container on every commit is not a sane development workflow in my eyes.

That's not what I planned to do, maybe this is where our discussion derailed.

could be

AFAIK Zuul's docs recommend this workflow:

1. A container build with every change, to check if the change is able to build a container, but that container is not uploaded to any registry.

2. When the changes are reviewed, a "speculative" upload will be made. The last build change container gets (rebuild and) uploaded.

wrong - there is no upload to public repo happening before the merge is triggered. There is a flow that uploads image to the interim registry which may be used when few projects need to access each other so that you have possibility to test such artifacts automatically (this is not public)

3. When the changes are merged, this "speculative" container gets promoted to a real tag, for example commit id _AFTER_ the PR was merged.

wrong. Part of the gate the is an upload job which pushes change_<PR_NUM>latest tag of the image. The promote job (which runs only when change successfully merges) renames change<PR_NUM>_latest to whatever is the "regular" tag (i.e. latest)

4. Tagging and releasing the projects build a new container with given tags.

upon tagging a new image is being build. Well, it depends on which job you use but this is typically the case.

This is the workflow they call "Promotion via tags"

I want to modify the workflow to basically not build on a tag or release, just to promote again.

Basically:

1. Build a container, on change, to check if build is possible.

2. After PR is merged, promote last change container to commit container.

3. After Tag and Release promote last commit container to version container.

Once PR is merged and post pipeline completes Zuul has no access an no info about previously uploaded images. Therefore you can not implement something like: on tag take last uploaded container and retag it. This is the same for GitHub actions and is basically dictated by the PR workflow.

Please explain what do you understand under "promote"

I did go down the rabbit hole of the original Zuul jobs. AFAIK right now, this should be possible.

My main concern right now is, there are only build and upload jobs for scs in the zuul container jobs.

Another way would be to still use the check pipeline and a container build as check if the container can be build. Not upload any post PR merge containers, only a fully versioned container on a tag.

jobs:

• xxx-build - builds conatiner
• xxx-upload - uploads container to any repo
• xxx-promote - replaces tag of the image uploaded in the xxx-upload renaming it from change_<PR_NUM>_latest to i.e. latest (that is only possible during "merging" process and once PR goes to closed there is no access to anything done there).

P.S. there is huge difference between gate and merge from the GitHub pov in that Zuul merges PR once gate pipeline completes. If you bypass zuul merging and do it manually in GH gate pipeline is not triggered and the whole workflow looks all of a sudden differently

[gtema]

Not publishing tag before the PR got merged is exactly to prevent tag replacement. Publishing container on every commit is not a sane development workflow in my eyes.

That's not what I planned to do, maybe this is where our discussion derailed.

could be

AFAIK Zuul's docs recommend this workflow:

1. A container build with every change, to check if the change is able to build a container, but that container is not uploaded to any registry.

2. When the changes are reviewed, a "speculative" upload will be made. The last build change container gets (rebuild and) uploaded.

wrong - there is no upload to public repo happening before the merge is triggered. There is a flow that uploads image to the interim registry which may be used when few projects need to access each other so that you have possibility to test such artifacts automatically (this is not public). Exception is if there is a post-check pipeline invoking xxx-upload job.

3. When the changes are merged, this "speculative" container gets promoted to a real tag, for example commit id _AFTER_ the PR was merged.

wrong. Part of the gate the is an upload job which pushes change_<PR_NUM>latest tag of the image. The promote job (which runs only when change successfully merges) renames change<PR_NUM>_latest to whatever is the "regular" tag (i.e. latest)

4. Tagging and releasing the projects build a new container with given tags.

upon tagging a new image is being build. Well, it depends on which job you use but this is typically the case.

This is the workflow they call "Promotion via tags"
I want to modify the workflow to basically not build on a tag or release, just to promote again.
Basically:

1. Build a container, on change, to check if build is possible.

2. After PR is merged, promote last change container to commit container.

3. After Tag and Release promote last commit container to version container.

Once PR is merged and post pipeline completes Zuul has no access an no info about previously uploaded images. Therefore you can not implement something like: on tag take last uploaded container and retag it. This is the same for GitHub actions and is basically dictated by the PR workflow.

Please explain what do you understand under "promote"

I did go down the rabbit hole of the original Zuul jobs. AFAIK right now, this should be possible.
My main concern right now is, there are only build and upload jobs for scs in the zuul container jobs.
Another way would be to still use the check pipeline and a container build as check if the container can be build. Not upload any post PR merge containers, only a fully versioned container on a tag.

jobs:

* xxx-build - builds conatiner

* xxx-upload - uploads container to any repo

* xxx-promote - replaces tag of the image uploaded in the xxx-upload renaming it from change_<PR_NUM>_latest to i.e. latest (that is only possible during "merging" process and once PR goes to closed there is no access to anything done there).

P.S. there is huge difference between gate and merge from the GitHub pov in that Zuul merges PR once gate pipeline completes. If you bypass zuul merging and do it manually in GH gate pipeline is not triggered and the whole workflow looks all of a sudden differently