Describe how to handle dependencies between repositories

[m-linner-ericsson]

Description

Describe how to handle dependencies between repositories.

Currently Eiffel Intelligence Frontend is dependent on Eiffel Intelligence . If the front-end requires a feature in the back-end there is no point in releasing or building a Docker container for the front-end until the back-end has the functionality merged.

In ETOS on the other hand there is a wish to build the containers on every merge.

Motivation

Need a unified way of handling dependencies between repositories.

Exemplification

See description

Benefits

Avoiding unusable releases and docker containers

Possible Drawbacks

N/A

[m-linner-ericsson]

@t-persson @sodero @fdegir @tobiasake Please give your view on this or ask questions if the issue is hard to understand.

[t-persson]

ETOS is built with tag ":dev" with every merge, meaning that if you are following the ":dev" tag you might get stuff that is not working properly.
We do this in order to be able to test the system as changes come in and to have a system up and running to validate that the dependencies are met.

We then handle the dependencies whenever we create a production release of ETOS in order to make sure that all the components work together.

ETOS is a set of building blocks that, when put together, produces the system. There are no dependencies until the building blocks are put together

Thanks for notifying us :)

[fdegir]

@t-persson while I was working on bringing the basic builds for etos-test-runner-containers repo, I noticed that go, python, and rust container images pull etos-base-test-runner:$DEBIAN release during their build.

I suppose the image etos-base-test-runner:$DEBIAN_RELEASE is built in advance/manually (?) and pushed to the registry?
Just out of curiosity; I am wondering if it is required to build base image first followed by go, python, and rust images so they can use the latest one with :dev for their own :dev versions cause the go, python, and rust images may not be using the latest version of the base image? Or should etos-base-test-runner:$DEBIAN_RELEASE somehow be built and pushed automatically as well?

[tobiasake]

I guess its ok to make latest commit docker image builds, but I take no responsibility if there is incompatibilities between frontend and backend components.
Internally at Ericsson we use the GitHub release of Eiffel components docker images. We build it manually and push it to external DockerHub and also to Ericsson internal Artifactories/Docker registries. From there images is deployed to Ericsson internal Kubernetes clusters.
That I want is that we build Docker images per GitHub releases of Eiffel components also, where the docker image tag version is the same as the GitHub release version.
It has been discussion requests internally about not overwrite and remove a released version, so I want to add a rule also:
It should not be allowed/possible to overwrite nor remove a GitHub release Docker image version.

So I am ok with latest/dev docker image builds, but I would like to complement with that we build docker image when a GitHub release has been made.
Is that something that you other have in mind as well? Or you maybe have this written down somewhere that I have missed.

I have heavy occupied with other projects and do not work with Eiffel that much, so can be slow on answer, sorry for that :)

[t-persson]

@t-persson while I was working on bringing the basic builds for etos-test-runner-containers repo, I noticed that go, python, and rust container images pull etos-base-test-runner:$DEBIAN release during their build.

I suppose the image etos-base-test-runner:$DEBIAN_RELEASE is built in advance/manually (?) and pushed to the registry?
Just out of curiosity; I am wondering if it is required to build base image first followed by go, python, and rust images so they can use the latest one with :dev for their own :dev versions cause the go, python, and rust images may not be using the latest version of the base image? Or should etos-base-test-runner:$DEBIAN_RELEASE somehow be built and pushed automatically as well?

The test runner containers don't get rebuilt that often because they install most dependencies in run-time.
They are updated when one of these things happen

1. We make a change in the structure of the testrunner.
2. A new debian release is available (update base-test-runner and all test-runners)
3. A new tool version (go, python, rust) is released (update all test-runners and not base)

This makes it a bit harder for us to set up an automatic build process, I guess ( at least before we have an Eiffel queue that we can populate with new tool versions ;) )

How about the test-runner-containers jobs are created and we (etos-maintainers) manually trigger them, instead?
With the caveat that the build-args are available as input parameters to the jenkins job.

[fdegir]

@tobiasake thanks for the comments.

I guess its ok to make latest commit docker image builds, but I take no responsibility if there is incompatibilities between frontend and backend components.

I see 2 things about this.
First one is that anything that gets built using the latest commit is considered as dev version and it is not guaranteed to work.
Over time, we will hopefully have more loops in CI/CD to ensure latest from master is deployable at all times but this is also not guaranteed to happen always.

The other thing; what is happening right now is to make this type of stuff community driven/oriented/maintained. If things work with the help of community, it will indirectly benefit the companies consuming these since making a release will hopefully as simple as clicking few buttons since the things are already proven to work at all times. If things don't work, then it is again community's responsibility to look at them collaboratively.

Internally at Ericsson we use the GitHub release of Eiffel components docker images. We build it manually and push it to external DockerHub and also to Ericsson internal Artifactories/Docker registries. From there images is deployed to Ericsson internal Kubernetes clusters.

This is what is in our mind as well but for the community, done openly so everyone can see and contribute to how things are done, helping with moving to community driven approach.
Automated builds will be triggered upon release and the resulting artifacts will be pushed to Nordix Harbor/Artifactory.
These will then be tested on Eiffel Kubernetes Cluster to verify if things work.
Additional activities could be performed depending on what community wants.

That I want is that we build Docker images per GitHub releases of Eiffel components also, where the docker image tag version is the same as the GitHub release version.

The build jobs per GitHub release will be created soon once we settle the basic build process.
Please keep an eye on Eiffel repos for issues we will create for asking reviews/comments/contributions.

It has been discussion requests internally about not overwrite and remove a released version, so I want to add a rule also:
It should not be allowed/possible to overwrite nor remove a GitHub release Docker image version.

Yep, released versions will be kept as is and tags will not be reused.

So I am ok with latest/dev docker image builds, but I would like to complement with that we build docker image when a GitHub release has been made.
Is that something that you other have in mind as well? Or you maybe have this written down somewhere that I have missed.

Yes, definitely. The document linked below describes the current thinking.

https://hackmd.io/X3Nrxe4MRdSJI_lsTkSwiQ?view

I have heavy occupied with other projects and do not work with Eiffel that much, so can be slow on answer, sorry for that :)

No problem and thanks for your comments.
However, I want to stress that it would be great to continue getting your guidance in these since you are one of the people who know how the current process works so we need to pick your brain and get things automated within the community.

[fdegir]

How about the test-runner-containers jobs are created and we (etos-maintainers) manually trigger them, instead?
With the caveat that the build-args are available as input parameters to the jenkins job.

@t-persson we most definitely can do that. I'll send a review or two on Nordix Gerrit and add you as reviewer so we finalize ETOS builds and move to adjusting the builds of other ETOS repos to trigger on after merge.