Negative interactions between SaaS model and distribution release cycles

[AdamWill]

This is a broader topic growing out of https://pagure.io/fedora-iot/issue/57 .

We are currently in the final release process for Fedora 40. In that issue, it turns out we need a fairly small tweak to osbuild in order to produce images without pre-release warnings. The change has been made in the repos, but deployment to the production osbuild service is blocked by issues with an unrelated change set which Fedora does not need or want.

To me, this illustrates a general category of problem which I am worried will keep recurring. The current setup we're attempting to use involves Fedora - and, presumably, RHEL and CentOS - using a shared service-style instance of osbuild to produce images. Send a request to the osbuild service, get an image back.

Currently, it seems the service does not freeze during the release freezes of the distributions that use it, so general development work may be occurring on the service during a distribution development freeze. We might build one candidate for Fedora 40, or RHEL 10, then come back a few days later to build another one, and find that the osbuild service has been updated to a newer version, potentially with unexpected behaviour changes. In the current example, even if there had been no issues with the dependency resolution change set and it had landed smoothly along with the desired (from Fedora's perspective) pre-release note change, that's not what Fedora actually wants, ideally - Fedora wants only the change that is critical to its release to happen during its release freeze. I'm sure RHEL equally does not ideally want changes targeted at Fedora development to land in the service during its release freezes.

This is not (AIUI) how any of the client distributions handles things for pipelines that are entirely under their control. During release freezes, the compose configurations and tooling are also frozen. We do not deploy new versions of koji, pungi, lorax etc. during Fedora release freezes. Any changes to these must be targeted specifically to fix critical issues in the release, and have to go through a freeze break review process before they can land. I believe RHEL and CentOS work similarly.

I'm not sure how this can be entirely satisfactorily addressed with the current model as I understand it. The service could freeze during every client distribution's release freezes - it's frozen for everyone if Fedora is releasing, it's frozen for everyone if RHEL is releasing. I do worry that might go in the other direction and feel like it's freezing "too much", though. For RHEL development, maybe Fedora's constant short freezes would be disruptive. For Fedora development, maybe RHEL's occasional longer(?) freezes would be equally disruptive. Or we could go to a model where every distro has its own osbuild service, but then we lose the efficiency benefits of shared maintenance.

Tagging @nirik as I'm pretty sure he's also interested in this.

[supakeen]

Thank you for this post @AdamWill I'm going to tag @pcdubs and @nullr0ute as well as they work on the definitions for Fedora IoT and Fedora Minimal. Both of which also build through koji, through our SaaS. I hope they can also share their experiences.

The relevant people for the service are @croissanne, @ondrejbudai, and @ochosi so let's tag those too.

Since this is also about the RHEL release process let's get @thozza here too to complete the party 🙂

[croissanne]

I haven't been part of the discussions so far, and don't necessarily have a lot of time to get involved, but it seems to me that the parts that would need to be frozen are the image definitions, not the rest of the service.

So if we can find a way to freeze the image definitions for fedora, and you can keep the osbuild version stable by not updating the fedora workers during the freeze (even though osbuild is backwards compatible), the artefacts would be stable, and what I assume is the problem would be resolved.

Minor tweaks in the fedora image definitions could then be allowed only where it's blocking the release. Making the image definitions easier to work on is a separate problem which I don't think is in the scope of this thread.

[AdamWill]

I'm not sure that necessarily follows. Let's look at live images for an analogy. The "old fashioned" way of building these involves running livemedia-creator on Koji builders, with kickstart files as inputs. It seems to me like the kickstart files map to the "image definitions". And, yes, we freeze the kickstart repository. But we also freeze the entire build environment, including the Koji deployment, the packages installed on the builders, and the version of livemedia-creator itself that is used to build the images. We would not do a version update of livemedia-creator during the freeze (unless the new version specifically fixed a critical bug that we needed to fix for the compose).

It's possible I'm misunderstanding how osbuild works, of course, or how you conceptualize the different areas of it. Please do say if so.

[croissanne]

I think the analogy you make holds. And right, freezing to that extent would make it very hard on the other tenants in the service. If we need to do something like this we'd need to deploy a separate instance of the central service just for fedora, which is not impossible.

Though also keep in mind the entire service is deployed from the history of the main branch, so we might have an issue there if you want backport the required fixes, rather than bump the entire deployment up to where it would be needed. Again could be mitigated to some extent by having the image definitions not in code, or mostly not in code.

[AdamWill]

I don't think this only concerns Fedora - I was taking pains to phrase it in a generic way. We should ask RHEL folks to be sure, but it feels logical to me that if, say, RHEL 10 were in its release freeze, it would not want non-RHEL10-critical development work to be happening in the osbuild production service either. I'd be surprised if only Fedora were concerned about that.

edit: I did kinda wonder if it might be possible to do a sort of 'templated deployment' thing, where it would be easy to spin up multiple instances of the osbuild service from different git branches, so when an osbuild-using distro goes into a release freeze, we would just spin up a frozen instance of the service for it to use, and this would be a totally normal, unremarkable, ideally completely-automated operation. I've no idea how practical that would be, though, and how much work.