Proposal for new release management workflows

[cujomalainey]

I would like to propose some changes to our release workflows to help mitigate some gaps I spotted in our source control for the SOF repository. Please feel free to comment on these and give your thoughts.

Release branch life span

Right now release branches can be touched even though they are long unsupported as long as you have the reviews. This makes tracking targets hard, especially with the growing number of branches + all the loose development branches.

Proposed fix

Lock out dead branches and mark them RO in github to make it clear users are targeting the wrong branch. This can be done via protected branches.

Branch namespace pollution

We have a lot of random dev branches in a key repository

Proposal

Delete them and lock out the repo, forcing developers to use forks

Release branch owners

It is not easy to track who should be the reviewer on release branches (i.e. release manager).

Proposed fix

On release branch creation make the release owner a * codeowner so they are auto cced on changes to that branch

Cherrypicking to release branches

Right now reviewers have no ability to verify if a fix landed on main first. We have seen changes miss main and cause problems.

Proposed fix

Require cherry-picks to use -x and have a github action to check if it exists on main otherwise complain so reviewers are aware of the deviation.

[dbaluta]

We also need to better handle tags, because tag is used to display SOF version at runtime. Right now with main you will get sof-2.0.0 because it seems that lately we didn't push anymore tags for releases.

For i.MX branches I handled this separately but it would be nice to have a better rule for main branch.

[mwasko]

Release branch life span
(...)
Proposed fix
Lock out dead branches and mark them RO in github to make it clear users are targeting the wrong branch. This can be done via protected branches.

The question is do we need to keep those dead branches at all? When new platform release branch is created 'platform-n+1-stable' then we probably might delete the previous one and just keep the reference to the code with tag 'release/platform-n-stable'.

Branch namespace pollution
We have a lot of random dev branches in a key repository

Proposal
Delete them and lock out the repo, forcing developers to use forks

The dev branches today are definitely abused in our key repository, but I still see a value in creating large feature branches for reference in the repository.

[cujomalainey]

Release branch life span
(...)
Proposed fix
Lock out dead branches and mark them RO in github to make it clear users are targeting the wrong branch. This can be done via protected branches.

The question is do we need to keep those dead branches at all? When new platform release branch is created 'platform-n+1-stable' then we probably might delete the previous one and just keep the reference to the code with tag 'release/platform-n-stable'.

Deleting releases is dangerous in my opinion, unless we can validate the build is used absolutely no where, and I mean in even LTS distros, then I think we should not do this.

Branch namespace pollution
We have a lot of random dev branches in a key repository

Proposal
Delete them and lock out the repo, forcing developers to use forks

The dev branches today are definitely abused in our key repository, but I still see a value in creating large feature branches for reference in the repository.

Sure then feature owners can own them, or even they can manage a feature branches in their own fork, adding additional remotes is a simple process. Currently its the wild west.

[mwasko]

Deleting releases is dangerous in my opinion, unless we can validate the build is used absolutely no where, and I mean in even LTS distros, then I think we should not do this.

The release tag would be a safety point here, so the reference to the release code will be stored. If there will be such need a release branch can always be restored from the tag.

[cujomalainey]

The release tag would be a safety point here, so the reference to the release code will be stored. If there will be such need a release branch can always be restored from the tag.

Fair point, if we started using tags more liberally then we could delete branches

[cujomalainey]

@mwasko @lgirdwood @bhiregou @dbaluta I wrote a very simple github action to cover issue 4, what are your thoughts?

Example output when checking simply by subject

Checking 334e5cc4f topology1: set Speakers PPL running on C1 by default...OK
Checking ee1d6c760 topology1: define SPK_PLAYBACK_CORE for Speakers running core...OK
Checking c7ff5b764 Tools: Topology1: Add three demuxed PCMs for DMIC capture example...FAIL
  Cannot find Tools: Topology1: Add three demuxed PCMs for DMIC capture example in origin/main
Checking 7f8e01dfc mux: consider sinks on same direction only for prepare/reset propagation...FAIL    
  Cannot find mux: consider sinks on same direction only for prepare/reset propagation in origin/main

Example output when checking by (cherry-picked from tag)

Checking range 9611d126ba5d4939c8a5a841e10663aac2b89e41..0ee2d350dbf43ede990e08246b41f44746c73e6a which contains 76 commits
Checking 0ee2d350d cavs: memory bank powerup flow adjustment...FAIL                                                                                            
  Cannot find aaac08a07251f92ad8d8e356b323ff8803df8302 from 0ee2d350dbf43ede990e08246b41f44746c73e6a in origin/main
Checking 3c1842373 topology1: Align DAI format as SSP config to 24-bit for rt1019...OK                                                                         
Checking 334e5cc4f topology1: set Speakers PPL running on C1 by default...OK
Checking ee1d6c760 topology1: define SPK_PLAYBACK_CORE for Speakers running core...OK
Checking c7ff5b764 Tools: Topology1: Add three demuxed PCMs for DMIC capture example...FAIL       
  Cannot find reference in commit message
  Hint: use -x with git cherry-pick

Actions can be set to run on an inclusion or en exclusion list so it won't run on main. I think this would be a good check that should pass unless there is a good reason not to.

[dbaluta]

@cujomalainey

I agree with your proposal regarding verifying that first the patch is in main then it is backported to release branch. One comment here: We need the possibility to override this by the owner because it can happen that close to an internal release we really need to quickly push a patch in a release branch without waiting for merging into main.

It happened to us, when we needed to wait 5-7 days until the patch was merged even if it got approvals but because of some CI problems we couldn't land it in main in time.

[bhiregoudar]

@cujomalainey

How about the cases where we want to have case based deviation code in release branch than the main? May be due to some H/W issues or specific customer customization? In such scenario, main may not match with release branches.

[cujomalainey]

@bhiregou @dbaluta you can override the check, this is meant to be informational for the reviewers rather than a hardline rule. Also you can set your own rules for release branches on whether or not they need admin override on checks, or if the checks are even required for non-admin submission (see settings screenshot specific for one protected branch namespace)

[cujomalainey]

Also once you require status checks you can select which ones in particular you want. Unless no one has any issues I will go ahead and get this setup as an informational check.

[cujomalainey]

#7263 is the PR for 4

@mwasko will you handle 1 in the conversion to tags?

sounds like 2 and 3 might need a bit more discussion

[mwasko]

#7263 is the PR for 4

@mwasko will you handle 1 in the conversion to tags?

@cujomalainey yes. I have already locked all the tgl, adl and rpl release branches. Next step is to Tag them and then delete branches itself to complete conversion.

Once complete we will continue with clean up of other Intel platform release branches.